Page 1 of 2 12 LastLast
Results 1 to 10 of 14

Thread: Testing out products - benchmarking software

  1. #1
    Only african to own a PC! Cider's Avatar
    Join Date
    Jun 2003
    Location
    Israel
    Posts
    1,683

    Testing out products - benchmarking software

    Hey all

    I have been asked to test some software. I have installed a VM and now have to test certain AV products and do a comparison (boot time once installed , scan time etc etc). What is the best procedure? I have created a Vista VM (running VM workstation) and are going to fully update it. I will load no other software.

    Will the performance not be accurate because its in a VM on this machine? Must I do anything to my machine as in not do anything while performing tests? Must I disconnect from the network?

    Any tips would be appreciated.
    Thanks.
    The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it.
    Albert Einstein

  2. #2
    Senior Member t34b4g5's Avatar
    Join Date
    Sep 2003
    Location
    Australia.
    Posts
    2,391
    Disconect it from the network set it up as though it's just a normal desktop PC that grandma and grandpa would use.

    Fresh install no VM, and do the test not only on Vista but XP etc. just set it up so it's got bare essentials but is fully updated ie all service packs etc.

    and when installing the app just use the normal install method don't custom setup.

    That way when testing each one they are all on a level playing field.

    Also while testing each one make sure you visit site's with malware etc and see witch one's pick up and witch one's don't. Also disable the av after the first scan and purposly drop a virrii/trojan or something bad onto the computer and re-enable it and do a scan and see witch one's detect it and witch don't.

    Place the virrii in the exact location for each app test that way it's all fair.
    Last edited by t34b4g5; January 28th, 2009 at 01:10 PM.

  3. #3
    Only african to own a PC! Cider's Avatar
    Join Date
    Jun 2003
    Location
    Israel
    Posts
    1,683
    Hey T,

    This will require an extra box which I don't have at the moment - I am going to have to do it in a VM, however after installing KAV vista crashes on startup! IM trying to time the boot time, sigh.

    Anyway , thanks for the advice.
    The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it.
    Albert Einstein

  4. #4
    AO's Filibustier Cheap Scotch Ron's Avatar
    Join Date
    Nov 2008
    Location
    Swamps of Jersey
    Posts
    378
    I agree with t34b4g5's approach re: no VM and disconnected. However, after performing the "out of network" evals, I would place them "in network" for real (your specific env) world experiences.

    Also, before you start you might want to ghost your test machine so that you can easily "uninstall" and try a different AV product with the same image. This will remove OS differences as a potential variable. It also might save time and headaches.
    In God We Trust....Everything else we backup.

  5. #5
    Senior Member nihil's Avatar
    Join Date
    Jul 2003
    Location
    United Kingdom: Bridlington
    Posts
    17,188
    Cider,

    1. Do not use VM for testing of this nature. OK for checking updates and fixes, but not for evaluating new stuff.

    2. You need "reference machines" that are identical, or a good stopwatch and logging system, and run the tests sequentially.

    And another word of advice.............. if you are testing AV products or any other commercial software for that matter............. read the bloody terms and conditions FIRST!!!!!

    Standard AUPs generally forbid that you publish anything you find, or even use this other than for your own decision making? If you are working in a commercial environment and use this information to advise customers you will need formal, legal, permission from the owners of the IP rights to even conduct such a test.

    I will bet a trillion Rands that your POS employers do not have such permission

    Your employers will fire you and make you totally unemployable if it would save them a few thousand rands in legal fees and compensation?

  6. #6
    Only african to own a PC! Cider's Avatar
    Join Date
    Jun 2003
    Location
    Israel
    Posts
    1,683
    Hey Nihil,

    Wow - didn't actually think of that. I think I will go and tell my boss that this is illegal. Im not even going to read the EULA because common sense tell me that obviously its in there.

    Meh < stupid me.
    The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it.
    Albert Einstein

  7. #7
    Senior Member
    Join Date
    Oct 2003
    Location
    MA
    Posts
    1,052
    Its only illegal to publish it I think. If it is for your own company you may be alright as long as you do not make your findings accessible to the public, i.e. on the internet :-p Its too bad vmware cant help you with testing with this nature because snapshots are awesomeee. But you may want to consider to use at LEAST xp restore points that way you can restore after you are done testing that a/v so the reminants do not interfere with the next tests.

  8. #8
    Only african to own a PC! Cider's Avatar
    Join Date
    Jun 2003
    Location
    Israel
    Posts
    1,683
    Hey oofki,

    Good point on the restore points. This evaluation will be used for sales purposes, it wont be public however the company we are pitching too is quite a large corporation with offices around the globe so yeah I think there might be legal implications.

    VMWARE snapshots the next best thing besides Vistas snipping tool.
    The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it.
    Albert Einstein

  9. #9
    Senior Member
    Join Date
    Oct 2003
    Location
    MA
    Posts
    1,052
    Actually what is probably even better then a built in snapshot is ghost corporate (if you have access to it) You just make a clean install and make a cd with a bootable image of the clean install. Nothing needs to be installed on the system its self and who knows what is left behind after using other snapshot utilities..

  10. #10
    Only african to own a PC! Cider's Avatar
    Join Date
    Jun 2003
    Location
    Israel
    Posts
    1,683
    The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it.
    Albert Einstein

Similar Threads

  1. Trojans - Ports
    By GbinaryR in forum AntiVirus Discussions
    Replies: 11
    Last Post: October 30th, 2008, 09:33 AM
  2. Review: Penetration Testing Software
    By mmkhan in forum Network Security Discussions
    Replies: 3
    Last Post: July 27th, 2005, 05:05 AM
  3. Tcp/ip
    By gore in forum Newbie Security Questions
    Replies: 11
    Last Post: December 29th, 2003, 08:01 AM
  4. Software Bugs In Your Dishwasher
    By tonybradley in forum AntiOnline's General Chit Chat
    Replies: 0
    Last Post: April 28th, 2003, 10:20 AM
  5. Security Software Links! ;)
    By Ennis in forum Security Archives
    Replies: 9
    Last Post: February 1st, 2002, 08:51 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •