-
January 28th, 2009, 07:41 AM
#1
Testing out products - benchmarking software
Hey all
I have been asked to test some software. I have installed a VM and now have to test certain AV products and do a comparison (boot time once installed , scan time etc etc). What is the best procedure? I have created a Vista VM (running VM workstation) and are going to fully update it. I will load no other software.
Will the performance not be accurate because its in a VM on this machine? Must I do anything to my machine as in not do anything while performing tests? Must I disconnect from the network?
Any tips would be appreciated.
Thanks.
The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it.
Albert Einstein
-
January 28th, 2009, 01:07 PM
#2
Disconect it from the network set it up as though it's just a normal desktop PC that grandma and grandpa would use.
Fresh install no VM, and do the test not only on Vista but XP etc. just set it up so it's got bare essentials but is fully updated ie all service packs etc.
and when installing the app just use the normal install method don't custom setup.
That way when testing each one they are all on a level playing field.
Also while testing each one make sure you visit site's with malware etc and see witch one's pick up and witch one's don't. Also disable the av after the first scan and purposly drop a virrii/trojan or something bad onto the computer and re-enable it and do a scan and see witch one's detect it and witch don't.
Place the virrii in the exact location for each app test that way it's all fair.
Last edited by t34b4g5; January 28th, 2009 at 01:10 PM.
-
January 28th, 2009, 01:33 PM
#3
Hey T,
This will require an extra box which I don't have at the moment - I am going to have to do it in a VM, however after installing KAV vista crashes on startup! IM trying to time the boot time, sigh.
Anyway , thanks for the advice.
The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it.
Albert Einstein
-
January 28th, 2009, 01:34 PM
#4
I agree with t34b4g5's approach re: no VM and disconnected. However, after performing the "out of network" evals, I would place them "in network" for real (your specific env) world experiences.
Also, before you start you might want to ghost your test machine so that you can easily "uninstall" and try a different AV product with the same image. This will remove OS differences as a potential variable. It also might save time and headaches.
In God We Trust....Everything else we backup.
-
January 28th, 2009, 10:13 PM
#5
Cider,
1. Do not use VM for testing of this nature. OK for checking updates and fixes, but not for evaluating new stuff.
2. You need "reference machines" that are identical, or a good stopwatch and logging system, and run the tests sequentially.
And another word of advice.............. if you are testing AV products or any other commercial software for that matter............. read the bloody terms and conditions FIRST!!!!!
Standard AUPs generally forbid that you publish anything you find, or even use this other than for your own decision making? If you are working in a commercial environment and use this information to advise customers you will need formal, legal, permission from the owners of the IP rights to even conduct such a test.
I will bet a trillion Rands that your POS employers do not have such permission
Your employers will fire you and make you totally unemployable if it would save them a few thousand rands in legal fees and compensation?
-
January 29th, 2009, 08:49 AM
#6
Hey Nihil,
Wow - didn't actually think of that. I think I will go and tell my boss that this is illegal. Im not even going to read the EULA because common sense tell me that obviously its in there.
Meh < stupid me.
The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it.
Albert Einstein
-
January 29th, 2009, 08:57 AM
#7
Its only illegal to publish it I think. If it is for your own company you may be alright as long as you do not make your findings accessible to the public, i.e. on the internet :-p Its too bad vmware cant help you with testing with this nature because snapshots are awesomeee. But you may want to consider to use at LEAST xp restore points that way you can restore after you are done testing that a/v so the reminants do not interfere with the next tests.
-
January 29th, 2009, 09:32 AM
#8
Hey oofki,
Good point on the restore points. This evaluation will be used for sales purposes, it wont be public however the company we are pitching too is quite a large corporation with offices around the globe so yeah I think there might be legal implications.
VMWARE snapshots the next best thing besides Vistas snipping tool.
The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it.
Albert Einstein
-
January 29th, 2009, 09:39 AM
#9
Actually what is probably even better then a built in snapshot is ghost corporate (if you have access to it) You just make a clean install and make a cd with a bootable image of the clean install. Nothing needs to be installed on the system its self and who knows what is left behind after using other snapshot utilities..
-
January 29th, 2009, 10:08 AM
#10
The world is a dangerous place to live; not because of the people who are evil, but because of the people who don't do anything about it.
Albert Einstein
Similar Threads
-
By GbinaryR in forum AntiVirus Discussions
Replies: 11
Last Post: October 30th, 2008, 09:33 AM
-
By mmkhan in forum Network Security Discussions
Replies: 3
Last Post: July 27th, 2005, 05:05 AM
-
By gore in forum Newbie Security Questions
Replies: 11
Last Post: December 29th, 2003, 08:01 AM
-
By tonybradley in forum AntiOnline's General Chit Chat
Replies: 0
Last Post: April 28th, 2003, 10:20 AM
-
By Ennis in forum Security Archives
Replies: 9
Last Post: February 1st, 2002, 08:51 PM
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
|