Testing out products - benchmarking software

Hey all

I have been asked to test some software. I have installed a VM and now have to test certain AV products and do a comparison (boot time once installed , scan time etc etc). What is the best procedure? I have created a Vista VM (running VM workstation) and are going to fully update it. I will load no other software.

Will the performance not be accurate because its in a VM on this machine? Must I do anything to my machine as in not do anything while performing tests? Must I disconnect from the network?

Any tips would be appreciated.
Thanks.

Disconect it from the network set it up as though it's just a normal desktop PC that grandma and grandpa would use. :)

Fresh install no VM, and do the test not only on Vista but XP etc. just set it up so it's got bare essentials but is fully updated ie all service packs etc.

and when installing the app just use the normal install method don't custom setup. ;)

That way when testing each one they are all on a level playing field.

Also while testing each one make sure you visit site's with malware etc and see witch one's pick up and witch one's don't. Also disable the av after the first scan and purposly drop a virrii/trojan or something bad onto the computer and re-enable it and do a scan and see witch one's detect it and witch don't.

Place the virrii in the exact location for each app test that way it's all fair.

Hey T,

This will require an extra box which I don't have at the moment - I am going to have to do it in a VM, however after installing KAV vista crashes on startup! IM trying to time the boot time, sigh.

Anyway , thanks for the advice.

I agree with t34b4g5's approach re: no VM and disconnected. However, after performing the "out of network" evals, I would place them "in network" for real (your specific env) world experiences.

Also, before you start you might want to ghost your test machine so that you can easily "uninstall" and try a different AV product with the same image. This will remove OS differences as a potential variable. It also might save time and headaches.

Cider,

1. Do not use VM for testing of this nature. OK for checking updates and fixes, but not for evaluating new stuff.

2. You need "reference machines" that are identical, or a good stopwatch and logging system, and run the tests sequentially.

And another word of advice.............. if you are testing AV products or any other commercial software for that matter............. read the bloody terms and conditions FIRST!!!!!

Standard AUPs generally forbid that you publish anything you find, or even use this other than for your own decision making? If you are working in a commercial environment and use this information to advise customers you will need formal, legal, permission from the owners of the IP rights to even conduct such a test.

I will bet a trillion Rands that your POS employers do not have such permission :D

Your employers will fire you and make you totally unemployable if it would save them a few thousand rands in legal fees and compensation?

Hey Nihil,

Wow - didn't actually think of that. I think I will go and tell my boss that this is illegal. Im not even going to read the EULA because common sense tell me that obviously its in there.

Meh < stupid me.

Its only illegal to publish it I think. If it is for your own company you may be alright as long as you do not make your findings accessible to the public, i.e. on the internet :-p Its too bad vmware cant help you with testing with this nature because snapshots are awesomeee. But you may want to consider to use at LEAST xp restore points that way you can restore after you are done testing that a/v so the reminants do not interfere with the next tests.

Hey oofki,

Good point on the restore points. This evaluation will be used for sales purposes, it wont be public however the company we are pitching too is quite a large corporation with offices around the globe so yeah I think there might be legal implications.

VMWARE snapshots the next best thing besides Vistas snipping tool.

Actually what is probably even better then a built in snapshot is ghost corporate (if you have access to it) You just make a clean install and make a cd with a bootable image of the clean install. Nothing needs to be installed on the system its self and who knows what is left behind after using other snapshot utilities..

http://www.symantec.com/business/ghost-solution-suite

You talking about this?

Actually no it doesnt look like they make ghost enterprise edition anymore :(

Hi Cider,

It is OK to test stuff for your own purposes, in fact you would be rather foolish if you didn't. [All the AV companies test their competitors' products but you never see them try to use the results in their marketing.]

You can then say "We have tried the major solution providers in this area and product "xyz" is the one that we recommend and are confident in supporting" or something like that.

There are a number of independent research organisations that test security software. I suppose you could use the results that they publish as they are in the public domain (you might have to subscribe to get the reports).

Problem with that is the league table changes every time they do a test so the results may not suit your purposes?

To be honest with you, when I look at software I will accept third party comments on whether it can do the job and all, but I would test for each individual environment for compatibility, resource consumption, and speed/performance.

:)

EDIT: As for VM; I would only teast in VM if the target environment was going to be VM. Otherwise the results might be unreliable and you may have issues that are a result of the VM.

Oh I asked at work about what happened to ghost corporate and Cider yes you were right.. sorta... haha I guess they took a few of their products and put them together in the package you found there.

Ill check it out :)

Thanks for the tips.