As i understand
It is three parts:
1) do analyse of malware in the virtual enviroment
Wiki: http://en.wikipedia.org/wiki/Virtual_machine
2) debugg and monitoring of runing processes
http://en.wikipedia.org/wiki/Debugger
3) network sniffing
http://en.wikipedia.org/wiki/Packet_sniffer

It is alot of other software that can be used.
As diskimaging software, disassembler, instruction preventaion software.
Use herustic alalys and Instruction preventation software to protect and test protection against differnt kind of malware.