IDS with AI?

[r04dki11]

I have been working with (read monitoring) active and passive IDSs for about five years now. I have tested various blends of string/rule based, anomaly based, behaviour based and have been in various lengthy discussions on AI and nerual net IDS.

The biggest drawback with any system that will "learn" is that if a particular pattern of activity is seen often enough by the AI it will "learn" that it is valid activity. With a monitored system of any basis you will have the human (a.k.a. button-pushing monkey) sitting on the other end of a connection who can apply far more interpretation than any AI or nerual net I have seen demo'd so far. A human monitor can recall a past event and find similarities (so long as they are paying attention) and begin to see patterns where the neural net and AI systems will look back to see that the previous similar event was considered "safe" and had been normalized.

That is the biggest draw-back that I am aware of. The automated systems have the bad habit of accepting consistent events as normal given enough time. I am not saying AI and neural nets will not eventually work well - I hope they do - but I don't think the human monitor should ever be taken completely out of the loop.

[catch]

This has been done, it was actually the subject of a DARPA solicitation a while back. In fact the network at my company uses a setup like that, I have attached a map (I just threw this together as there would be legal issues with my sharing any of the actual network topographies so please excuse the format)

In this situation we start at the internet and work down to the outside IDS system, you will note that there are three of these, they rotate with one off at all times so if one is compromised to hide anything odd, the rotation will detect the irregularities. Next you move down to the router, in this case we'll stay to the right, next you have your content based firewall FWTK, Guantlet, Sidewinder whatever, now all potential malware is mirrored to another system using the same layout as the desktop systems only with a custom auditing system and basically no security. This system automatically executes anything directed to it and if this object tries anything tricky then aflicted desktops are automatically isolated.

An area lacking here is of course time delay malware, but ideally there is enough time to discover these before they detonate and a quick search of IDS logs should indicate what systems need to be cleaned.

This example is very vague but I think it should be useful to explain a simple method of realtime IDS learning.

catch

[r04dki11]

That is an impressive layout, catch. This is running completely on an AI based IDS???

I see the benefit behind the redundancy in the IDS sensors but I don't fully follow your layout (allowing that this was hastily sketched and is not the real topography). Wouldn't it work more to your advantage to have the malware detector in front of your desktops?

Also, are the IDSs running purely on anomaly and behaviours or is there still some use of a string/rule in there also? for that kind of complexity these systems would have to have some extreme horsepower.

Have you ever tried putting a router in front of the whole thing and have the IDS feed it an ACL update on all traffic falling foul of the rules? What about the problem of neural nets learning to normalize what they see often enough? how have you gotten passed that?

[catch]

The malware detection system is actually just a Windows 2000 pro system setup as a work alike to the desktops, it however has a few custom kernel drivers that audit for specific behaviors. Anythin deemed as a new executable by the proxying firewall is forked to the malware detection system (which is actually several systems but you get the idea). From here it is automatically executed and its behavior observed, this allows it to 'learn' new evil signatures. these signatures are then fed into the "normal" signature based IDS systems atop the diagram.

The malware detector isn't a filter, and to use it as such would slow down network traffic a standstill at times as it tests new software before passing it on. The idea of the malware detector in this situation is to effectively quarentine worms and the likes by educating IDS systems enterprise-wide after first discovery. This approach grants that a few systems will be infected and takes the aim of detection and isolation rather than prevention as preventing every new attack against COTS desktops is unrealistic to say the least.

No nueral net is used, just a system to detect if new software does stuff it oughtn't but outside the current protection of the software on hand. This struck us as a far more effective learning environment than attempting to utilize more conventional AI ideals.

catch