IDS with AI?

Just wanted to ask some of you all that would have an informed opinion about the idea of artificial intelligence in IDS's and firewalls

I've got a guy who's a wiz at AI and dabbles quite a bit in computer/network security and thinks it's a good idea, but he wants me and a friend to work on it with him. Acctually it's my friends Dad. He thinks we can train his latest ai implementation that he wrote in java to do ids stuff. and oh yeah, it's genetic, i.e. like natural selection type stuff where it rewrites it's own code based on what works. I'm not gonna try to understand the specifics at least yet. Though he wants to teach me.

Just wanted to throw something out for discussion. I'd love some ideas, but we're probably gonna try and market this, so you'd just be helping a goofy closed source (for now at least, he's got patents on the stuff) endeavor.

I don't have specifics yet, i'm a humble c++ coder turned PC tech, not exactly sure. I'm not even sure i'm gonna do it yet. He says we'd have to make modules that set ids info up for the ai to process. you know, do the scans, check the exe names, tell it how to tell what's going on on the comp. Well, that's all I know, and hadn't heard anything about it here yet, though i don't come here every day. thanks in advanced for comments

That's a very interesting subject but I'm sorry to tell u that it is not a brand new subject.

I don't see really how a String based or signature based IDS such as snort would use an AI implementation since AI is not so far able to define new attacks signature.

But it could help for:
- signature-based active IDS that would dynamically take the decision to block e frame or reply with false information to a probe.
- False alarm & negative DoS
- log correlation
- a my opinion AI would hugely heuristics based IDS.

If a site is created for your project shares the url...

I know it's not new, that's why I was asking for opinions and such. And we haven't even committed to it yet, so there's no site. But I'd be glad to pass it along if we set one up.

I'm just putting out something to talk about that's not a goofy newbie problem or whatever, just something to chat about, I thought it was interesting.

Its a cool idea, been messing with it for a while;however, i beleive cisco netranger has some what of an AI....Im not sure i would call it A.I but it does do **** automatically I suppose, like automatically block ip that are harrassing the network stuff like that, tweaks rule sets if there seems to be to many pos-falsitives....etc.

that sounds more like an expert system, like most rule based ids's ai's different, It's supposed to learn/come up with it's own ideas and test them. Provided of course, you can write a module for it that gives it relavant data for tests

My alma mater (university) is currently involved in a research tying IDS together with AI. I think they are currently using some sort of a neural net, but I haven't talked to the lead professor in a year or so about it (was thinking about making this a master's thesis topic).

Went to their web site to see if the abstracts were out there but unfortunately that section seems to be a good bit out of date.

/nebulus

on one hand, it seems like that much more complicated code and therefore that much more that could go wrong, but then again, having a firewall or intrusion detection system that could independently make decisions as to what traffic goes in and out and what connections are valid, sounds pretty good to me.

But we still would have the same problems with false positives, in fact i think you could actually have more just because if the ids tests an "idea" of its own and thinks it works, then what happens if that "idea" just screwed alot of users out of vpn access?

It's a very good idea,

But surely if a machine is capable of figuring out new attacks, then your going to have a lot of script kiddies running AI programs to find new vunerabiltys, so the AI on an IDS would always have to be that one step ahead,

Very interesting idea, the implementation on neural nets really interest me,

I suppose the AI would basically be looking out for key factors in an attack, like port scans and the likes.

the thing is a system like this is brilliant it would just be let down by people writing down there passwords for others to see, dumpster diving and social engineering. the human factor really is the weak link.......as always

i2c

Quote:

---

But we still would have the same problems with false positives, in fact i think you could actually have more just because if the ids tests an "idea" of its own and thinks it works, then what happens if that "idea" just screwed alot of users out of vpn access?

---

This is exactly why my university is using Neural nets. The Neural net is able to learn what normal traffic looks like and based on what you tell it to ignore or not, will continually learn more and more to the point to where the chances of false positives are very small (in theory). The second benefit is you now have a reliable anomaly based IDS (not signature based) and have the potential to catch much more traffic than something like Snort (because they are signature based you can alter your attack to not match the signature yet still be effective). IMHO, in the next five years, this type of an approach will become the standard (in practice it will probably be a hybrid).

/nebulus

I have been working with (read monitoring) active and passive IDSs for about five years now. I have tested various blends of string/rule based, anomaly based, behaviour based and have been in various lengthy discussions on AI and nerual net IDS.

The biggest drawback with any system that will "learn" is that if a particular pattern of activity is seen often enough by the AI it will "learn" that it is valid activity. With a monitored system of any basis you will have the human (a.k.a. button-pushing monkey) sitting on the other end of a connection who can apply far more interpretation than any AI or nerual net I have seen demo'd so far. A human monitor can recall a past event and find similarities (so long as they are paying attention) and begin to see patterns where the neural net and AI systems will look back to see that the previous similar event was considered "safe" and had been normalized.

That is the biggest draw-back that I am aware of. The automated systems have the bad habit of accepting consistent events as normal given enough time. I am not saying AI and neural nets will not eventually work well - I hope they do - but I don't think the human monitor should ever be taken completely out of the loop.

This has been done, it was actually the subject of a DARPA solicitation a while back. In fact the network at my company uses a setup like that, I have attached a map (I just threw this together as there would be legal issues with my sharing any of the actual network topographies so please excuse the format)

In this situation we start at the internet and work down to the outside IDS system, you will note that there are three of these, they rotate with one off at all times so if one is compromised to hide anything odd, the rotation will detect the irregularities. Next you move down to the router, in this case we'll stay to the right, next you have your content based firewall FWTK, Guantlet, Sidewinder whatever, now all potential malware is mirrored to another system using the same layout as the desktop systems only with a custom auditing system and basically no security. This system automatically executes anything directed to it and if this object tries anything tricky then aflicted desktops are automatically isolated.

An area lacking here is of course time delay malware, but ideally there is enough time to discover these before they detonate and a quick search of IDS logs should indicate what systems need to be cleaned.

This example is very vague but I think it should be useful to explain a simple method of realtime IDS learning.

catch

That is an impressive layout, catch. This is running completely on an AI based IDS???

I see the benefit behind the redundancy in the IDS sensors but I don't fully follow your layout (allowing that this was hastily sketched and is not the real topography). Wouldn't it work more to your advantage to have the malware detector in front of your desktops?

Also, are the IDSs running purely on anomaly and behaviours or is there still some use of a string/rule in there also? for that kind of complexity these systems would have to have some extreme horsepower.

Have you ever tried putting a router in front of the whole thing and have the IDS feed it an ACL update on all traffic falling foul of the rules? What about the problem of neural nets learning to normalize what they see often enough? how have you gotten passed that?

The malware detection system is actually just a Windows 2000 pro system setup as a work alike to the desktops, it however has a few custom kernel drivers that audit for specific behaviors. Anythin deemed as a new executable by the proxying firewall is forked to the malware detection system (which is actually several systems but you get the idea). From here it is automatically executed and its behavior observed, this allows it to 'learn' new evil signatures. these signatures are then fed into the "normal" signature based IDS systems atop the diagram.

The malware detector isn't a filter, and to use it as such would slow down network traffic a standstill at times as it tests new software before passing it on. The idea of the malware detector in this situation is to effectively quarentine worms and the likes by educating IDS systems enterprise-wide after first discovery. This approach grants that a few systems will be infected and takes the aim of detection and isolation rather than prevention as preventing every new attack against COTS desktops is unrealistic to say the least.

No nueral net is used, just a system to detect if new software does stuff it oughtn't but outside the current protection of the software on hand. This struck us as a far more effective learning environment than attempting to utilize more conventional AI ideals.

catch