Information Overload!

[KorpDeath]

If you look around AO you'll see HOWTO's on IDS's and firewalls, tarpits, and honeynets. But I think there is something very integral that is missing here. So here's a little story.

So I come in today to Gigs of syslog messages, the IDS event log messages, firewall messages, tarpit messages, etc. etc. etc. The list can go on and on.

My new task is to get all of these seperate logging mechanisms together so i can correlate the information into a digestible form. In other words I have information overload. See here.

I know of people who do it manually, some try to automate the tasks with scripts, and still others that regularly ignore it all. How do you handle the plethora of security information at your work? Any and all reccomendations are appreciated.

[RoadClosed]

This field is very interesting, there is a lot of buzz around security even management. I have made attempts at using Cisco Secure policy Manager to try and keep up with some of it. But I am hopeful that future products can integrate all devices and provide a crisp snapshot of the overal posture of a network.

[KorpDeath]

I've been through all of the Openview demos, VitalSuite demos, you name it I've seen it. You can spend millions on this and still not solve any of the issues you have. There's got to be a way to inexpensively crunch all these logs and have some semblance of accuracy at the end.

Anyone?

[RoadClosed]

I am always looking for freebies, 2 I have downloaded for Cisco Routers and Switches. Alas, I have not taken the time to test. They give visualizing tools to monitor peremiter traffic. It's not inclusive but maybe a start.

Cflowd and flowscan - They are from the Cooperative Associaton of Internet Data Analysis, also a starting point. That's all I can say for now, it's on my list but just setting there at the moment. But eventually I have got to get a better snapshot of my network, for peace of mind.

links:

www.caida.org
www.ciada.org/tools

These things look very promising, and please anyone who has the time at the moment. Comment on these tools and give us some info. Thanks.

[Tiger Shark]

Ok Korp..... Now the little nuisance is gone........

Rider:- I am a windows only shop so this is a bit more complicated than it would be in a *nix/syslog compatible system.

Run Kiwi Syslog Daemon, (available here ) as a service on a secure server. Mine is a log only server. It is not even a member of the domain and stands alone doing nothing but logging and occasionally, (due to it's placement), it is used to ethereal something of interest.

For the snort logs make sure the snort.conf file has the output to alert syslog with the host being the log server - that takes care of the IDS's

For the firewall, (assuming it's syslog compatible), send it's logs to the syslog server too. If not try something creative like having it send SNMP or whatever to an non-existent server and have snort log it rather than alert on it then snort will send the info to syslog - that should work.

For Win4.0/2k boxes use Snare, (available here ) to capture all the event log traffic and send it to the syslog server.

For IIS logs use Backlog IIS, (available here) to send the IIS logs, (note: they must be logging to %system root%\system32\logs), to send the logs to the syslog server.

That gets me a single log file of about 40Mb/day. I have the logs roll over daily and a series of scripts to archive them away, (read "squirrel"), on other hard to find places. The log is pretty much chronological as events are detected/transmitted so I get a nice picture of everything.

I use a text file line stripper, (available here ), to search for strings that interest me, (such as an IP address), and dump them to a separate file.... It's a marvellous tool I have posted about before here. It has a command line function that allows you to do any of it's functions from a script so I got busy and wrote a script to extract all the things that interest me from the previous days file and dump them to individual files. The script writes a report and saves it and emails me a copy of the report too. I have this scheduled to run each morning just after my "squirrelling" has taken place. When I arrive at work I have a report like this:-

Security Logs Analysis for 6/24/2003 at 12:30:04 AM
****************************************************************************


File being analyzed: 2003-06-23.txt. Size 43348442 bytes.
====================================


2003-06-23-Snort.txt 120167 Data Recorded 12:30:14 AM
2003-06-23-Alerts.txt 92541 Data Recorded 12:30:14 AM
2003-06-23-Stealth.txt Zero length Deleted 12:30:19 AM
2003-06-23-Portscan.txt 3326 Data Recorded 12:30:25 AM
2003-06-23-IPv6.txt Zero length Deleted 12:30:30 AM
2003-06-23-Blocked.txt 210250 Data Recorded 12:30:35 AM
2003-06-23-DenyIn.txt 464057 Data Recorded 12:30:41 AM
2003-06-23-DenyOut.txt 121976 Data Recorded 12:30:46 AM
2003-06-23-ICMP.txt 209456 Data Recorded 12:30:52 AM
2003-06-23-FortFirewall.txt 67347 Data Recorded 12:30:57 AM
2003-06-23-IISLogs.txt 1301863 Data Recorded 12:31:02 AM
2003-06-23-IIS404.txt 170586 Data Recorded 12:31:08 AM
2003-06-23-IIS403.txt 1199 Data Recorded 12:31:14 AM
2003-06-23-VPN.txt 32369 Data Recorded 12:31:19 AM
2003-06-23-VPNBadAuth.txt Zero length Deleted 12:31:24 AM
2003-06-23-VPN_SYN.txt Zero length Deleted 12:31:30 AM
2003-06-23-TermServ.txt 1364 Data Recorded 12:31:35 AM
2003-06-23-SSL.txt 123831 Data Recorded 12:31:41 AM
2003-06-23-Lockouts.txt Zero length Deleted 12:31:46 AM
2003-06-23-XXXMain.txt 5542 Data Recorded 12:31:51 AM
2003-06-23-NS2.txt Zero length Deleted 12:31:57 AM
2003-06-23-MAIL.txt 33745 Data Recorded 12:32:02 AM
2003-06-23-XXXPC.txt Zero length Deleted 12:32:08 AM
2003-06-23-XXXBU.txt Zero length Deleted 12:32:13 AM
2003-06-23-CANFPC.txt 487980 Data Recorded 12:32:18 AM
2003-06-23-FORTPC.txt 24648 Data Recorded 12:32:24 AM
2003-06-23-FORTBU.txt 7064 Data Recorded 12:32:29 AM
2003-06-23-XXX-ADMIN.txt 5088 Data Recorded 12:32:35 AM


Analysis Complete at 12:32:35 AM
______________________________________________________________

Begin Archive at 12:32:35 AM

Log Archived to server at 12:33:19 AM
2003-06-23-Alerts.txtmoved at 12:33:20 AM
2003-06-23-Blocked.txtmoved at 12:33:20 AM
2003-06-23-CANFPC.txtmoved at 12:33:20 AM
2003-06-23-XXX-ADMIN.txtmoved at 12:33:20 AM
2003-06-23-DenyIn.txtmoved at 12:33:20 AM
2003-06-23-DenyOut.txtmoved at 12:33:20 AM
2003-06-23-FORTBU.txtmoved at 12:33:20 AM
2003-06-23-FortFirewall.txtmoved at 12:33:20 AM
2003-06-23-FORTPC.txtmoved at 12:33:20 AM
2003-06-23-ICMP.txtmoved at 12:33:20 AM
2003-06-23-IIS403.txtmoved at 12:33:20 AM
2003-06-23-IIS404.txtmoved at 12:33:20 AM
2003-06-23-IISLogs.txtmoved at 12:33:21 AM
2003-06-23-XXXMain.txtmoved at 12:33:21 AM
2003-06-23-MAIL.txtmoved at 12:33:21 AM
2003-06-23-Portscan.txtmoved at 12:33:21 AM
2003-06-23-Snort.txtmoved at 12:33:21 AM
2003-06-23-SSL.txtmoved at 12:33:21 AM
2003-06-23-TermServ.txtmoved at 12:33:21 AM
2003-06-23-VPN.txtmoved at 12:33:21 AM

Email generated at 12:33:21 AM

Report moved at 12:33:27 AM

As you can see most of my work is done by 12:35am......... You will note that the script deletes files of zero length and notes that so if someone deleted files to hide their tracks I can see that the file was not zero length and go back to my "secret" copies and regenerate the whole thing.

I can "sanitize" the script and put a copy up here if anyone is interested. You'd have to edit it for yourselves but it'll give you an idea.

[RoadClosed]

Hey Man, if you would take the time to sanitize it and post, I'll buy you a big ice cold beer.

For Win4.0/2k boxes use Snare, (available here ) to capture all the event log traffic and send it to the syslog server.

I am assuming that needs a client installed on each box? I have something I think is similar called Aida32 (www.aida32.hu/aida32.php)??

Oooh that's real time capturing. Any real life data on over head for the puppy? My outying remote sites are slow on the WAN.

Dude: I have been wondering the same thing as Korp and you just opened the flood gate of info. for me! I have been trying the pay as you go route.

[KorpDeath]

Very nice, TigerShark. And yeah I'm running most of what you have listed except for the text file stripper and Snare. I'll have to check those out.

Anyone else have some tidbits to share? I know there is more than one of you doing this very thing.

[souleman]

Gigs of information? either your not checking it enough or you have got some serious problems, since it is all text files.

Personally I set up scripts to alert me to any known problems, and had it email me (ie netbus attack) and then once a week I would go through the logs looking for anything suspicious. Then again, I only had 50 clients and 60 users. So it wasn't that bad going through it manually, and I was only running a firewall and IDS. And I only checked external connections. I wasn't to worried about internal connections because once a month I would go through and see what was running on each machine. But then the company I worked for had pretty crappy policies so...

[Tiger Shark]

RoadClosed: I'll write whole thing up as a Tutorial and post it. Actually, I'll write the docs for my system which I have been meaning to do for a while, sanitize that and post it as a Tut..... Overhead is actually minimal but I'll document that for you as part of the Tut.

Korp: You'll love them as I do......

Souleman: You are incorrect in believing that they are all necessarily text files. There are some firewalls, (mine included), that have a proprietary, encrypted logging system. Until I began using the system I do now I was generating around 4G of firewall logs per day and I wasn't really logging $h1T, (which in itself bothered the crap outta me). That's when I went off in search of the "Wholly Grail" and came up with my current system. Now I log everything in and out of the firewall and with everything else included I average around 40M a day. I would still say that 40 meg is too much to check properly on a daily basis, (too easy to miss something on those days you are in a hurry), so I wrote my script to show me the "fun" stuff. The reports show me the file sizes so once you know the patterns you will see if something went screwy in one area or another. They alert me to things that may trigger a search for something else across the entire log file for the previous day _or_ even better there is a tool that will take all the daily log files I place in a given dir and join them all together so I can search for the entire activity of a single IP, (for example) across any given period.

[Maestr0]

Nice stuff Tiger I'd like to see your script as well. I do much the same except use Kiwi to log to ODBC database(Snort will also do this) instead of text, anyone know if there are any generic packages with some useful quries/report packages for analyzing syslog stuff in a DB?(If not maybe I should write some ) I'd much rather use SQL than plod through all those text files (good for analysis too). I think this is definately a good topic to kick around, I know I could use a quality management system for all this crap and it looks like I'm not the only one

-Maestr0