Information Overload!
Page 1 of 2 12 LastLast
Results 1 to 10 of 16

Thread: Information Overload!

  1. #1
    Priapistic Monk KorpDeath's Avatar
    Join Date
    Dec 2001
    Posts
    2,628

    Information Overload!

    If you look around AO you'll see HOWTO's on IDS's and firewalls, tarpits, and honeynets. But I think there is something very integral that is missing here. So here's a little story.

    So I come in today to Gigs of syslog messages, the IDS event log messages, firewall messages, tarpit messages, etc. etc. etc. The list can go on and on.

    My new task is to get all of these seperate logging mechanisms together so i can correlate the information into a digestible form. In other words I have information overload. See here.

    I know of people who do it manually, some try to automate the tasks with scripts, and still others that regularly ignore it all. How do you handle the plethora of security information at your work? Any and all reccomendations are appreciated.
    Mankind have a great aversion to intellectual labor; but even supposing knowledge to be easily attainable, more people would be content to be ignorant than would take even a little trouble to acquire it.
    - Samuel Johnson

  2. #2
    Senior Member RoadClosed's Avatar
    Join Date
    Jun 2003
    Posts
    3,834
    This field is very interesting, there is a lot of buzz around security even management. I have made attempts at using Cisco Secure policy Manager to try and keep up with some of it. But I am hopeful that future products can integrate all devices and provide a crisp snapshot of the overal posture of a network.
    West of House
    You are standing in an open field west of a white house, with a boarded front door.
    There is a small mailbox here.

  3. #3
    Priapistic Monk KorpDeath's Avatar
    Join Date
    Dec 2001
    Posts
    2,628
    I've been through all of the Openview demos, VitalSuite demos, you name it I've seen it. You can spend millions on this and still not solve any of the issues you have. There's got to be a way to inexpensively crunch all these logs and have some semblance of accuracy at the end.

    Anyone?
    Mankind have a great aversion to intellectual labor; but even supposing knowledge to be easily attainable, more people would be content to be ignorant than would take even a little trouble to acquire it.
    - Samuel Johnson

  4. #4
    Senior Member RoadClosed's Avatar
    Join Date
    Jun 2003
    Posts
    3,834

    Freebies

    I am always looking for freebies, 2 I have downloaded for Cisco Routers and Switches. Alas, I have not taken the time to test. They give visualizing tools to monitor peremiter traffic. It's not inclusive but maybe a start.

    Cflowd and flowscan - They are from the Cooperative Associaton of Internet Data Analysis, also a starting point. That's all I can say for now, it's on my list but just setting there at the moment. But eventually I have got to get a better snapshot of my network, for peace of mind.

    links:

    www.caida.org
    www.ciada.org/tools

    These things look very promising, and please anyone who has the time at the moment. Comment on these tools and give us some info. Thanks.
    West of House
    You are standing in an open field west of a white house, with a boarded front door.
    There is a small mailbox here.

  5. #5
    AO Ancient: Team Leader
    Join Date
    Oct 2002
    Posts
    5,197
    Ok Korp..... Now the little nuisance is gone........

    Rider:- I am a windows only shop so this is a bit more complicated than it would be in a *nix/syslog compatible system.

    Run Kiwi Syslog Daemon, (available here ) as a service on a secure server. Mine is a log only server. It is not even a member of the domain and stands alone doing nothing but logging and occasionally, (due to it's placement), it is used to ethereal something of interest.

    For the snort logs make sure the snort.conf file has the output to alert syslog with the host being the log server - that takes care of the IDS's

    For the firewall, (assuming it's syslog compatible), send it's logs to the syslog server too. If not try something creative like having it send SNMP or whatever to an non-existent server and have snort log it rather than alert on it then snort will send the info to syslog - that should work.

    For Win4.0/2k boxes use Snare, (available here ) to capture all the event log traffic and send it to the syslog server.

    For IIS logs use Backlog IIS, (available here) to send the IIS logs, (note: they must be logging to %system root%\system32\logs), to send the logs to the syslog server.

    That gets me a single log file of about 40Mb/day. I have the logs roll over daily and a series of scripts to archive them away, (read "squirrel"), on other hard to find places. The log is pretty much chronological as events are detected/transmitted so I get a nice picture of everything.

    I use a text file line stripper, (available here ), to search for strings that interest me, (such as an IP address), and dump them to a separate file.... It's a marvellous tool I have posted about before here. It has a command line function that allows you to do any of it's functions from a script so I got busy and wrote a script to extract all the things that interest me from the previous days file and dump them to individual files. The script writes a report and saves it and emails me a copy of the report too. I have this scheduled to run each morning just after my "squirrelling" has taken place. When I arrive at work I have a report like this:-

    Security Logs Analysis for 6/24/2003 at 12:30:04 AM
    ****************************************************************************


    File being analyzed: 2003-06-23.txt. Size 43348442 bytes.
    ====================================


    2003-06-23-Snort.txt 120167 Data Recorded 12:30:14 AM
    2003-06-23-Alerts.txt 92541 Data Recorded 12:30:14 AM
    2003-06-23-Stealth.txt Zero length Deleted 12:30:19 AM
    2003-06-23-Portscan.txt 3326 Data Recorded 12:30:25 AM
    2003-06-23-IPv6.txt Zero length Deleted 12:30:30 AM
    2003-06-23-Blocked.txt 210250 Data Recorded 12:30:35 AM
    2003-06-23-DenyIn.txt 464057 Data Recorded 12:30:41 AM
    2003-06-23-DenyOut.txt 121976 Data Recorded 12:30:46 AM
    2003-06-23-ICMP.txt 209456 Data Recorded 12:30:52 AM
    2003-06-23-FortFirewall.txt 67347 Data Recorded 12:30:57 AM
    2003-06-23-IISLogs.txt 1301863 Data Recorded 12:31:02 AM
    2003-06-23-IIS404.txt 170586 Data Recorded 12:31:08 AM
    2003-06-23-IIS403.txt 1199 Data Recorded 12:31:14 AM
    2003-06-23-VPN.txt 32369 Data Recorded 12:31:19 AM
    2003-06-23-VPNBadAuth.txt Zero length Deleted 12:31:24 AM
    2003-06-23-VPN_SYN.txt Zero length Deleted 12:31:30 AM
    2003-06-23-TermServ.txt 1364 Data Recorded 12:31:35 AM
    2003-06-23-SSL.txt 123831 Data Recorded 12:31:41 AM
    2003-06-23-Lockouts.txt Zero length Deleted 12:31:46 AM
    2003-06-23-XXXMain.txt 5542 Data Recorded 12:31:51 AM
    2003-06-23-NS2.txt Zero length Deleted 12:31:57 AM
    2003-06-23-MAIL.txt 33745 Data Recorded 12:32:02 AM
    2003-06-23-XXXPC.txt Zero length Deleted 12:32:08 AM
    2003-06-23-XXXBU.txt Zero length Deleted 12:32:13 AM
    2003-06-23-CANFPC.txt 487980 Data Recorded 12:32:18 AM
    2003-06-23-FORTPC.txt 24648 Data Recorded 12:32:24 AM
    2003-06-23-FORTBU.txt 7064 Data Recorded 12:32:29 AM
    2003-06-23-XXX-ADMIN.txt 5088 Data Recorded 12:32:35 AM


    Analysis Complete at 12:32:35 AM
    ______________________________________________________________

    Begin Archive at 12:32:35 AM

    Log Archived to server at 12:33:19 AM
    2003-06-23-Alerts.txtmoved at 12:33:20 AM
    2003-06-23-Blocked.txtmoved at 12:33:20 AM
    2003-06-23-CANFPC.txtmoved at 12:33:20 AM
    2003-06-23-XXX-ADMIN.txtmoved at 12:33:20 AM
    2003-06-23-DenyIn.txtmoved at 12:33:20 AM
    2003-06-23-DenyOut.txtmoved at 12:33:20 AM
    2003-06-23-FORTBU.txtmoved at 12:33:20 AM
    2003-06-23-FortFirewall.txtmoved at 12:33:20 AM
    2003-06-23-FORTPC.txtmoved at 12:33:20 AM
    2003-06-23-ICMP.txtmoved at 12:33:20 AM
    2003-06-23-IIS403.txtmoved at 12:33:20 AM
    2003-06-23-IIS404.txtmoved at 12:33:20 AM
    2003-06-23-IISLogs.txtmoved at 12:33:21 AM
    2003-06-23-XXXMain.txtmoved at 12:33:21 AM
    2003-06-23-MAIL.txtmoved at 12:33:21 AM
    2003-06-23-Portscan.txtmoved at 12:33:21 AM
    2003-06-23-Snort.txtmoved at 12:33:21 AM
    2003-06-23-SSL.txtmoved at 12:33:21 AM
    2003-06-23-TermServ.txtmoved at 12:33:21 AM
    2003-06-23-VPN.txtmoved at 12:33:21 AM

    Email generated at 12:33:21 AM

    Report moved at 12:33:27 AM
    As you can see most of my work is done by 12:35am......... You will note that the script deletes files of zero length and notes that so if someone deleted files to hide their tracks I can see that the file was not zero length and go back to my "secret" copies and regenerate the whole thing.

    I can "sanitize" the script and put a copy up here if anyone is interested. You'd have to edit it for yourselves but it'll give you an idea.
    Don\'t SYN us.... We\'ll SYN you.....
    \"A nation that draws too broad a difference between its scholars and its warriors will have its thinking done by cowards, and its fighting done by fools.\" - Thucydides

  6. #6
    Senior Member RoadClosed's Avatar
    Join Date
    Jun 2003
    Posts
    3,834

    Script

    Hey Man, if you would take the time to sanitize it and post, I'll buy you a big ice cold beer.

    For Win4.0/2k boxes use Snare, (available here ) to capture all the event log traffic and send it to the syslog server.
    I am assuming that needs a client installed on each box? I have something I think is similar called Aida32 (www.aida32.hu/aida32.php)??

    Oooh that's real time capturing. Any real life data on over head for the puppy? My outying remote sites are slow on the WAN.

    Dude: I have been wondering the same thing as Korp and you just opened the flood gate of info. for me! I have been trying the pay as you go route.
    West of House
    You are standing in an open field west of a white house, with a boarded front door.
    There is a small mailbox here.

  7. #7
    Priapistic Monk KorpDeath's Avatar
    Join Date
    Dec 2001
    Posts
    2,628
    Very nice, TigerShark. And yeah I'm running most of what you have listed except for the text file stripper and Snare. I'll have to check those out.

    Anyone else have some tidbits to share? I know there is more than one of you doing this very thing.
    Mankind have a great aversion to intellectual labor; but even supposing knowledge to be easily attainable, more people would be content to be ignorant than would take even a little trouble to acquire it.
    - Samuel Johnson

  8. #8
    AntiOnline Senior Member souleman's Avatar
    Join Date
    Oct 2001
    Location
    Flint, MI
    Posts
    2,884
    Gigs of information? either your not checking it enough or you have got some serious problems, since it is all text files.

    Personally I set up scripts to alert me to any known problems, and had it email me (ie netbus attack) and then once a week I would go through the logs looking for anything suspicious. Then again, I only had 50 clients and 60 users. So it wasn't that bad going through it manually, and I was only running a firewall and IDS. And I only checked external connections. I wasn't to worried about internal connections because once a month I would go through and see what was running on each machine. But then the company I worked for had pretty crappy policies so...
    \"Ignorance is bliss....
    but only for your enemy\"
    -- souleman

  9. #9
    AO Ancient: Team Leader
    Join Date
    Oct 2002
    Posts
    5,197
    RoadClosed: I'll write whole thing up as a Tutorial and post it. Actually, I'll write the docs for my system which I have been meaning to do for a while, sanitize that and post it as a Tut..... Overhead is actually minimal but I'll document that for you as part of the Tut.

    Korp: You'll love them as I do......

    Souleman: You are incorrect in believing that they are all necessarily text files. There are some firewalls, (mine included), that have a proprietary, encrypted logging system. Until I began using the system I do now I was generating around 4G of firewall logs per day and I wasn't really logging $h1T, (which in itself bothered the crap outta me). That's when I went off in search of the "Wholly Grail" and came up with my current system. Now I log everything in and out of the firewall and with everything else included I average around 40M a day. I would still say that 40 meg is too much to check properly on a daily basis, (too easy to miss something on those days you are in a hurry), so I wrote my script to show me the "fun" stuff. The reports show me the file sizes so once you know the patterns you will see if something went screwy in one area or another. They alert me to things that may trigger a search for something else across the entire log file for the previous day _or_ even better there is a tool that will take all the daily log files I place in a given dir and join them all together so I can search for the entire activity of a single IP, (for example) across any given period.
    Don\'t SYN us.... We\'ll SYN you.....
    \"A nation that draws too broad a difference between its scholars and its warriors will have its thinking done by cowards, and its fighting done by fools.\" - Thucydides

  10. #10
    Senior Member Maestr0's Avatar
    Join Date
    May 2003
    Posts
    604
    Nice stuff Tiger I'd like to see your script as well. I do much the same except use Kiwi to log to ODBC database(Snort will also do this) instead of text, anyone know if there are any generic packages with some useful quries/report packages for analyzing syslog stuff in a DB?(If not maybe I should write some ) I'd much rather use SQL than plod through all those text files (good for analysis too). I think this is definately a good topic to kick around, I know I could use a quality management system for all this crap and it looks like I'm not the only one

    -Maestr0
    \"If computers are to become smart enough to design their own successors, initiating a process that will lead to God-like omniscience after a number of ever swifter passages from one generation of computers to the next, someone is going to have to write the software that gets the process going, and humans have given absolutely no evidence of being able to write such software.\" -Jaron Lanier

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •