A look into IDS/Snort part 1 of 3 : LONG
well i have been writing this paper for about 2 weeks now, and i am done with the first part of 3. the format is not good because of the post, so i attached the same paper in pdf format. please feel free to comment about it, thanks.
-------------------------------------------------------------------------------------
A look into IDS/Snort
By: Q.o.D
<[email protected]>
01/01/2004
Introduction:
Over the years IDS has gained popularity amongst organizations, with the rise of security risks, we needed a methods of detecting and possibly stopping intrusions. Last year alone (2002) we were hit with many viruses which could have been avoidable. In this paper we will discuss some of the concepts behind IDS, an infant technology that till recently has saw demand in businesses.
Disclaimer and Copyright:
Copyright © 2004 , 2005 Q.o.D <[email protected]>
This document is free software; you can redistribute and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version.
This document is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
1.1What is an IDS?
Intrusion detection systems(IDS) could be defined as a system that employs process of gathering information (though logs or sniffing) and analyzing that information for possible attempts of intrusion. Throughout this paper Intrusions would be referring to both misuse and intrusions unless otherwise specified. Intrusions are attacks originating from outside of your network, while misuse on the other hand refers to attacks that originate from the inside of you network.
To further clarify this definition think of a burglar alarm or a surveillance system that watches your house when you are on vacation. If you house is robed then you could use logs from the burglar alarm and the video tape from the surveillance camera to identify the robber. An IDS functions in much the same way on your network that constantly looks through the network packets trying to detect an intrusion. Once an intrusion is detected it will take the proper action that you specified (sending an email to the security guy in the network, or just logging the alert). It is important that you understand that just like a surveillance camera, IDS is used for detection and not prevention.
Numerous IDS types are in the market today, although this paper will talk about HIDS, DIDS, and Hybrid IDS our main focus would be NIDS. Please note that IDS is not effective if you lack essential security. For example do not expect an IDS to be effective on a network that uses telnet or FTP for authentication and does not have a firewall in place.
1.2 With firewalls
IDS is just an added layer of protection that substitutes of what firewalls lack, this includes:
1. Reliable logs
Most attackers (or at least smart ones) will clean up after they are done with there system compromise. Implemented effectively, IDS could block attackers from editing the IDS log files or at least present some more difficult challenge for the attacker.
These logs are also important if you later want to prosecute the attacker, remember that logs would be your only evidence that the attack even took place, and the attacker was the one who cracked your system.
2. Detailed logs
A good IDS will provide you with a detailed log and a captured packet of the attack. This might help you fix the problem with your security.
3. Real-time alert
Real-time alerting would notify you when an attack is underway this is definitely important against attacks that depend on speed and how much your system could handle before it crashes, an example of such attack is a Denial of Service(Dos) attack.
4. Detecting prelude(beginnings) of an attack
Since most hackers need to follow a process before attacking a network, beginning mostly with footprinting and network probing (portscaning, vulnerability scanning, etc.). It is quite possible that a smart admin (or a lucky one) could be able to catch an attack before it even happens.
5. Insider threat
While firewalls do a great job in detecting attacks originating from the outside of the network, they could do little to stop or even detect attacks originating from the inside. A recent study done by CSI/FBI 2003 showed that 45% of reported attacks originated from the inside of the network, mainly because they know too much about the systems around them and the daily routine.
6. Possible policy violation
For example some networks prohibits the use of P2P programs, such as Kazaa and GNUTella because the expose your network to many security threats. A good IDS could be configured to detect these kinds of programs on your network and report their use to the network or system administrator.
One thing to keep in mind though, just like you would not put money on the street, monitor it and expect it to be there the following morning, you should not leave your network wide open and vulnerable expecting your IDS to help you. An IDS does not substitute for security they just aid in making it better.
1.3 HIDS
Over the years many types of ID systems have emerged, but only two NIDS and HIDS are widely used today. HIDS (Host based intrusion detection) functions by examining your log files, these logs might include your system logs or application logs, they then search for unexpected system behavior and report them. HIDS is installed on your system you want to monitor and will only detect actions on that host (See figure 1.2). Some of the best examples of HIDS include Honey pot, PortSentry, Tripwire(the most popular HIDS on Linux), and even RPM could be used as an HIDS.
There are two main types of HIDS:
1) Target monitoring HIDS
This type detects changes in some of the important files, like index.html if you are running a website. The target monitoring works by constantly checking the hash of the files (MD5 is the most popular) and comparing it to the previous hash, if any thing is different it logs it and takes the appropriate action. Tripwire is famous for this type of detection.
2) Log monitoring HIDS
Deferent from the target monitoring HIDS is different from target monitoring in many ways. It works by searching the logs for suspicious behavior, for example if user Joe has escalated his privilege to root then an alarm goes on and the HIDS takes the proper action.
Some of the advantages of using HIDS over other types of IDS technologies include:
1) Could cost less than NIDS
HIDS does not require any dedicated hardware.
2) Logs could offer more detail about the attack
HIDS records every action the attacker takes on the system. HIDS like honeypots even log the attackers keystrokes.
3) Produces ess false negatives
HIDS could detect attacks that might appear to be normal to a NIDS.
4) Can Handle encrypted attacks.
5) Detects what happens to your system after the attack
6)Works well with mobile devices such as laptops
While some of the disadvantages include:
1) Only detects attacks after they have occurred.
2) Could be disabled by a talented attacker
if the attacker hacks your system, he might be able to disable or alter the logs so the HIDS would lead in becoming untrustworthy.
3) Produces some CPU overhead, this is troublesome if you need every bit of your CPU processes.
4) HIDS is not available for every OS
Because of the way they work, HIDS tend to be OS specific especially the in the Log monitoring HIDS, so an HIDS that is made for a FreeBSD server cannot be used for Windows 2000 server. So if you have an old or rare OS it might be hard for you to find a good HIDS.
5) HIDS has no knowledge of the system around them
One thing that this is important is when an attacker attacks you from a system on your network that does not have HIDS installed and you have a trust relationship with.
6) Each system that you need to monitor must have HIDS installed on it.
This might lead to higher costs on large networks.
7) If the IDS has no centralized logging capability monitoring it would lead to an administrator's nightmare.
8) HIDS can not log attacks if it is turned off
If an attacker uses a denial of service(DoS) attack against your system and it goes down, you will not know what happens next. And if the attack makes the system go down instantly then the HIDS might not be able to detect anything.
As you might see HIDS has its good and bads, another type of IDS, Network based IDS (NIDS) will be discussed next. Keep in mind that this paper focuses more on NIDS and not HIDS.
1.4 NIDS
The most popular and widely deployed IDS today is the Network based IDS. NIDS serves a different function than the HIDS, it monitors the entire network instead of just on system like a burglar alarm at your house. They do that by analyzing the network packets and not the logs. They receive the packets by sniffing the network(sniffing and how it works is discussed later).
There are two main types of NIDS although they function similarly they offer a different type of service at the end:
1) Network based IDS
This type works by sniffing the network off the network, by putting the network interface card into promiscuous mode. They could easily monitor an entire network, and are the most popular.
2) Network node based IDS
Network node based IDS on the other hand only monitor the host they reside on, similar to HIDS but they analyze the network packets and not the logs. One advantage that this type has over the other is that not all network cards are able to function in promiscuous mode.
Some of the most popular NIDS include Snort, Cisco secure IDS, BlackICE Guard, ISS Realsecure, Dragon, and Shadow. But by far Snort is the most widely used and preferred NIDS, we will discuss Snort later.
Advantages:
1) One NIDS should be enough to monitor a network
This might not be true for a huge network, but it is true for medium and small sized networks. There are two advantages for this: 1) Central logging capability, so you do not want to go from . workstation to workstation to analyze the logs, 2) It is cheaper.
2) Real time detection and alerting
since many hackers follow a pattern to attack your network, you might be able (with some luck) be able to detect and stop the alert in real time.
3) Passive
You could drop a NIDS into a network, without caring about obsolete operating system and it will still detect attack, although i would recommend you tweak the NIDS to stop getting false alarms.
4) Independent system
This could be a disadvantage and an advantage, a disadvantage would be that you need another system, an advantage would be that the NIDS will not consume your precious system resources.
5) Detecting lower level attacks
NIDS function at the network layer and because of that they are able to detect low level attacks such as arp spoofing etc...
Disadvantages:
1) Misses some attacks
Going back to the fact that NIDS operate at the network layer. Some NIDS do not make sense of higher level protocols such as HTTP. They are greatly however challenged by encryption, which will be discussed later in this paper.
2) Single point of failure
If your NIDS goes down because of some sort of a DoS attack then all of your IDS system will fail.
3) Tends to produce higher false alerts than HIDS.
4) Some network cards do not support promiscuous mode.
5) Most current NIDS do not function well under high speed networks such as Gigabyte Ethernet.
As you have seen several limitations of NIDS it would become apparent that you will need a combination of both NIDS and HIDS on your network. This is the primary reason why a new bread of IDS has emerged hybrid based IDS.
1.5 Hybrid IDS
This is a new breed of IDS in what some security people believe is the future. They combine the goods(and some the bads) of both HIDS and Network node IDS. There are some disadvantages though in using hybrid IDS:
1) This is an infant technology and some commercial products could be expensive. There is however a great open source hybrid IDS called Prelude which could be found at www.Prelude-IDS.org . Some of the commercially available hybrid IDS include NFR HIDS, and Stormwatch.
2)They can only monitor one host just like the HIDS.
1.6 LIDS
LIDS (Linux IDS) is a Linux kernel patch that would allow users to limit the power of root, by giving programs the rights they need, with no excess. LIDS currently support both the 2.2 and the 2.4 kernel. The importance of this comes when a program is compromised and gives the attacker right to whatever the owner was, so if it was root then the attacker will have GOD access to the system. If LIDS implemented however and a process running as root is compromised it will limit the attacker to whatever root has access to, for example root might not be able to run ' rm -fr /* '.
From the LIDS FAQ:
"LIDS is an enhancement of Linux kernel written by Xie Huagang and Philippe Biondi. It implements several security features that are not in the Linux kernel natively. Some of these include: mandatory access controls (MAC), a port scan detector, file protection (even from root), and process protection."
One noticeable disadvantage to LIDS is that they are hard to configure, and if implemented wrong they would break the applications.
1.7 DIDS
DIDS (Distributed IDS) compose of multiple HIDS or NIDS sensors reporting to a main system called the management station. This is mostly seen in large organizations with multiple subnets, networks, and offices.
Advantages:
1) Central logging, and management.
2) Makes updating and modifying rules for the systems a "piece of cake".
Disadvantages:
1)Not all NIDS and HIDS support this function, some only support DIDS from the same manufacturer.
1.6 Signature based Detection
Now that we have what IDS and types out of the way we will see how the IDS nows what is an intrusion. The first oldest and widely used method is Signature based detection. Signature based detection works by analyzing each packet against a know attack signature, like an Anti-virus scanner. Signatures are a database of attack signatures, and a signature is a unique characteristic or pattern that must be followed to be successful, by defining how the attack would look like. Signature files include key strings such as ../.. , commands like cmd.exe, or even text such as "login failed".
The IDS follows some basic steps:
1) The logs or packets are captured
2) The logs and packets are normalized if they use a different format that the IDS can't detect.
3) The captured are then are compared byte by byte against every attack signature that you have loaded with your IDS.
4) The action you specify is then taken.
Many of the modern IDSs in the market today use signature based detection, and they are sometimes used in conjunction with protocol based analysis. Protocol based analysis analyzes network packets and detects ones that violate the standards(code red is easily detected by them). In Snort signature files are called rule files, other IDSs use different name, for example SHADOW uses the name filters.
Advantages
1) They are fast.
If implemented correctly, they are able to detect intrusions faster than any other analyzing method.
2) Highly customizable
With some IDSs you will be able to make your own rules and delete the owns you do not know.
3) The signature files are released often
Again this is dependent on your IDS provider. With Snort the signature files are released within a day or even hours after the attack goes into the wild.
Disadvantages:
1) False positives
Signature files tend to be general about the attack, this is due to the fact that if an attacker could change some option or a variant of the attack attacks your network then the signature files will not match and the IDS will not detect it. So using general signature files will detect those attacks but will produce higher false positives(discussed later).
2) False negatives
If an attack has no written signature for your IDS and you did not bother to make one then you have two options. 1) Forget about it and follow the method of "Security by obscurity." or 2) Make an IDS signature for your IDS your self(difficult in some products). The signature based IDS could also be evaded by skillful attackers(discussed later).
3) Do not make sense of encrypted packets
4) Needs updating
If you do not constantly update your signature files then variations or new attacks will not be detected by your IDS.
5) Prone to Denial of Service attacks (discussed later in this paper).
18 Anomaly based detection
Anomaly based detect is different in that it determines what is normal system activity and what is not, by learning it. Although this technology is in its infancy and has not left the research labs, they are still a valid topic to talk about. Commonly referred as AI(Artificial Intelligence) they work not by analyzing network traffic, but rather by defining what is authorized and unauthorized network or host behavior. This is determined during its training phase. The training phase is the time the IDS need to configured to detect normal traffic from abnormal traffic could take weeks if not months, and it usually needs heavy administration. This type of detection is also called behavior, abnormality, or statistical IDS. To get a better picture on how they work i will give you an example:
If Kevin logs in to his computer Monday through Friday from 8:00 A.M. to 3:30 P.M. to the companies FTP server and then one day he logs in on Saturday at 3:00 A.M. then you will know that there is something is heuristic(not normal) and the Anomaly detection should detect it.
Although there are no products that use Anomaly based detection solely there are some IDS that use them, some of the best examples is SHADOW which is open sources and could be found at www.nswc.navy.mil/ISSEC/CID , other product that use this technology are Snort and ISS Real Secure.
Advantages:
1) Could detect never before seen attacks
Because they are based on what is normal behavior and what is not, almost all attacks are heuristic by nature.
2) Difficult to evade
Because they do not use signatures they are difficult to evade.
3) Active research area
Disadvantages:
1) False Positives
Because of the unpredictable nature of your networks and employee's action (maybe Kevin did log on to that FTP server, and not some other guy across the world).
2) False Negatives
Anomaly based detection assumes that every attack starts with scanning, this is not true because an attack could just try the most resent exploit on you and if he is lucky he will be successful, because of that they tends to miss most OS exploits, buffer overflows, and P2P attacks.
3) Hard to configure
Configuring what is normal and what is not requires great knowledge about your network. It is also possible that if you miss-configure your IDS it would result in a great amount of false positives and false negatives.
1.9 Other methods of detection
Other methods of detection that IDs use that are not heavily mentioned in this paper include:
1) Protocol based Anomaly IDS
This method of detection looks for any abnormality in the way the packet was assembled or delivered. They would also watch for packets that do not follow their standards.
2) Audit trails
This method of detection analyzes the system log files for signs of intrusions. It is mostly used on HIDS.
3) Target based
IDSs that use target based detection usually detect intrusions or changes to files. This is done by doing a hash of the files periodically, and checking them to the previous results.
1.10 False Positives
One of the many problems with IDS is that they are prone to many false positives, as a matter of fact almost 90% of all alerts are false positives. False positives is best described as Stefan Axelsson said: "If you call everything with a large red nose a clown, you'll spot all the clowns, but also Santa's reindeer, Rudolph, and Vice versa." This is a great example of what are false positives. They are alerts and logs that classify authorized strange behavior as an attack while in fact it is not.
The reason why false positives are problematic is that they waste precious time and resources. They are administrative intensive and will not affect your network. But because every alert needs to be analyzed this could require many people to do, and these people might miss the real attack.
There are many reasons that cause false positives:
1) Poorly written attack signatures.
Because attack signatures are often written general enough to detect variants of the attack, they also produce a higher false positive rate. As an example, say there is a signature files that detects when a cmd.exe is passed your web server, this would catch many attacks against your web server. But what if a user (Kevin) has a password of cmd.exe, every time Kevin will log in the IDS will generate a false positive.
2) Poorly Configure IDS
Tuning your IDS to work with your network is your best choice against false positives, remember that it might take you weeks if not months just to fine tune your IDS. A couple of guidelines include:
1) Do not put attack signatures that you will not need. Not only that is process intensive, but it is also wastefully. If you do not have an FTP server why would you be afraid of it being attacked, and why would you need the FTP attack signatures?
2) Do not detect hosts that you do not have on your network. If you have 100 computers on your network, then you would only need to monitor those 100 systems, and not the whole subnet.
3) Remove attack signatures that produce too many false positives. Not all attack signatures are created equal, so if you find an attack signature that produces too much false positives, they you could rewrite the attack signature, or just delete it.
1.11 False negatives
False negatives on the other hand are when an IDS does not detect a real attack, and they cause more havoc than false positives. They are usually cause by new exploits(0-day), variants of known attacks, or an none-updated IDS signature file. To defeat them make sure that you secure your network, and also keep your signature files up-to-date.
1.12 Sniffers
Sniffing, which is also called eavesdropping or wiretapping, is similar to the wiretaps that the FBI installs on suspected criminals. It is used by the NIDS to capture all the traffic on your network. They serve two functions:
1) For you:
Used by the NIDS as there source of analysis, without them the NIDS will have nothing to analyze.
2) Against you:
If an attacker installs a sniffer on your network he would be able to see the contents of every packet that is there, and make sense of the unencrypted ones. So he might be able to pickup some passwords that are used in FTP and telnet sessions(that are sent in clear text).
Some of the best sniffers around include TCPDump, Etheral, Snoop, Sniffit, Snort , DSniff, and Ethrape. As you could see there are many sniffers out there, but the best ones have to be TCPDump, Etherreal, DSniff, and Snort.
How sniffers work in a paragraph:
They work by putting your network interface card into promiscuous mode. While normally the NIC card would only accept packets that are sent to its MAC address. In promiscuous mode the NIC will accept all the network packets on the wire. One thing to keep in mind however is that not all NIC cards could go in promiscuous mode.
1.13 IDS placement debate
This is a debatable question, that depends on what is your threat and what is your network like, something I do not know and obviously can not decide for you. Putting your IDS inside or outside of your firewall.
Placing your IDS inside your firewall would make your IDS not being ale to detect attacks on your firewall something that sometimes defeats the purpose of the IDS in the first place. But it will detect attacks from the inside of your network.
Placing your IDS on the outside of your firewall, will make your IDS detect all the attacks targeting your network from the inside, but will not make your IDS detect attacks from the inside of the network. This would also produce higher false positives.
The best option in my opinion is to place your IDS inside an outside your firewall, that way your will get the best of the two. The only disadvantage is that this will you analyze two logs instead of just one.
The best discussion (in my view) on this topic is in the Snort FAQ, here is the section for the Snort FAQ (www.snort.org/docs):
_______________________________________________________________________
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
2.5 Where's a good place to physically put a Snort sensor?
This is going to be heavily influenced by your organizations policy, and what you want to detect. One way of looking at it is determining if you want to place it inside or outside your firewall. Placing an IDS outside of your
firewall will allow you monitor all attacks directed at your network,
regardless of whether or not they are stopped at the firewall. This almost certainly means that the IDS will pick up on more events than an IDS inside the firewall, and hence more logs will be generated. Place an IDS inside your firewall if you are only interested in monitoring traffic that your firewall let pass. If resources permit, it may be best to place one IDS outside and one IDS inside of your firewall. This way you can watch for everything directed at your network, and anything that made it's way in.
ADDENDA AD NAUSEUM
Note: So this one still gets a lot of traffic even though it's in the FAQ. Erek Adams has noted this comprehensive and authoritative discussion of this perpetual discussion item - mildly edited, also see faq question about switches hubs and taps -dr
If your router/switch can do port mirroring then just connecting a network IDS to it would be fine. Else a hub could be another option. Most of network IDS can have a NIC that acts as a passive sniffer anyway.
As to where to place the sensor. I would go for both, one to monitor the
external, one for the internal. I work in a distributor for security products, so over instrumentation is fun :) And in any case, if the traffic do not pass by the Sensor it will not get monitored. So some people deploy IDS on their internal segments too I believe.
In ``front'' of the firewall(s):
Pro: Higher state of alert you know what attacks you are facing.
Con: Wall to Wall of data, boring? If your firewall has NAT turned on, tracking the sources originating from your internal network is difficult.
``Behind'' the firewall(s):
Pro: Only what gets through the firewall gets monitored? Less load on the IDS analyst. You get to see what hosts are sending traffic to the internet.
Con: Less idea of the state of the environment, false sense of safety.
Where should IDS be placed relative to firewalls? Explore the pros and cons off placing IDS inside or outside firewall. What are the drawbacks of each?
* MARCUS RANUM from NFR Security: "I'd put mine inside. Why should I care if someone is attacking the outside of my firewall? I care only if they succeed, which my IDS on the inside would ideally detect. Placing the IDS on the outside is going to quickly lull the administrator into complacency. I used to have a highly instrumented firewall that alerted me whenever someone attacked it. Two weeks later I was deleting its alert messages without reading them. Another important factor arguing for putting it inside is that not all intrusions come from the outside or the firewall. An IDS on the inside might detect new network links appearing, or attackers that got in via another avenue such as a dial-in bank.''
* CURRY from IBM: ``The IDS should be placed where it will be able to see as much of the network traffic you're concerned about as possible. For example, if you're concerned about attacks from the Internet, it makes the most sense to put the IDS outside the firewall. the most sense to put the IDS outside the firewall. This gives it an "unobstructed" view of everything that's coming in. If you put the IDS inside the firewall, then you're not seeing all the traffic the bad guys are sending at you, and this may impact your ability to detect intrusions.''
* SUTTERFIELD from Wheel Group: ``IDS ideally plays an important role both inside and outside a firewall. Outside a firewall, IDS watches legitimate traffic going to public machines such as e-mail and Web servers. More importantly IDS outside a firewall will see traffic that would typically be blocked by a firewall and would remain undetected by an internal system. This is especially important in detecting network sweeping which can be a first indication of attack. External systems will also give you the benefit of monitoring those services that firewalls determine are legitimate. Putting an IDS inside the firewall offers the added benefit of being able to watch traffic internal to the protected network. This adds an important element of protection against insider threats. The major drawback of IDS inside a firewall is that it cannot see a good deal of important traffic coming from untrusted networks and may fail to alert on obvious signals of an impending attack.''
* CHRIS KLAUS from ISS: ``Outside the firewall is almost always a good idea-it protects the DMZ devices from attack and dedicates an additionalprocessor to protecting the internal network. Just inside the firewall is also useful-it detects attempts to exploit the tunnels that exist through the firewall and provides an excellent source of data for how well your firewall is working. Throughout your intranet may be the best place for IDS deployment, however. Everyone agrees that attacks aren't the only things we're worried about-there's internal mischief, fraud, espionage, theft, and general network misuse. Intrusion detection systems are just as effective inside the network as outside, especially if they're unobtrusive and easy to deploy.''
* GENE SPAFFORD: ``The IDS must be inside any firewalls to be able to detect insider abuse and certain kinds of attacks through the firewall. IDS outside the firewall may be useful if you want to monitor attacks on the firewall, and to sample traffic that the firewall doesn't let through However, a true IDS system is likely to be wasted there unless you have some follow-through on what you see.''
* Bottom Line:
DRAGOS RUIU: ``just pick a spot you're likely to look at the logs for :-)''
_______________________________________________________________________
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
You have heard the experts and now you should decide what is best for your network..
In the next section we will be talking about Snort as the IDS of choice, see you then.
Great Information - Thanks
Thanks for the work on the IDS TUT.
Very good summary on IDS's
i didn't understand the part about network cards being in a promiscuous mode. If you can elaborate more, iwill be appreciatable.
