SecureIIS Irony

As background information, the SecureIIS Application Firewall (produced by eEye Digital Security, http://www.eeye.com/html/Products/SecureIIS/index.html) is designed to inspect all incoming requests prior to passing them onto IIS (and vice versa). Any request violating the rules put in place by the administrator is dropped and a SecureIIS banner is displayed to the user to let them know their request was dropped. In essence, SecureIIS guards layers 4-7 of the 7-layer OSI model from predefined, potentially malicious requests.

While acceptance testing SecureIIS for our web hosting environment, an interesting product "feature" was discovered in which SecureIIS itself acts as a slow, but sure denial of service mechanism. It appears to be a buffer overflow issue between the SecureIIS ISAPI filter and MS Windows, but since I cannot view the SecureIIS source code, I cannot be sure.

Here are the details regarding the test and the conclusion reached:
Many hosting facilities have load-balancing appliances within their environment to help ensure web servers respond in a timely manner, scalability, availability, etc. Load-balancers must continuously poll the web server hosts to ensure they are "alive" and able to handle requests--this can be performed through ICMP (ping) and http GET/ requests. It is usually best to incorporate both measures as it is possible for the web service within IIS to crash while the host (and OS) remain stable. Since the load-balancer is an appliance, it does not have an http browser defined when making the http GET/ requests to the web server (e.g., Mozilla-compatible). As a result, the load-balancer makes the request to the web server through a telnet session over port 80.

SecureIIS was installed on a WinNT, IIS 4.0 and a W2K, IIS 5.0 server (all patches as up to date as possible) and placed within a load-balanced environment. The servers were monitored for CPU, disk usage, and many other perfmon selections. Over the course of 24 hours, both servers ASR'd (automatic shutdown & reboot). Further investigation into the perfmon logs revealed that the OS thread handle count would continue to rise when SecureIIS was installed and armed on the servers. A sniffer was placed on the wire to capture the traffic traversing between various components of the environment and the only traffic captured was the health checking traffic from the load-balancer to the web server and a response from the web server to the load-balancer. To emulate this traffic, a telnet session was opened and an http GET/ request was sent over port 80 to validate the health of the web server. The result was that SecureIIS was blocking the request from reaching the web service (as expected). However, each time a non-browser (port-based) http GET/ request was sent to the server, the inetinfo.exe service continued to consume more system resources and, based on the perfmon data, the thread handle count would rise nearly commensurate with the number of non-browser http GET/ requests made to the server.

Over time, the thread handle counts rise to approximately 70,000 and the server ASRs (normally, MS thread handle counts are between 7,500 and 15,000, depending on server utilization, etc.). When SecureIIS is disarmed, the server thread handle counts remain stable at around 10,000 OR if SecureIIS is armed and the load-balancer health checking is disabled, the thread handle counts remain stable...

The conclusion: The SecureIIS ISAPI filter does not release consumed thread handles after rejecting connection attempts by the load-balancing appliance (or any other port-based/non-browser-based communication attempt via port 80). This is indicative of a memory leak within the application or a need to adjust the ISAPI filter to release used thread handles. An application modification needs to be made to allow SecureIIS to function in the hosting environment. Over time, the application firewall is acting as a denial of service agent on the web server and eventually prevents the web server from functioning.

I thought it to be ironic that the security software designed to protect web servers against outages is, itself, a denial of service tool.

In closing, don't get me wrong, the SecureIIS product is an incredible product with a vast array of capabilities. I would strongly recommend its use given the state of MS web server security (if feasible, given the previous issue).

If anyone else has noted an issue similar, please chime in...

...aberration...

If you've done the testing yourself, impressive work! If however, you're just posting someone else's test results, please post the source, it's considered good manners.
Cheers,
cgkanchi

Have no fears... this is my work. It took about a week to work through component isolation and why the components were behaving the way they were. As a note, I have notified the vendor and I am awaiting a response from their tech support team.