IIS Site Cache Filled. Why?

[aberration]

I have two web servers (WinNT, IIS 4.0) running many web sites, but 3 sites in particular have had unexplainable outages over the past 3-4 days (2 sites on one box, 1 on the other). Due to the configuration of the web sites, there are 30-40 IP addresses per server (very common and easy to set up). A couple of days ago, the web support team received pages that sites were not responding to port 80 requests (via a monitoring tool). A review of the firewall logs showed syn's going to the web site and the web server sending resets immediately with no bytes transferred (a.k.a., the site is too busy to handle any new requests).

When performing a netstat on the servers, approximately 1200 open connections could be seen running; however, when checking the firewall stats, the highest number of connections during the site outage times was approximately 1020 connections. When the web site service was cycled, the number of connections dropped to less than 50 connections and the site began to take requests once again. All of the other sites on these servers were able to handle port 80 requests without a problem and during the outage, the sites affected responded to icmp echo requests.

It appears that the web site, in conjunction with the stack, is not releasing connections and the sockets remain full.

My question is what is causing this condition and how can it be fixed?

Any advice would be greatly appreciated.

[AngryBob]

sounds like you got DoS...are the syns coming from the same IP or the same class? that could be a tip that someones doing that to you. put the log on here if you can so we can peek at it.

[aberration]

It's not a DoS, since the number of connections on the firewall remained relatively low. The highest number of concurrent connections for the entire DMZ (~150 servers, 400+ sites) was only around 1,000 during the times when we observed the web servers rejecting connections. The IDS didn't pick up on anything either and it's pretty well tuned to watch for DoS attacks.

I am convinced that the cache on the web server filled up and did not release old connections, but there is not a reason why it shouldn't release them. Is there a connection timeout variable that one can set within NT or IIS?

Again, any thoughts or ideas would be appreciated...

[DBEAUCHAMP]

It could also be SYN-Attack, there is somes registry keys you can add to your TCP/IP settings: You could first check those listed in this Technet or MS Knowledge base article: [glowpurple]HOW TO: Harden the TCP/IP Stack in Windows 2000 Against Attacks [Q315669][/glowpurple] it's kind of the same in NT 4.0

I've also been experiencing the same kind of problem causing all the ASP portion of my site to go numd.

It's been a problem that a lot of people have been having on NT with IIS 4.0. I had a MS open ticket for a while without getting finding the problem.

May I suggest migrating to W2k with IIS 5.0 !

Good luck !

[aberration]

Thanks for the suggestions, dbeauchamp. I looked into the stack tuning articles at MS and they didn't help me hone in on the problem. We have opened a ticket with MS as well and they have given us some steps to capture the issue mid-stream. Although, now that we're watching it--it probably won't recur...

All of the web servers should be migrated to W2K and IIS 5.0 shortly (I'm in the security group, the engineering and support groups do all the planning for migrations...).

Thanks for all of your help everyone, it is appreciated.

...aberration...

[DBEAUCHAMP]

I'm glad your problem as been solve or did not occur again ! Let me know if you ever find the problem and solution for it !

By the way, you could install the Microsoft debugger in case it happens again ! MS will be able to get more info about what went wrong !

Good luck with MS !

[aberration]

The problem is still intermittantly occurring, but there is no rhyme or reason to it... Only one site on the server is affected, while others on the server function normally--it's just odd.

I have taken the liberty of installing the debugger on a few of the systems to see if I can capture some kernal processes or stack exchanges.

Again, thanks for your help!

...aberration...