So you can ping the billing and surfing pc's and they can ping others on the network but not access the Internet?
Printable View
So you can ping the billing and surfing pc's and they can ping others on the network but not access the Internet?
When you say you can ping the website are you talking about external webpages?
If so, what I am to understand is only the billing PC's (which have DHCP reserved IP Addresses) cant access the Internet but can view the Intranet with no problems?
Have there been any new GPO's released? Is there any screwy stipulations setup on the DHCP server? Reservations that are odd or anything? Any new updates, patches in the environment that would lead to these issues?
Okay now this problem isn't reserved to billing machines.. I just got a call informing me that machines used by the customer are also affected..
There have been no new gpo's except one change that I've included for wsus.. In last one week all machines have been patched up to current levels.. Currently 70 %. Machines are at 100 % patch level others are following quickly because I've put deadlines on all patches.. Except this no change in the domain is done.. Dhcp servers haven't been touched, there is no ip conflict detected too.. We use sep on our machines and I've seen machines with sep installed and updated showing a green dot indicating machine is protected but the machine in reality is infected.. I've even checked these machines for infection or rootkit but ive found nothing.. We are loosing more machines by the hour now and it's really scary now.. Sigh..
Thank you very much to everyone who has replied..
At this point (and I'm still not sure I understand the problem) I would say you are in a DoS situation and most likely it's a bad switch or router.
Check the ARP tables of an affected machine and then the arp table on the first switch it's attached to.
Do you have SNMP Traps set on all your routers and switches? Do any mac filtering?
I've seen switches just dump and rebuild the spanning tree for no apparent reason every few seconds never completely rebuilding.
Really I would focus (again if I understand your situation) on layer 2 and 3.
Hope this helps
Just a quick thought Are all the PC's affected at 100% patch level and non of the PC's affected at 70%.
Or vice-versa?
When did you put the deadline on all patches? Before this mess started or after?
Affected machines also include completely patched machines.
I have checked ARP table of few machines but of these (about 8) only one had malicious entry pointing to another infected machine in the same VLAN. Other have only one entry pointing to local gateway (switch)Quote:
Check the ARP tables of an affected machine and then the arp table on the first switch it's attached to.
NopesQuote:
Do you have SNMP Traps set on all your routers and switches? Do any mac filtering?
Umm I didn’t see any of the switches do that except one that sent lot of ARP traffic then just stopped.Quote:
I've seen switches just dump and rebuild the spanning tree for no apparent reason every few seconds never completely rebuilding.
I am focusing on an intrusion because these branches are spread all across the country. Every branch has 2 switches ( overall 250+ branches and about 20 of them are having this issue). I really don’t think so many switches at the same time would start acting crazy, especially without ANY change being implemented.Quote:
Really I would focus (again if I understand your situation) on layer 2 and 3.
This is a long shot but there isnt some form of replication happening between all these affected machines are there? I have seen this happen when DFS or something similar was activated and someone accidentally dropped a 1gb file into the shared folder...
Have you had a chance to pull anything new from any of your logs since your original posts?
No no.. no replication at all.. None of the machine communicate to each other for anything..
I've not found anything new post begining of this activity.. If you want me to run a sniffer again or collect some data please let me know
Thank you.
Hello Bytewrangler,
Havent been around in a while but another thing you can check that is if you haven't already is do a traceroute towards your web server with both a machine that works and a machine which doesn't. this will allow you to check if they follow the same path. Another thing i didn't see it mentioned but you dont have a wins server on your network do you? I have seen some strange problems coming from WINS but it would be unlikely to affect your network switches.
I would also check the web servers to see if there is a firewall or filtering on that side if you can.
What has me confused is you stated that when you change the IP addresses of the machines (still within the same VLAN) but outside the DHCP reservation the issues go away?
Do the issues stay away or do they replicate again after X amount of time?
If they don't is there any change you somehow have any ACL's that got setup on your switches or elsewhere that could be affecting them?