Page 1 of 3 123 LastLast
Results 1 to 10 of 22

Thread: Loss of network connectivity.

  1. #1

    Loss of network connectivity. (Weird problem)

    Here is the scenario:
    We have 5 AD's housing 5000 machine separated geographically (branches in various cities). Every branch has 1 (max 2) machine(s) which have billing enabled on them. These are the only machine which can be used to give prints and transfer data to removable media. These machines usually run on power user privileges. We use DHCP with MAC address binding for all machines. Except the billing machines others run on user rights (restricted to an extent). All machines run XP. All of them have an AV – updated. None of them are infected (at least most of them). Firewall is OFF on all machines because we have a HIPS package (AV, Firewall, Proactive and this and that!)

    From last few days these machines (THE BILLING ONE'S) have been "mysteriously" going off the network and here is what happens:

    I can ping the website.
    I can trace to the website.
    I can access other machines on the network.

    But somehow they just can’t access any WEBSITE (internet and intranet), through IE; it just dies out.

    I have flushed dns cache and still then pinged a website successfully. So at least DNS isn’t the problem. I have stopped dnschache service and pinged and it works. I have checked firewall / AV logs and found ABSOLUTELY NOTHING (sort of surprising). I have stopped the HIPS package to see if it is the cause of the problem but still it doesn’t work!

    Windows logs (all – app, system, IE, security) show up nothing.

    Here is the interesting part when I change the IP address of the machine (manually) to an IP which is not allocated (in the same VLAN of course) the machine starts working perfectly! Also from an average of 35 machines per branch why does the billing machine only die out?

    I ran a sniffer and process monitor (not process explorer) side by side and found hardly anything. Only thing that stood out was at in a 20 minutes capture after 8 minutes there was a HUGE ARP broadcast coming from the switch IP. The switch (gateway for this VLAN) was sending exceedingly large number of ARP requests to broadcast IP – around 1300 packets before it stopped. This is the only thing which stood out. This goes on for two minutes and then everything calms down. Then again after 30 minutes or so this activity comes up again but I am not sure since I didn’t have enough time at the location where I was gathering data. I can confirm this tomorrow.

    I am willing to share logs and sniffed data for those who want it. ANY HELP WOULD BE GREATLY APPRECIATED.

    THANK YOU VERY MUCH IN ADVANCE.
    Last edited by ByTeWrangler; April 2nd, 2009 at 03:31 PM.
    Parth Maniar,
    CISSP, CISM, CISA, SSCP

    *Thank you GOD*

    Greater the Difficulty, SWEETER the Victory.

    Believe in yourself.

  2. #2
    Gonzo District BOFH westin's Avatar
    Join Date
    Jan 2006
    Location
    SW MO
    Posts
    1,187
    Could be way off base here, but has any of the proxy info in IE been changed?

    Also, have you tried disabling the Windows firewall to make sure that it is not malfunctioning?

    It would be odd for either of these to affect multiple systems though...

    Best of luck to you!

    edit:

    This is pretty old, and I believe the problem has been fixed by MS, but just in case, you might want to search AD for any computers named WPAD.
    Last edited by westin; April 2nd, 2009 at 03:20 PM.
    \"Those of us that had been up all night were in no mood for coffee and donuts, we wanted strong drink.\"

    -HST

  3. #3
    Thanks Westin.. I just finished editing the post.. Firewall is definitely not the problem. We don’t used inbuilt firewall and the HIPS package was switched off during testing !
    Parth Maniar,
    CISSP, CISM, CISA, SSCP

    *Thank you GOD*

    Greater the Difficulty, SWEETER the Victory.

    Believe in yourself.

  4. #4
    My feeling is some DoS attack.. But machines processor level doesn’t go way high nor are the ARP entries malformed, there are no concurrent connections.. ! Nothing out of the blue..

    Really Really REALLY ODD !
    Parth Maniar,
    CISSP, CISM, CISA, SSCP

    *Thank you GOD*

    Greater the Difficulty, SWEETER the Victory.

    Believe in yourself.

  5. #5
    Gonzo District BOFH westin's Avatar
    Join Date
    Jan 2006
    Location
    SW MO
    Posts
    1,187
    I see. Well, there goes that idea!

    If you are still worried about ARP, you can run Ettercap with the ARP_Cop filter. That looks for any machines doing suspicious things with ARP on your network.

    If it were a DoS attack, you would think that you would notice some issues with ping times, or pings not responding.

    Puzzling indeed. Can you access the sites by IP in IE? I hope I am not going on a wild goose chase here.
    \"Those of us that had been up all night were in no mood for coffee and donuts, we wanted strong drink.\"

    -HST

  6. #6
    AO's Filibustier Cheap Scotch Ron's Avatar
    Join Date
    Nov 2008
    Location
    Swamps of Jersey
    Posts
    378
    But somehow they just can’t access any WEBSITE (internet and intranet), through IE; it just dies out.
    What was in the sniffer log when you tried this? specifically, could you see the outbound packets from the BILLING workstation trying to get to the WEBSITE?
    In God We Trust....Everything else we backup.

  7. #7
    There are couple of things I wanted to add.. I did some analysis on my way back..

    I ran alternative browser (opera) and like IE even it can’t open any website.


    I ran teamviewer software (it’s a server based VPN / remote access software - www.teamviewer.com), anyway the software activates by checking into its server. When I ran the software it successfully checked into its server and I could connect to the same machine remotely.

    Point 2 is the sniffer logs.. I'm going through them right now.. I ran a sniffer entire night on the machine and I’m going through that too.. Ill update as soon as I can.. I will post sniffer logs when i queried the website in few minutes.


    PS: The machine still can’t open a website. I can access through netmeeting though.
    Parth Maniar,
    CISSP, CISM, CISA, SSCP

    *Thank you GOD*

    Greater the Difficulty, SWEETER the Victory.

    Believe in yourself.

  8. #8
    I spent 2 days going through everything I had (network logs, process logs, security logs)..

    Is there anyone who can suggest something or help ?

    I really need help on this one.. If you want I will send wireshark logs or any log required..

    Just need some help..
    Parth Maniar,
    CISSP, CISM, CISA, SSCP

    *Thank you GOD*

    Greater the Difficulty, SWEETER the Victory.

    Believe in yourself.

  9. #9
    THE Bastard Sys***** dinowuff's Avatar
    Join Date
    Jun 2003
    Location
    Third planet from the Sun
    Posts
    1,253
    Let me get this straight.

    All Machines have the EXACT same problem
    You say the machines disapear from the network and you can ping the websites?
    Are you saying that these machines are web servers?
    Are all the "billing" machines on their own vLan? How are they connected - Hardware?
    09:F9:11:02:9D:74:E3:5B8:41:56:C5:63:56:88:C0

  10. #10
    ALL the machine currently affected have the same problem. There are around 40 machines out of 4500 which are currently having this problem.

    Billing machines are same as all other machines except with billing software. They are not running any kind of server service. Billing and surfing machines are all on the same VLAN.

    They are connected to a switch (locally at the store) which then uses an uplink (DLC) to connect to DC's, Billing / web servers or using a proxy they connect to the internet..
    Parth Maniar,
    CISSP, CISM, CISA, SSCP

    *Thank you GOD*

    Greater the Difficulty, SWEETER the Victory.

    Believe in yourself.

Similar Threads

  1. Trojans - Ports
    By GbinaryR in forum AntiVirus Discussions
    Replies: 11
    Last Post: October 30th, 2008, 09:33 AM
  2. Secure your wireless network
    By DeadAddict in forum The Security Tutorials Forum
    Replies: 10
    Last Post: July 21st, 2008, 12:16 AM
  3. Windows 2000 Tips
    By Nokia in forum Tips and Tricks
    Replies: 0
    Last Post: June 12th, 2004, 05:13 PM
  4. A look into IDS/Snort Whole thing by QoD
    By qod in forum The Security Tutorials Forum
    Replies: 6
    Last Post: February 27th, 2004, 03:03 AM
  5. mini-tutorial on network topologies
    By cwk9 in forum Other Tutorials Forum
    Replies: 6
    Last Post: June 3rd, 2002, 06:57 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •