How do you deal with patching

[Spyrus]

I work for a fairly large corporation and we have been tackling this issue more recently than not. I am kind of disappointed in the way it started out but have made the best recommendations based on the needs.

when a new patch comes out that is warranted as "Critical" the company was just pushing it out to everyone immediately through SMS and that included servers, all patches got put out, if it breaks anything then figure it out. (I can feel all of you cringing just like I still do)

Now since we can't fully replicate a lot of our environment in a lab
For Desktops:
we push the patch out to a sample population
-if there are no issues within a 24 hour period we push it out to everyone

For Servers:
Push the patch out to non critical servers,
-Test for 24 hours then hit all the critical servers.

Now this doesnt happen this way every time but it is a decent method. If we know there are going to be some major virus/hijack issues related to a specific exploit, and there is no work around other than the patch... it gets pushed out as soon as possible to all machines with minimal impact to the business (tough when you run 3 shifts 7 days a week)

A tool you might look into if you cant afford SMS is Shavlik. There is a free version and a pay for version.
Free gives you CLI ability to query the machines and then apply the patches
Pay for gives you a cool GUI with some other options.

I think the free version is fine if you are in a smaller environment and can whip up a batch file.

[RoadClosed]

I might as well be working at McDonalds because they're more open with their money...

Nah dude you are gaining critical experience. Stick it out, you have beer money.

My patches depend. Critical servers do not have immediate access to the internet. So they wait. These are the back end servers. Front line servers are patched ASAP by anything MS deems critical through WSUS. Have I said how much I love Windows Server Update Services? It's FREE too. BUT here is the catch I do a backup every night so if that failed I may wait to patch and before I EVER reboot I update the ERD and recently started building recovery CD ROMS in the Winternals Admin Pack. It also depends on if the patch has rollback capability. Some do, but recently they don't. The fallout from not patching could be worse but a written patch policy will protect you. Some guidelines to follow. For instance sometimes the Exchange patch does not require a reboot and if you look at my other post you will note that I almost got fuxored before this one was found. Real story one day while patching I rebooted an exchange server which did not come back up. It lost a mirror boot partition and the downtime was significant. I almost never reboot those during the day because there are other factors involved. For instance only the smtp port 25 is open to it, so a browser patch doesn't warrant imediate risk to rebooting. In this case today however, it is the construct of an email that can comromise it. In addition to those items things like outgoing HTTP being monitored so I know when it's accessing the web. Other factors... stay away from Mcdonalds!

[RoadClosed]

I might as well be working at McDonalds because they're more open with their money...

Nah dude you are gaining critical experience. Stick it out, you have beer money.

My patches depend. Critical servers do not have immediate access to the internet. So they wait. These are the back end servers. Front line servers are patched ASAP by anything MS deems critical through WSUS. Have I said how much I love Windows Server Update Services? It's FREE too. BUT here is the catch I do a backup every night so if that failed I may wait to patch and before I EVER reboot I update the ERD and recently started building recovery CD ROMS in the Winternals Admin Pack. It also depends on if the patch has rollback capability. Some do, but recently they don't. The fallout from not patching could be worse but a written patch policy will protect you. Some guidelines to follow. For instance sometimes the Exchange patch does not require a reboot and if you look at my other post you will note that I almost got fuxored before this one was found. Real story one day while patching I rebooted an exchange server which did not come back up. It lost a mirror boot partition and the downtime was significant. I almost never reboot those during the day because there are other factors involved. For instance only the smtp port 25 is open to it, so a browser patch doesn't warrant imediate risk to rebooting. In this case today however, it is the construct of an email that can comromise it. In addition to those items things like outgoing HTTP being monitored so I know when it's accessing the web. Other factors... stay away from Mcdonalds!

[ric-o]

Hey HT:

Little late with this posting but here's what we do at my company when a vendor releases a patch for hw or sw we have deployed...

1. Do a risk analysis based on: the vendors bulletin, 3rd party analysis (we subscribe to Symantec's Deep Sight Alert Service), chatter on security boards (ISC, AO, etc), has exploit code been released, assess how easily it can be exploited in our environment. Internet facing gets the most scrutiny. This all culmulates into a 2-3 page writeup on our vulnerability analysis form.

This writeup gets sent out to all IT managers and key/lead system admins and desktop supervisors.

2. Review it with a team from all areas of IT and finalize a action plan and schedule.

3. We give the vulnerabilities a risk rating: CRITICAL = must patch either immediately or that evening, HIGH = patch within 3-4 days, MODERATE = patch within 7-10 days (approximately), LOW = patch during next maintenance cycle, NONE = no action

4. We then have a test schedule for each area: desktops/laptops, internal servers, Internet web servers.

5. We perform the testing and then either revise the production patching schedule (sooner or later). If decided to delay the patch we take a close look at what that means regarding risk and sometimes take temporary measures to mitigate the risks such as blocking attachments on emails.

As mentioned in #5 we assess whether to take temporary measures to mitigate risk if the risk is high enough while we perform our testing. If we cant do this sometimes we have to cut short the testing and just make sure we have good backups and understand what our fallback position is if we have problems (uninstalling, etc).

Patching Mechanisms
* For desktops/laptops we use Microsoft's Software Update Services (SUS) and will be moving to WSUS soon.
* For internal and Internet facing servers we use Update Expert and some manual work.

Regarding Latest Microsoft Bulletins
* Tested on desktops/laptops already and rolling out as of 12pm today
* Started testing on servers with hope to rollout the Exchange patch on the 1/22 which is our pre-scheduled monthly maintenance window. We will opt to do it sooner if exploit activity is reported.

Happy patching....sigh.

[ric-o]

Hey HT:

Little late with this posting but here's what we do at my company when a vendor releases a patch for hw or sw we have deployed...

1. Do a risk analysis based on: the vendors bulletin, 3rd party analysis (we subscribe to Symantec's Deep Sight Alert Service), chatter on security boards (ISC, AO, etc), has exploit code been released, assess how easily it can be exploited in our environment. Internet facing gets the most scrutiny. This all culmulates into a 2-3 page writeup on our vulnerability analysis form.

This writeup gets sent out to all IT managers and key/lead system admins and desktop supervisors.

2. Review it with a team from all areas of IT and finalize a action plan and schedule.

3. We give the vulnerabilities a risk rating: CRITICAL = must patch either immediately or that evening, HIGH = patch within 3-4 days, MODERATE = patch within 7-10 days (approximately), LOW = patch during next maintenance cycle, NONE = no action

4. We then have a test schedule for each area: desktops/laptops, internal servers, Internet web servers.

5. We perform the testing and then either revise the production patching schedule (sooner or later). If decided to delay the patch we take a close look at what that means regarding risk and sometimes take temporary measures to mitigate the risks such as blocking attachments on emails.

As mentioned in #5 we assess whether to take temporary measures to mitigate risk if the risk is high enough while we perform our testing. If we cant do this sometimes we have to cut short the testing and just make sure we have good backups and understand what our fallback position is if we have problems (uninstalling, etc).

Patching Mechanisms
* For desktops/laptops we use Microsoft's Software Update Services (SUS) and will be moving to WSUS soon.
* For internal and Internet facing servers we use Update Expert and some manual work.

Regarding Latest Microsoft Bulletins
* Tested on desktops/laptops already and rolling out as of 12pm today
* Started testing on servers with hope to rollout the Exchange patch on the 1/22 which is our pre-scheduled monthly maintenance window. We will opt to do it sooner if exploit activity is reported.

Happy patching....sigh.