How do you deal with patching

Hey Hey,

The thread title basically sums it up... How do you deal with patching?

Common sense would dictate to:

Read the reports of others luck/problems when installing the patch.
Filter (if possible) to protect the machine in question while testing/researching.
Apply the patch in a test environment..

I'm looking at the TNEF Decoding Vuln and wondering how to treat it...

This isn't something where I can close a port/filter content (not that I have the ability anyways) while I read the results of others...
I don't have a test environment or even a single test machine... I was barely able to create a test XP Machine (we had one extra license)...

That leaves me with patch immediately... or leave it open and wait to read others luck with it...

My primary thought is to patch immediately... It all comes back to CYA... but my question is... is anyone else in this same situation? What did you decide to do... Sure NGS is going to wait 3 months before disclosing the details, but still... how many people are now playing with this for fun? Will we start to see it live before then... I'd think so.. I'm sure it's minor and not something I'd even have to worry about.. but again.. CYA comes to mind...

Peace,
HT

Some of it depends on the seriousness of the flaw and the likelihood of it being exploited, versus the inherent risks of applying the patch.

For example, with the WMF patch I manually tried out a W2K and XP workstation and then rolled it out straight away. In this case it was a serious flaw being actively exploited, and the risks of being hit by a nasty outweighed the risks of the patch screwing something important up.. and even if it did have problems, it was only going to be pretty limited.

On the other hand, when it comes to IE patching we are much more careful - exploits via IE tend to be one machine at a time and can be largely mitigated by anti-virus and antispyware apps. IE patches have a tendency to break business critical applications for us too, so on balance we tend to evaluate those for much longer.

Basically, when Patch Tuesday comes around we do an analysis of all the different factors individually for each patch and come up with an action plan for each individual case. We draw up a draft plan on the Wednesday and authorise any "no brainers" and then that gives us a couple of days further investigation before we hit the weekend (which is a good time for patching servers).

It's important that you set some time aside every month for an analysis of the updates. I've set up a reminder function in Exchange for all the relevant people on our side (starting on the Pre-Patch-Tuesday-Friday) as a reminder.

I always patch servers off hours..usually on weekends...gives me more time to test, restore and\or fix...just encase ;)

HT...just block .dat files til you can patch.

I filtered it yesterday afternoon...so far no issues.

Most of our overseas business is done via email....and I havent heard any whining from users.......... yet.

and if I did.... I would have them contact the sender and tell them to use straight text til we can resolve the issue.

MHO..as always

MLF

Hey Hey,

Thanks for the input... I generally follow discussions on AO, FD, BugTraq, FunSec, DailyDave, ISC and a few others for the week of the patches before I'd apply anything.. With the exception of my machine... I've got all my "Admin" tools on a share on the file server that I use and I've got a single folder of all my downloads that would be unaffected by a reinstall... I can go from broken system to fresh install in a couple of hours (It'd be much easier if I had ghost... maybe next time I'll use g4u/g4l and store the image)... If My machien breaks, I stop the others from being updated.... However, I came in to a place where everyone has automatic update enabled and was told to click install as soon as the icon came up... for the most part I've left them to that... at most they need their email to function and they can always use a spare Mac and safari to access that... Also since they dont' listen overly well I can't change it. heh..

I think for me the big thing is Exchange... I've always been a little leary of updating it... but when I see this

Quote:

---

Source: http://www.microsoft.com/technet/sec...ate=2006-01-10
On vulnerable versions of Exchange, an attacker who successfully exploited this vulnerability could take complete control of an affected system. This vulnerability could be exploited automatically without user interaction. An attacker could then install programs; view, change, or delete data; or create new accounts with full user rights.

We recommend that customers apply the update immediately.

---

it makes me think that maybe I should just throw the update on... Especially since I don't have a test server to play with... I haven't done it yet though... I'm just gonna wait and see what comes across the mailing lists...

The problem is that reading and researching "aren't my job" ... and I should be working...

heh... mid post (well one line ago) I was just asked to "calibrate the colour" on the monitors... they're on 24x7 and their starting to die... but they want me to calibrate the colour so that they all look the same....

mlf: It was my understanding that you had to actually filter the mime-type to prevent against it... Will .dat filtering alone actually do the trick?

Well looks like I'm off for a bit,

Peace,
HT

Quote:

---

HT...just block .dat files til you can patch.

---

Yes, it's worth looking at the workarounds for each problem too. Sometimes the workarounds are very easy to implement and are often more effective.

HT:

I'd put it on the server for your CYA reason... But I'd tell your boss before you do that while the risk of compromise is low it's possible and that compromise of the exchange server is the compromise of your entire network due to it's multi-application nature thus it should be done. Also point out that the patch _could_ break the box which will bring down the organization due to the multi-application nature of the box. Then ask him if he can say "Catch 22"..... Then ask for enough money to split those applications to other resources so that your current Catch 22 can't have such a wide ranging effect on the organization in the future.... Throwing the phrase "best practice" in there wherever you can might be a good idea too.... ;)

http://www.microsoft.com/technet/se...n/MS06-003.mspx

Quote:

---

Note Filtering only for attachments that have the file name Winmail.dat may not be sufficient to help protect your system. A malicious file attachment could be given another file name that could then be processed by the Exchange Server computer. To help protect against malicious e-mail message’s, block all application/ms-tnef MIME type content.

---

Your right....may stop some though

Patching clients as we speak

MLF

FYI

Quote:

---

When this security bulletin was issued, had this vulnerability been publicly disclosed?No. Microsoft received information about this vulnerability through responsible disclosure. Microsoft had not received any information to indicate that this vulnerability had been publicly disclosed when this security bulletin was originally issued.

When this security bulletin was issued, had Microsoft received any reports that this vulnerability was being exploited?
No. Microsoft had not received any information to indicate that this vulnerability had been publicly used to attack customers and had not seen any examples of proof of concept code published when this security bulletin was originally issued.

---

Quote:

---

Originally posted here by Tiger Shark
HT:

I'd put it on the server for your CYA reason... But I'd tell your boss before you do that while the risk of compromise is low it's possible and that compromise of the exchange server is the compromise of your entire network due to it's multi-application nature thus it should be done. Also point out that the patch _could_ break the box which will bring down the organization due to the multi-application nature of the box. Then ask him if he can say "Catch 22"..... Then ask for enough money to split those applications to other resources so that your current Catch 22 can't have such a wide ranging effect on the organization in the future.... Throwing the phrase "best practice" in there wherever you can might be a good idea too.... ;)

---

It wouldn't do any good... I already tried that when I had to dump SQL on the already overloaded and publicly available server... Keep it running and if it breaks fix it... that's the moto around here... we can pay $750/month for software that I still consider to be useless.... but they can't get me what I need... but then again.. after taxes I make 8.75 an hour... (Canadian)... I might as well be working at McDonalds because they're more open with their money...

I got to replace file storage (with a RAID config for the extra backup) and our backup solution.. that was it.. now that those are up-to-date nothing else matters (we'll just restore from backup when it breaks)...

Peace,
HT

Quote:

---

Originally posted here by Tiger Shark
HT:

I'd put it on the server for your CYA reason... But I'd tell your boss before you do that while the risk of compromise is low it's possible and that compromise of the exchange server is the compromise of your entire network due to it's multi-application nature thus it should be done. Also point out that the patch _could_ break the box which will bring down the organization due to the multi-application nature of the box. Then ask him if he can say "Catch 22"..... Then ask for enough money to split those applications to other resources so that your current Catch 22 can't have such a wide ranging effect on the organization in the future.... Throwing the phrase "best practice" in there wherever you can might be a good idea too.... ;)

---

It wouldn't do any good... I already tried that when I had to dump SQL on the already overloaded and publicly available server... Keep it running and if it breaks fix it... that's the moto around here... we can pay $750/month for software that I still consider to be useless.... but they can't get me what I need... but then again.. after taxes I make 8.75 an hour... (Canadian)... I might as well be working at McDonalds because they're more open with their money...

I got to replace file storage (with a RAID config for the extra backup) and our backup solution.. that was it.. now that those are up-to-date nothing else matters (we'll just restore from backup when it breaks)...

Peace,
HT

I work for a fairly large corporation and we have been tackling this issue more recently than not. I am kind of disappointed in the way it started out but have made the best recommendations based on the needs.

when a new patch comes out that is warranted as "Critical" the company was just pushing it out to everyone immediately through SMS and that included servers, all patches got put out, if it breaks anything then figure it out. (I can feel all of you cringing just like I still do)

Now since we can't fully replicate a lot of our environment in a lab
For Desktops:
we push the patch out to a sample population
-if there are no issues within a 24 hour period we push it out to everyone

For Servers:
Push the patch out to non critical servers,
-Test for 24 hours then hit all the critical servers.

Now this doesnt happen this way every time but it is a decent method. If we know there are going to be some major virus/hijack issues related to a specific exploit, and there is no work around other than the patch... it gets pushed out as soon as possible to all machines with minimal impact to the business (tough when you run 3 shifts 7 days a week)

A tool you might look into if you cant afford SMS is Shavlik. There is a free version and a pay for version.
Free gives you CLI ability to query the machines and then apply the patches
Pay for gives you a cool GUI with some other options.

I think the free version is fine if you are in a smaller environment and can whip up a batch file.

I work for a fairly large corporation and we have been tackling this issue more recently than not. I am kind of disappointed in the way it started out but have made the best recommendations based on the needs.

when a new patch comes out that is warranted as "Critical" the company was just pushing it out to everyone immediately through SMS and that included servers, all patches got put out, if it breaks anything then figure it out. (I can feel all of you cringing just like I still do)

Now since we can't fully replicate a lot of our environment in a lab
For Desktops:
we push the patch out to a sample population
-if there are no issues within a 24 hour period we push it out to everyone

For Servers:
Push the patch out to non critical servers,
-Test for 24 hours then hit all the critical servers.

Now this doesnt happen this way every time but it is a decent method. If we know there are going to be some major virus/hijack issues related to a specific exploit, and there is no work around other than the patch... it gets pushed out as soon as possible to all machines with minimal impact to the business (tough when you run 3 shifts 7 days a week)

A tool you might look into if you cant afford SMS is Shavlik. There is a free version and a pay for version.
Free gives you CLI ability to query the machines and then apply the patches
Pay for gives you a cool GUI with some other options.

I think the free version is fine if you are in a smaller environment and can whip up a batch file.

Quote:

---

I might as well be working at McDonalds because they're more open with their money...

---

Nah dude you are gaining critical experience. Stick it out, you have beer money. :p

My patches depend. Critical servers do not have immediate access to the internet. So they wait. These are the back end servers. Front line servers are patched ASAP by anything MS deems critical through WSUS. Have I said how much I love Windows Server Update Services? It's FREE too. BUT here is the catch I do a backup every night so if that failed I may wait to patch and before I EVER reboot I update the ERD and recently started building recovery CD ROMS in the Winternals Admin Pack. It also depends on if the patch has rollback capability. Some do, but recently they don't. The fallout from not patching could be worse but a written patch policy will protect you. Some guidelines to follow. For instance sometimes the Exchange patch does not require a reboot and if you look at my other post you will note that I almost got fuxored before this one was found. Real story one day while patching I rebooted an exchange server which did not come back up. It lost a mirror boot partition and the downtime was significant. I almost never reboot those during the day because there are other factors involved. For instance only the smtp port 25 is open to it, so a browser patch doesn't warrant imediate risk to rebooting. In this case today however, it is the construct of an email that can comromise it. In addition to those items things like outgoing HTTP being monitored so I know when it's accessing the web. Other factors... stay away from Mcdonalds!

Quote:

---

I might as well be working at McDonalds because they're more open with their money...

---

Nah dude you are gaining critical experience. Stick it out, you have beer money. :p

My patches depend. Critical servers do not have immediate access to the internet. So they wait. These are the back end servers. Front line servers are patched ASAP by anything MS deems critical through WSUS. Have I said how much I love Windows Server Update Services? It's FREE too. BUT here is the catch I do a backup every night so if that failed I may wait to patch and before I EVER reboot I update the ERD and recently started building recovery CD ROMS in the Winternals Admin Pack. It also depends on if the patch has rollback capability. Some do, but recently they don't. The fallout from not patching could be worse but a written patch policy will protect you. Some guidelines to follow. For instance sometimes the Exchange patch does not require a reboot and if you look at my other post you will note that I almost got fuxored before this one was found. Real story one day while patching I rebooted an exchange server which did not come back up. It lost a mirror boot partition and the downtime was significant. I almost never reboot those during the day because there are other factors involved. For instance only the smtp port 25 is open to it, so a browser patch doesn't warrant imediate risk to rebooting. In this case today however, it is the construct of an email that can comromise it. In addition to those items things like outgoing HTTP being monitored so I know when it's accessing the web. Other factors... stay away from Mcdonalds!

Hey HT:

Little late with this posting but here's what we do at my company when a vendor releases a patch for hw or sw we have deployed...

1. Do a risk analysis based on: the vendors bulletin, 3rd party analysis (we subscribe to Symantec's Deep Sight Alert Service), chatter on security boards (ISC, AO, etc), has exploit code been released, assess how easily it can be exploited in our environment. Internet facing gets the most scrutiny. This all culmulates into a 2-3 page writeup on our vulnerability analysis form.

This writeup gets sent out to all IT managers and key/lead system admins and desktop supervisors.

2. Review it with a team from all areas of IT and finalize a action plan and schedule.

3. We give the vulnerabilities a risk rating: CRITICAL = must patch either immediately or that evening, HIGH = patch within 3-4 days, MODERATE = patch within 7-10 days (approximately), LOW = patch during next maintenance cycle, NONE = no action

4. We then have a test schedule for each area: desktops/laptops, internal servers, Internet web servers.

5. We perform the testing and then either revise the production patching schedule (sooner or later). If decided to delay the patch we take a close look at what that means regarding risk and sometimes take temporary measures to mitigate the risks such as blocking attachments on emails.

As mentioned in #5 we assess whether to take temporary measures to mitigate risk if the risk is high enough while we perform our testing. If we cant do this sometimes we have to cut short the testing and just make sure we have good backups and understand what our fallback position is if we have problems (uninstalling, etc).

Patching Mechanisms
* For desktops/laptops we use Microsoft's Software Update Services (SUS) and will be moving to WSUS soon.
* For internal and Internet facing servers we use Update Expert and some manual work.

Regarding Latest Microsoft Bulletins
* Tested on desktops/laptops already and rolling out as of 12pm today
* Started testing on servers with hope to rollout the Exchange patch on the 1/22 which is our pre-scheduled monthly maintenance window. We will opt to do it sooner if exploit activity is reported.

Happy patching....sigh.

Hey HT:

Little late with this posting but here's what we do at my company when a vendor releases a patch for hw or sw we have deployed...

1. Do a risk analysis based on: the vendors bulletin, 3rd party analysis (we subscribe to Symantec's Deep Sight Alert Service), chatter on security boards (ISC, AO, etc), has exploit code been released, assess how easily it can be exploited in our environment. Internet facing gets the most scrutiny. This all culmulates into a 2-3 page writeup on our vulnerability analysis form.

This writeup gets sent out to all IT managers and key/lead system admins and desktop supervisors.

2. Review it with a team from all areas of IT and finalize a action plan and schedule.

3. We give the vulnerabilities a risk rating: CRITICAL = must patch either immediately or that evening, HIGH = patch within 3-4 days, MODERATE = patch within 7-10 days (approximately), LOW = patch during next maintenance cycle, NONE = no action

4. We then have a test schedule for each area: desktops/laptops, internal servers, Internet web servers.

5. We perform the testing and then either revise the production patching schedule (sooner or later). If decided to delay the patch we take a close look at what that means regarding risk and sometimes take temporary measures to mitigate the risks such as blocking attachments on emails.

As mentioned in #5 we assess whether to take temporary measures to mitigate risk if the risk is high enough while we perform our testing. If we cant do this sometimes we have to cut short the testing and just make sure we have good backups and understand what our fallback position is if we have problems (uninstalling, etc).

Patching Mechanisms
* For desktops/laptops we use Microsoft's Software Update Services (SUS) and will be moving to WSUS soon.
* For internal and Internet facing servers we use Update Expert and some manual work.

Regarding Latest Microsoft Bulletins
* Tested on desktops/laptops already and rolling out as of 12pm today
* Started testing on servers with hope to rollout the Exchange patch on the 1/22 which is our pre-scheduled monthly maintenance window. We will opt to do it sooner if exploit activity is reported.

Happy patching....sigh.