loading critical patches for mission critical servers

[mrlucifer]

just out of curiosity, how do admins load microsoft critical patches for mission critical servers that can't afford any downtime? Is there a way to load the patches without a reboot? just curious. your input is appreciated.

[morganlefay]

well...mostly..no...most critical patches I have seen require a reboot....cause its ties to the protocol...or some system file that needs to be reloaded.

Some patches you can just restart the services........but you REALLY have to know how all those services relate to eachother......and startup order.....probably easier to reboot...

I ALWAYS have a verified backup...before applying the patch...and usually do it on the weekend...because of downtime....and applying patches while the server is in use can slow it down to a crawl....or out right fail....or cause other issues.....also using the weekend...allows me extra time to fix it....before monday...and the big guys come in....and start whinning about how they cant get their email...

I guess you would have to weigh the risk of applying the patch compared to the downtime..

I usually send out an email to warn users
that network services will be unavailable due to critical patches....always give a larger window on the downtime...

That way you are a hero when it comes back up in less time then you stated

MLF

[MURACU]

Basically you do a planned downtime on the server. that is you tell everyone that from about midnight till 1 for example the server will be down. In a lot of cases a mission critical server will be redundant so that if one is down the other can take the load.
A word of warning you can apply patches with out rebooting the server but as Morgan said you have restart all the services and processes to be sure the patch is applied. I would also insist that you reboot the server before applying the patch especially on a mission critical machine which is nearly never rebooted. That way if it doesn't boot then at least you know it is not because of your patch. I've had cases where people patched servers without doing a boot either before or after applying the patch. Then wasting hours a month later when they tried to boot and the server didn't boot which would have been fine only i used get called at all hours of the night for problems like that.

[mrlucifer]

wow interesting info. I was just curious if i was doing the patches correct just like any other microsoft admin. i just hate coming in the weekends to load patches to 25 different servers, so much (boring) time invested. How do you guys load patches for servers? manual downloads, Windows Update, or some other push down software? What is your preference. I currently do the Windows Update. Thanks for the quick responses.

[Tedob1]

most of my servers just like all the workstations get their updates from SUS which will become WUS next week. this way i can wait to be sure none will cause me problems by reading all the security boards before i publish them for download.

using gpedit.msc from the run command permits you to install the updates pretty much as you'd like to.(see attached jpg) heck sus/wus is free and has already saved my ass.

[MURACU]

I'm no longer a windows admin now but we used a SUS solution also. We had about 700 servers to update so doing it manually wasn't possable.

[mohaughn]

Our highly available applications are clustered. So when we have to reboot a server for maintenance, we are only look at a minute to two minutes of downtime for the failover. That is for exchange. Other applications that we have running on clustering can fail over in less than 30 seconds. Which is generally not enough downtime to be noticed by users.


We install in our lab first, let it sit in the lab for two or three days while we are testing functionality.. Once the lab test is ok we deploy to lower priority servers the first night, and then over the course of two or three nights deploy to the rest of our servers. We never slam a patch on all servers in one night.

It should be noted that we choose to use workarounds or other means of defeating the vulnerability as opposed to always installing the patches. I prefer to wait for roll-up patches or service packs.. For instance, we never install IE hotfixes as we never use IE on the servers, and only administrators can log in.

[phishphreek]

I use a WUS servers and depoly to "test" groups first. If there are no problems, then I deploy to the rest of the machines. My groups are broken up by department and operating system.

Tedob1: Upgrading from SUS to WUS is pretty painless. Just make sure you have all the minimum requirements installed and you'll be good to go. The benefits are plenty and you'll like WUS way more than SUS.

[dynamoo]

To be honest, only a small minority of patches need to be applied to most servers.

For example - there's no real need to patch IE on a file server because you're not going to be surfing for p0rn on it. Similarly, issues with Word, PowerPoint, Windows Media Player etc etc aren't critical.

For most other patches, have a look at the workarounds and mitigating factors. Often these are BETTER solutions than patching and won't require a reboot. For example, if the flaw is in a service that you don't actually use then you could just disable that service altogether.

Yes, you probably should apply these patches to your servers sometime, but in most cases you don't actually need them or can work around the problem. You can patch them at a later date when there's something that you really DO have to patch the servers for.

[Black Cluster]

If you run a serious business and want a 99.99% availibilty and no downtime {caused by batches unless it is a core batch} run Linux ....