Is Patching Really A Waste?

[gore]

http://www.ranum.com/security/comput...tzu/index.html



What Sun Tzu Would Say

Computer security practitioners love to quote Sun Tzu. I have no idea why, other than that he wrote many profound statements of the obvious, and security is really pretty much obvious stuff. My favorite quote from Sun Tzu regarding computer security is from his lost scrolls, which were written and subsequently pawned for services at a local massage parlor in 1750BC. They have never resurfaced, so the translation I am using is of questionable provenance. It appears to be the result of a machine-translation from an early website.

The Tale of Wise Master Tzu and the Prince's Patching Policy
It happened that The Prince of Wu was reading a bunch of Master Tzu's USENET postings and concluded that Master Tzu was wise in the way of strategy. Sending his bannermen forth, he offered Master Tzu a golden Ipod in return for a brief consulting visit in which The Master was to assess Prince Wu's Patching Policy. Prince Wu had been to numerous security conferences, and subscribed to the daily scrolls from Master SANS, and spent 3 hours per day meditating to achieve Security Focus - he was sure that Master Tzu would be impressed with his efforts and would position him well in the Magic Quadrant of visionaries. The day came when Master Tzu came to inspect Prince Wu's fortifications and Patching Policy. Alighting from his palanquin, which was carried by mighty slaves, Master Tzu sniffed the air and quickly observed to the attentive Prince Wu:
If you are fighting a losing battle, it is likely one of three things:
a) You are continuing a trend in a losing war - and therefore should not be surprised
b) You have chosen to fight the wrong battle
c) You are stupid
Master Tzu cocked an eyebrow, "which applies here?" Then he pocketed the golden Ipod, packetized himself back into his palanquin, and went home. Prince Wu fell to his knees, enlightened, had his Internet-facing servants all put to death, and never installed another patch in all his days.

OK, Joking aside, let me ask you a serious question: if patching hasn't been working, why are we still doing it?
It Feels So Good When I Stop

I recently got a call from a security journalist who wanted my recommendations for the "Best Practices" in system patching. I explained patiently to Prince Wu, uh, urr, the journalist, that I don't patch my systems at all. My home file server is still running OpenBSD from 5 years ago (and it works fine) and my Solaris machine is running some ancient version that is compatible with some of the development software I rely on. I pretty much set my network up, and don't screw with it. In return, it pretty much just works unless a cable jiggles loose or the dogs chew on something.

Somehow, the computer security industry has become addicted to patching systems - a process that is fundamentally doomed to failure. As Prince Wu realized when Master Tzu enlightened him, patching is an endless losing battle that we're stupid to engage in. So what's going on? Patching, as it's being practiced today by security practitioners, is basically the security equivalent of a fad diet. It's a tremendously expensive and painful palliative that is undertaken instead of doing something simple and obvious - namely:

* Run software that does not suck
* Absolutely minimize Internet-facing services

In the typical fad diet, the dieter engages in all kinds of weird and expensive eating habits that presumably allow them to avoid the basic reality that if you eat less, you'll lose weight - the successful fad diets are the ones that manipulate the dieter into eating less (or eating things that are less efficiently metabolized) so that they achieve the same results (i.e.: they are eating less) without realizing they are eating less. For example, eating Atkins "bread" made out of hard-to-metabolize recycled plaster and sawdust allows you to eat "all you want" - except that you won't want very much of that garbage. So you'll sneak a real bagel once a month - which is probably a good intake rate for bagels anyhow. Why not cut to the chase and just eat one bagel a month? That'd require discipline but it'd save you a lot of money sunk on expensive awful-tasting food from multi-mega diet food conglomerates.

Rather than running software that does not suck, why not run the same crud that everyone else is running, and spend one full-time engineer on upgrading it every week? Hmmm... That sounds like a great strategy except it doesn't work. So you put yourself on the patch treadmill and sink all these costs into chasing the latest mostly-works version, and you're still going to get clobbered by the next big worm that comes along and exploits a vulnerability that you and your 1.6million peers currently have installed. If you're a good patch addict, you'll have the patch installed nearly immediately - unlike me - and your window of exposure will be hours instead of days or even years. But the problem is that you'll still be exposed for a while. It might be too long. Me? I'm not exposed to IIS bugs because I don't run IIS. I'm not exposed to IE bugs because I don't run IE. I'm not exposed to Outlook bugs because I don't run Outlook. 2 years ago Lance Spitzner and I were teaching a class at SANS and people started getting up and bolting for the door. Even Lance looked worried. A new vulnerability had been found in SSHd and suddenly everyone had to run and compile a new version and install it on their most crucial systems - or else. We called a break and everyone fixed their systems except for me: I don't run SSHd. I'm not exposed to SSHd bugs. Do you detect a pattern here?

Patching shows an acceptance that the administrator has not solved the problem - it shows an acceptance that you have signed up for an endless war that you cannot win. Master Tzu might say it indicates you are stupid or, at the very least, hammered into stupidity by the constant stream of vulnerabilities in mission critical software. It should be pretty obvious that constantly upgrading mission critical software is a bad idea from a systems reliability standpoint, too.

The Tale of Wise Master Tzu and the Production Network
Master Tzu was visiting with his friend Willow Blossom, who ran a mission critical network for a large E-commerce site. Blossom complained, "I hate software these days; I cannot trust that my system will work from one day to the next because code is so buggy. I am losing sleep, and my hair is falling out." Master Tzu opined that this was tragic because Willow Blossom's hair was a gorgeous cascade of deep black - as black and shiny and deep as a null device on a spring morning. He bowed and excused himself, and asked for an audience with Prince Ciao (pronounced "Cee Eye Oh") who was lord of Willow Blossom's castle. He took a brush, and on the floor of the audience chamber wrote in ink:
1) Set up the production systems
2) Make them work
3) Test them
4) While true; do
If they are working; Continue; Endif
If they are not working; GOTO 2; Endif
5) Done

Prince Ciao studied Master Tzu's writing for weeks even to the point of missing his golf games, and was finally enlightened. He summoned Willow Blossom and explained Tzu's wisdom, then had her head and its beautiful hair mounted on a stick in the NOC as an example to the others, even though it was his own policy that Willow install patches as fast as they came from the vendors. The next time Master Tzu was invited to the castle, he politely declined.

During the 90's we were assaulted with a welter of products, the majority of which were half-assed and largely useless. And during that time, because Prince Ciao read all the marketing literature and WIRED magazine, network and system administrators were forced or "encouraged" to field beta-test code at an absolutely insane rate. The mainframe programmers of the 70's and 80's used to write of a practice called "Change Control" - in which production systems were managed with care and forethought. During the late 90's the last of the Change Control believers were taken out and shot, and their cubicles were given to the consultants who were there to mark everything up in XML in order to make everything better in some manner nobody understands yet. During that time, security practitioners were forced to repeatedly bend over and grip their ankles by business units that had already spent good money on bad products so by golly they were going to field them because otherwise Prince Ciao would have their heads. Of course nobody wanted to admit that. In 2000 I was Prince Ciao for a small start-up. Our sales VP went over my head to the CEO and bought the company Seibel's sales/customer management tool at the incredibly low price of only $500,000. Of course, it required 3 consultants working for 9 months to learn that it actually needed 5 consultants working for 12 months to make it work. I began to sharpen my stake. The icing on the cake was the discovery that Seibel required the use of Internet Explorer in order to function properly. Guess what happened? Explorer went in, of course. Where was Master Tzu when I needed him?

I truly believe that the patching fad in which we are currently living is not going to last much longer. It can't. In another couple years, we'll have one full-time patcher to each system administrator. What's odd is that if companies simply exercised a bit of discipline, it wouldn't be necessary at all. Back in 1996 a buddy of mine and I set up a web server for a high-traffic significant target. It was not the Whitehouse; it was a porn site. We invested 8 hours (of our customer's money) writing a small web server daemon that knew how to serve up files, cache them, and virtualize filenames behind hashes. It ran chrooted on a version of UNIX that was very minimized and had code hacked right into the IP stack to toss traffic that was not TCP aimed at port 80. 10 years later, it's still working, has never been hacked, and has never been patched. If you compute the Return On Investment (Or ROI in the language of Prince Ciao) it's gigantic. A client of mine works for a fairly large bank, which bought an E-banking app from a 3rd party. The E-banking app required months and months of HTML development, consulting, and customization before they could put it into test. When they were well into their testing, they hired me to come look at it and I was horrified to discover that the app (which cost $400,000) ran on an old version of NT, and required use of an old version of Microsoft IIS. When I got onto some con-calls with the provider they explained that my client could protect the NT server "with a firewall" and that they were focused on providing connectivity, not security: that was left as an exercise for their customer. It went live, of course, but only after tons more money was spent on remediation for what was fundamentally a poor choice of tools. What was the ROI on this project? I don't want to think about it. Somehow, Prince Ciao has convinced himself that "Off The Shelf Software" is good while "Custom" software is bad. What Prince Ciao doesn't understand is that that thing consultants do is called "customization" and by the time you've configured it with a lot of firewalls, patches, and 10,000 other fixes and hot-swaps it's not exacly "off the shelf" software anymore.

We need to challenge the conventional wisdom, for that's what patching has become. Organizations insist on fielding software that should not be fielded, and their justification for fielding it is "we can patch it to the point where it's OK to field it." But there is no amount of patching that is sufficient to make some of this stuff Internet-worthy. If we're still fighting that battle 5 years from now, we're stupid. It's the wrong battle. The battle we need to be fighting is against the concept of using crappy software for mission critical apps. Someone needs to look at the ROI of a complete lifecycle of "write it and forget it" software versus "light it up and patch it forever" junkware. Then, the beheadings will begin.

mjr.
In the air over Dallas,
July 26 2004

[The Grunt]

Incredibly interesting train of thought... Thanks for posting this...

It makes sense, but the only problem is it requires a highly skilled person to start things in the first place... This causes problems for people straight out of devry with no real understanding of things, who are ready to use off the shelf **** to get stuff going...

[cacosapo]

About 20 years ago (Mesozoic Era) we started to apply patch more than once a year. It was the "online Era". New application got "real time" processing - we started with CICS 1.3.
On that time we just only patch for stability - "if it is working, no one will touch it". When a problem recurred, we patched the software. No security concerns at that time.
When the "Internet" came, we faced a new enemy - "the outsider" - although we already have the "the insider", noone really cares about them.
With Internet, came new software - the crappy ones called "mid platform" - *nix flavor and Windows. (*nix already exist before, but a few companies had - we were all dynossaurs).
And the problems begun.
Nowadays banks (some of my clients) have mixed platforms - windows, *nix, Solaris, Hp-ux and others.
And they (banks) have a problem (here at least): if they get robbed thru a "unpatched well known vulnerability" CIO (and above) can be charged and go to jail.
So the fear of "group toillete" make them patch everything on "day+1".
I agree with you Gore. We have now a culture of "security patching". We apply patches that we dont even need. Just because "there is a new patch".
Our major problem is that we (IT) cant anymore dictate rules on organisation:
if "the user" want a software application we cant say anymore
"is it for Linux only? sorry you cant use it here because we have only Solaris".
More and more users dont give a **** for our opinion. They want new services and want now .
So we install unknown things (we just installed an Apache & tomcat on a client because the new application needs - but the client has zero *nix culture) just because business.
Its the new days.

[dinowuff]

I only patch if:

box is connected to the internet

then only if the patch addresses an existing issue on that box.

For example:

My file servers gateway address are 0.0.0.0

On of my file servers is NT4. The browser is 4.0 No need to upgrade or patch IE.

My mail servers are a different story. The majority of patches are applied.

Between IDS, Firewall, router acls, and Antivirus on the workstations - I don't worry about patching workstations. Haven't had a problem in over 6 years.

[thehorse13]

* Run software that does not suck
* Absolutely minimize Internet-facing services

If only life were this simple but reality seems to hit me like a 16lb sledge hammer.

Organizations are moving towards exposing more internet-facing services, not reducing them. As for sucky software, Redmond isn't the only guilty party. Look at those mom & pop software developers who low ball RFPs (Request For Proposals) and end up providing you with the equivelant of MS BOB.

[xmaddness]

Well, this works well for people's personal home computers or home networks, but what about networks that do in fact need these services running? As TH13 stated, more and more companies are beginning to expand services, and servers, that are there for the public. By working in the Information Business, you must ultimately provide information, not limit it or cut it off completely. The company I am an admin for is a 100% information company. All we deal with is information, and thusly, we need our dealers, customers, clients, personell, etc etc to have access to this information. This is how we make our money. I do not ever remember a time when a dealer of ours has said,

"Gee, I think this service for this information is great, but doesn't it cost you guys more money to run it? I don't need this critical information... that baddly. Save YOUR company more money and time by not providing MY company with the information I need to do my business."

It simply does not work that way in the business world.

So what then is the solution to this? Well, as was stated, do not get into that war in the first place. We develope all of our applications in house and security from the beginning is the power word for all developements. After all, I do work in a security company. With proper planning in the beginning, you do not need to worry as much later on, but there will always be those services that will need to be updated. That is just the way it works.

xmad

[gore]

What about the problem of cheapness and not wanting to spend money on good people? Who here would hire that guy to set up their web servers like he did that porn page? If you have a box that does nothing but be a web server, why not take away every ability but web serving? As he said, stating the obvious is sometimes needed.


The FTP server here is my SUSE 9.2 box, it has almost nothing but what is needed and Mutt so that I can read Root's mail.

Every night there are security scripts ran to see if any passwords are crackable, show me what is SUID, and what MD5s have changed since yesterday, and all that, the password results, the user who have logged in and tried to log in and all log files, are sent to me at root... And another email account that isn't local. This box has yet to have a problem.

I don't need to patch it but once in a while, because ther eis hardly any software.... sort of liek the amount Windows comes with.

[The Grunt]

I don't need to patch it but once in a while, because ther eis hardly any software.... sort of liek the amount Windows comes with.

But I thought you said you didn't have to patch it but once in a while :P

[nihil]

I think I can see where Gore is coming from.........

I have quite a few systems that don't need patching, either because I want them vulnerable, or they are just not exposed. A good example might be a mass spectrometer managed by a Windows OS PC?................all the computer does is talk to the hardware it is managing?

It is sort of "horses for courses"

In the past I recall that one applied "patches" because they improved performance and reliability...........today they mostly address security flaws?

Back then I would look at what a patch did, consider if I had that or did that and decide..........today there are so many I just apply them without thinking about it too much (broadband has a lot to answer for..........different story with 28.8 dial-up)

[gore]

Heh, the problem is probably money. I'd hire that guy to set up a web server if he knew enough to do that.

My FTP server is updated as needed.

It has Mutt because every night there are logs and security scripts that run and I'd like to read wht happened.


There isn't much allowed, and there are around 19 user accounts (Some members of AO, me and whoever else wants to either pay or knows me). They all can read and write to their home directories, and nothing else.

As said before updates are rare. If you strip a Linux box down to nothing but what Windows comes with, which isn't much, you're lucky to update once a year. I should do a tutorial on that heh, that should start some **** on AO and make a good convo.