Is Patching Really A Waste?
http://www.ranum.com/security/comput...tzu/index.html
What Sun Tzu Would Say
Computer security practitioners love to quote Sun Tzu. I have no idea why, other than that he wrote many profound statements of the obvious, and security is really pretty much obvious stuff. My favorite quote from Sun Tzu regarding computer security is from his lost scrolls, which were written and subsequently pawned for services at a local massage parlor in 1750BC. They have never resurfaced, so the translation I am using is of questionable provenance. It appears to be the result of a machine-translation from an early website.
The Tale of Wise Master Tzu and the Prince's Patching Policy
It happened that The Prince of Wu was reading a bunch of Master Tzu's USENET postings and concluded that Master Tzu was wise in the way of strategy. Sending his bannermen forth, he offered Master Tzu a golden Ipod in return for a brief consulting visit in which The Master was to assess Prince Wu's Patching Policy. Prince Wu had been to numerous security conferences, and subscribed to the daily scrolls from Master SANS, and spent 3 hours per day meditating to achieve Security Focus - he was sure that Master Tzu would be impressed with his efforts and would position him well in the Magic Quadrant of visionaries. The day came when Master Tzu came to inspect Prince Wu's fortifications and Patching Policy. Alighting from his palanquin, which was carried by mighty slaves, Master Tzu sniffed the air and quickly observed to the attentive Prince Wu:
If you are fighting a losing battle, it is likely one of three things:
a) You are continuing a trend in a losing war - and therefore should not be surprised
b) You have chosen to fight the wrong battle
c) You are stupid
Master Tzu cocked an eyebrow, "which applies here?" Then he pocketed the golden Ipod, packetized himself back into his palanquin, and went home. Prince Wu fell to his knees, enlightened, had his Internet-facing servants all put to death, and never installed another patch in all his days.
OK, Joking aside, let me ask you a serious question: if patching hasn't been working, why are we still doing it?
It Feels So Good When I Stop
I recently got a call from a security journalist who wanted my recommendations for the "Best Practices" in system patching. I explained patiently to Prince Wu, uh, urr, the journalist, that I don't patch my systems at all. My home file server is still running OpenBSD from 5 years ago (and it works fine) and my Solaris machine is running some ancient version that is compatible with some of the development software I rely on. I pretty much set my network up, and don't screw with it. In return, it pretty much just works unless a cable jiggles loose or the dogs chew on something.
Somehow, the computer security industry has become addicted to patching systems - a process that is fundamentally doomed to failure. As Prince Wu realized when Master Tzu enlightened him, patching is an endless losing battle that we're stupid to engage in. So what's going on? Patching, as it's being practiced today by security practitioners, is basically the security equivalent of a fad diet. It's a tremendously expensive and painful palliative that is undertaken instead of doing something simple and obvious - namely:
* Run software that does not suck
* Absolutely minimize Internet-facing services
In the typical fad diet, the dieter engages in all kinds of weird and expensive eating habits that presumably allow them to avoid the basic reality that if you eat less, you'll lose weight - the successful fad diets are the ones that manipulate the dieter into eating less (or eating things that are less efficiently metabolized) so that they achieve the same results (i.e.: they are eating less) without realizing they are eating less. For example, eating Atkins "bread" made out of hard-to-metabolize recycled plaster and sawdust allows you to eat "all you want" - except that you won't want very much of that garbage. So you'll sneak a real bagel once a month - which is probably a good intake rate for bagels anyhow. Why not cut to the chase and just eat one bagel a month? That'd require discipline but it'd save you a lot of money sunk on expensive awful-tasting food from multi-mega diet food conglomerates.
Rather than running software that does not suck, why not run the same crud that everyone else is running, and spend one full-time engineer on upgrading it every week? Hmmm... That sounds like a great strategy except it doesn't work. So you put yourself on the patch treadmill and sink all these costs into chasing the latest mostly-works version, and you're still going to get clobbered by the next big worm that comes along and exploits a vulnerability that you and your 1.6million peers currently have installed. If you're a good patch addict, you'll have the patch installed nearly immediately - unlike me - and your window of exposure will be hours instead of days or even years. But the problem is that you'll still be exposed for a while. It might be too long. Me? I'm not exposed to IIS bugs because I don't run IIS. I'm not exposed to IE bugs because I don't run IE. I'm not exposed to Outlook bugs because I don't run Outlook. 2 years ago Lance Spitzner and I were teaching a class at SANS and people started getting up and bolting for the door. Even Lance looked worried. A new vulnerability had been found in SSHd and suddenly everyone had to run and compile a new version and install it on their most crucial systems - or else. We called a break and everyone fixed their systems except for me: I don't run SSHd. I'm not exposed to SSHd bugs. Do you detect a pattern here?
Patching shows an acceptance that the administrator has not solved the problem - it shows an acceptance that you have signed up for an endless war that you cannot win. Master Tzu might say it indicates you are stupid or, at the very least, hammered into stupidity by the constant stream of vulnerabilities in mission critical software. It should be pretty obvious that constantly upgrading mission critical software is a bad idea from a systems reliability standpoint, too.
The Tale of Wise Master Tzu and the Production Network
Master Tzu was visiting with his friend Willow Blossom, who ran a mission critical network for a large E-commerce site. Blossom complained, "I hate software these days; I cannot trust that my system will work from one day to the next because code is so buggy. I am losing sleep, and my hair is falling out." Master Tzu opined that this was tragic because Willow Blossom's hair was a gorgeous cascade of deep black - as black and shiny and deep as a null device on a spring morning. He bowed and excused himself, and asked for an audience with Prince Ciao (pronounced "Cee Eye Oh") who was lord of Willow Blossom's castle. He took a brush, and on the floor of the audience chamber wrote in ink:
1) Set up the production systems
2) Make them work
3) Test them
4) While true; do
If they are working; Continue; Endif
If they are not working; GOTO 2; Endif
5) Done
Prince Ciao studied Master Tzu's writing for weeks even to the point of missing his golf games, and was finally enlightened. He summoned Willow Blossom and explained Tzu's wisdom, then had her head and its beautiful hair mounted on a stick in the NOC as an example to the others, even though it was his own policy that Willow install patches as fast as they came from the vendors. The next time Master Tzu was invited to the castle, he politely declined.
During the 90's we were assaulted with a welter of products, the majority of which were half-assed and largely useless. And during that time, because Prince Ciao read all the marketing literature and WIRED magazine, network and system administrators were forced or "encouraged" to field beta-test code at an absolutely insane rate. The mainframe programmers of the 70's and 80's used to write of a practice called "Change Control" - in which production systems were managed with care and forethought. During the late 90's the last of the Change Control believers were taken out and shot, and their cubicles were given to the consultants who were there to mark everything up in XML in order to make everything better in some manner nobody understands yet. During that time, security practitioners were forced to repeatedly bend over and grip their ankles by business units that had already spent good money on bad products so by golly they were going to field them because otherwise Prince Ciao would have their heads. Of course nobody wanted to admit that. In 2000 I was Prince Ciao for a small start-up. Our sales VP went over my head to the CEO and bought the company Seibel's sales/customer management tool at the incredibly low price of only $500,000. Of course, it required 3 consultants working for 9 months to learn that it actually needed 5 consultants working for 12 months to make it work. I began to sharpen my stake. The icing on the cake was the discovery that Seibel required the use of Internet Explorer in order to function properly. Guess what happened? Explorer went in, of course. Where was Master Tzu when I needed him?
I truly believe that the patching fad in which we are currently living is not going to last much longer. It can't. In another couple years, we'll have one full-time patcher to each system administrator. What's odd is that if companies simply exercised a bit of discipline, it wouldn't be necessary at all. Back in 1996 a buddy of mine and I set up a web server for a high-traffic significant target. It was not the Whitehouse; it was a porn site. We invested 8 hours (of our customer's money) writing a small web server daemon that knew how to serve up files, cache them, and virtualize filenames behind hashes. It ran chrooted on a version of UNIX that was very minimized and had code hacked right into the IP stack to toss traffic that was not TCP aimed at port 80. 10 years later, it's still working, has never been hacked, and has never been patched. If you compute the Return On Investment (Or ROI in the language of Prince Ciao) it's gigantic. A client of mine works for a fairly large bank, which bought an E-banking app from a 3rd party. The E-banking app required months and months of HTML development, consulting, and customization before they could put it into test. When they were well into their testing, they hired me to come look at it and I was horrified to discover that the app (which cost $400,000) ran on an old version of NT, and required use of an old version of Microsoft IIS. When I got onto some con-calls with the provider they explained that my client could protect the NT server "with a firewall" and that they were focused on providing connectivity, not security: that was left as an exercise for their customer. It went live, of course, but only after tons more money was spent on remediation for what was fundamentally a poor choice of tools. What was the ROI on this project? I don't want to think about it. Somehow, Prince Ciao has convinced himself that "Off The Shelf Software" is good while "Custom" software is bad. What Prince Ciao doesn't understand is that that thing consultants do is called "customization" and by the time you've configured it with a lot of firewalls, patches, and 10,000 other fixes and hot-swaps it's not exacly "off the shelf" software anymore.
We need to challenge the conventional wisdom, for that's what patching has become. Organizations insist on fielding software that should not be fielded, and their justification for fielding it is "we can patch it to the point where it's OK to field it." But there is no amount of patching that is sufficient to make some of this stuff Internet-worthy. If we're still fighting that battle 5 years from now, we're stupid. It's the wrong battle. The battle we need to be fighting is against the concept of using crappy software for mission critical apps. Someone needs to look at the ROI of a complete lifecycle of "write it and forget it" software versus "light it up and patch it forever" junkware. Then, the beheadings will begin.
mjr.
In the air over Dallas,
July 26 2004
