Much has been made about the security of various security minded, open source products like OpenBSD and qmail, but are these accolades warranted? The data seems to speak for itself:

“Only one remote hole in the default install, in more than 8 years!”
-http://www.openbsd.org

“My offer ($500) still stands. Nobody has found any security holes in qmail.”
-http://cr.yp.to/qmail/guarantee.html

These claims of nearly a decade of security do make a strong argument, the claims may have qualifications like how the insecurity must exist only in the current version or a minimal amount of functionality… however these don’t change the fact that the security record is excellent. Does a strong record mean they are secure?

You could argue that most systems have such appallingly bad audit trails, that perhaps they have been exploited by real attackers and just not by bugtraq junkies, but this would be just plain pessimistic and argumentative.

You could argue that OpenBSD is a downright rare operating system and even then mostly run by enthusiast and not high risk installations, consequently there has been minimal effort to break it. While this argument makes sense for OpenBSD it does not ring true for qmail, which is frequently listed as the third most popular SMTP server. Clearly it is more than just scarcity to account for a low rate of vulnerability.

The simple fact of the matter is that guys like Theo de Raadt and Dan Bernstein are vulnerability experts and very good programmers (or at least they know good code when they see it, I have no idea if either of them lift a finger or just “lead” like myself). They have done a remarkable job in removing more or less all code level errors that allow common exploits to occur.

If this is the case, why then do I feel that OpenBSD, qmail, et al are in fact poor examples of “secure products”? To understand that, we must first understand exactly what vulnerabilities are why they occur and more importantly, and why they are unavoidable at this time.

For the context of this document, it is important to understand both the classes of flaws that lead to vulnerabilities and their introduction within a product.

The seven classes (as defined by the Information Sciences Institute) are:

1. Incomplete Parameter Validation (Buffer Overflow)
2. Inconsistent Parameter Validation (Different routines w/ incongruent data formats)
3. Implicit Sharing of Privileged/Confidential Data (Covert channels)
4. Asynchronous Validation / Inadequate Serialization (Race Conditions)
5. Inadequate Identification / Authentication / Authorization (Trojans, Weak Passwords)
6. Violable Prohibition / Limit (Poor Bounds Handling)
7. Exploitable Logic Error (Poor Error Handling, Bad Resource Allocation)

These flaws may be introduced within the three primary elements of development (Requirements / Specification Design, Source Code, or Object Code), during maintenance, or even operation. 44% of these flaws occurring in the Requirements / Specification Design phase and 18% in the Operation phase compared to 30% in the source code itself according to Naval Research Laboratory studies.

Perhaps part of the reason such a large percentage of errors are introduced at the design level is because so few commercial products (an even fewer open source ones) even have a Formal Top-Level Specification (FTLS), much less a good one. This FTLS should state more or less exactly what conditions the product is to meet, and then in the case of a secure product the FTLS must be consistent its security policy. Much of the point of an FTLS is focused on validation, “Does this product do what we need it to?”

Products with an FTLS are then further verified with a formal (documented, precise, comprehensive) security policy, which has undergone formal (mathematical proofs, pre/post-conditions mapped) verification. Products lacking an FTLS may have either a formal security policy that has been formally verified or informal (un/incompletely documented, ambiguous) with no verification at this phase.

Where verification becomes more difficult is ensuring that implicit rules are not violated. Implicit policies differ from explicit rules in that they are not actually spelled out anywhere in the security policy. For example: “User data cannot exceed the allocated buffer” Is implicit while “Operators can delete logs older than seven days.” is explicit. While products with an FTLS can mathematically determine the implicit rules correctness, informal products merely address explicit rules (at best) implicit rules can only be verified with testing.

While testing is useful, (primarily for usability issues) it is not comprehensive enough to offer any real level of assurance as testing cannot prove the absence of vulnerabilities. Now review the flaw list above and think to yourself for a moment, “How many of these have you ever tested for?” Now keep in mind, I’ve only listed a few of the attacks under each classification and worse is that new ones arise all the time. How many of you knew what a Heap Overflow Attack was five years ago? Would you have though to test if such an attack could violate your security policy having never heard of such an attack? No amount of informal verification can overcome this lack of perfect knowledge and the potential infiniteness of the implicit rules.

Does secure software exist? Yes. Can secure software exist without formal validation and verification? No.

In time, as products harden, attackers will need to discover more varied attacks as the more trodden avenues become effectively shored up. New ways to exploit the flaws classes will be discovered and abused as there is currently no way to test such flaws, informally validated and verified systems will constantly find themselves in a race against attackers. A long history of success in this race is no guarantee of success in the future.

Cheers,

catch