This was taken from
Authors: Mike Frantzen
Download: OpenBSD 3.0 -CURRENT diff
Description: StackGhost is a patch to the OpenBSD kernel for the Sparc architecture. StackGhost transparently and automatically protects applications' stack frames; more specifically, it guards the return pointers. The protection mechanisms require no application source or binary modification and imposes only a negligible performance penalty.
On Sparc processors, the kernel is ultimately responsible for saving registers to the stack and restoring them from it. When a function is called, the registers are saved into a hardware circular buffer. When the buffer is about to overflow, the processor traps into the kernel which saves the oldest registers to the userland stack. When functions return, the last set of registers is popped off the ring buffer. If the registers are not in-hardware, the kernel is invoked to restore them from the stack. Please see the below whitepaper for a less diluted account.
StackGhost interfaces with the kernel trap handler that would normally write out registers to the stack and the handler that would read them back in. By XORing a cookie into the return-address saved in the user stack when it is actually written to the stack, and then XOR it out when the return-address is pulled from the stack, StackGhost can cause attacker corrupted return pointers to behave in a manner the attacker cannot predict. StackGhost can also use several unused bits in the return pointer to detect a smashed return pointer and abort the process.
A more sophisticated mechanism is under development that will keep a seperate per-process but kernel-space stack of the userland return addresses. If the process attempts to return to an address that is not associated with a function's caller, the process will be aborted. Since the return-address stack is in protected kernel-space, an attacker can under no circumstances access the return-address stack. There are still some serious threading and setjmp/longjmp issues to iron out.
Performance: The SPEC95int benchmark suite showed a StackGhost overhead of less than one percent over the geometric mean. Further benchmarks indicate that the deviation was more attributable to noise than StackGhost. One round of benchmarks actually showed a StackGhosted system was a percent faster than an un-StackGhosted machine.
Compatibility Issues: The only issues to date have been the breakage of GDB (the GNU Debugger). Time permitting, we will remedy the problem via the kernel core dump mechanism.
White Papers: Only a draft and its a bit raw in parts.
Mike Frantzen and Mike Shuey. "StackGhost: Hardware Facilitated Stack Protection." 2001. Published in USENIX Security Symposium '01. [ps] [pdf] [html]
Presentation Slides [ps] [pdf]
Copyright (c) 2001 Mike Frantzen, West Lafayette, IN, USA.