Erm, web site admins should probably learn to configure robots.txt so that agents used by search engines know what to pick up, and what not to.

A quick search of the sans reading room, the owasp project, security focus, google, and the score documentation section will bring up an assortment of documentation on better web site/application design and auditing. Depending on the documentation you will get anything from what you should and should not have in comments and where, application structure and file placement, server hardening, authentication auditing, etc.