Originally posted here by ;TT
Ever heard of robots.txt? Look it up, it'll stop search engines and the scenario you laid out. Sure there's ways to get secret information... but if you do, I doubt it'll be because a search engine coughed them up for you.
Robots.txt will stop Google, but that's because Google honours it. Not all search engines do. And if anybody links to a page behind the robots.txt file, it still does no good.

The robots.txt file is kind of like leaving the door to your house wide open, but posting a sign saying "please don't come in here". Not very effective against the more shady types.

To really stop search engine spiders you need links designed to 'trap' them in a slew of endlessly looping links. I'm not a web developer, and don't know exactly how to di it, but I do know that more affirmative action than that file needs to be taken against some web spiders.