-
December 8th, 2004, 05:08 AM
#31
Originally posted here by ;TT
Ever heard of robots.txt? Look it up, it'll stop search engines and the scenario you laid out. Sure there's ways to get secret information... but if you do, I doubt it'll be because a search engine coughed them up for you.
Robots.txt will stop Google, but that's because Google honours it. Not all search engines do. And if anybody links to a page behind the robots.txt file, it still does no good.
The robots.txt file is kind of like leaving the door to your house wide open, but posting a sign saying "please don't come in here". Not very effective against the more shady types.
To really stop search engine spiders you need links designed to 'trap' them in a slew of endlessly looping links. I'm not a web developer, and don't know exactly how to di it, but I do know that more affirmative action than that file needs to be taken against some web spiders.
Government is like fire - a handy servant, but a dangerous master - George Washington
Government is not reason, it is not eloquence - it is force. - George Washington.
Join the UnError community!
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
|