http://www.google.com/search?hl=en&i...8&q=robots.txt
Printable View
Why is it that most of these posts in this thread arn't about Google?
It could be because the point of the thread was poor. It is not google's fault the web site admins/developers do not understand the matters relating to what they are trying to do (i.e. creating a robots.txt file to control search engine agents, creating their web site/application with an appropriate file structure, etc).
If the people putting the web sites together did just a smidge more research, development and testing before throwing a server and some code out, they might not have soo many problems.