|
-
April 27th, 2006, 11:37 AM
#4
I consider it as a good thing. Those spiders index your webpages so that people searching for certain info will arrive at your website. Your website will be exposed to larger audiences. If you wish to prevent them from crawling and indexing certain files or directories for some reasons, you can use a robots.txt file http://www.robotstxt.org/wc/norobots.html and here is a validator for robots.txt to check for syntax errors http://tool.motoricerca.info/robots-checker.phtml
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
|