BEWARE OF GOOGLE (what every sysadmin NEEDS to know about search engines)

[maxim_86ualb2]

tht is right....... more tutorials........

[n01100110]

Excellent tutorial though breakology.. I tried some search strings playing around and i am appauled at some of the stuff i found.. Excellent though nontheless...

[rapier57]

What it amounts to (on the web sites and other-than-personal web/share locations displayed on the links), is sloppiness on the part of the sys ads, webmasters or what have you (or have you not?).

Don't blame Google. There are other search engines out there probing web sites, and there are web crawlers and spider bots that dig web sites for more than search engines. One of the more insidious are those that scrape email addresses from sites and make up SPAM lists.

One line of defense for web sites is to protect it against robots (web crawlers and spiders). You create a robots.txt file in your web root, and any other entry point on your web site.

Instructions and detailed information are here:

http://www.robotstxt.org/wc/robots.html

Create a robots.txt files for all entry points, sub-webs and intranets or plan to have everything open to the universe.

[n01100110]

I don't blame google at all..Google helps me live every day As you stated , it is sloppiness on the server administrators part..

[maxim_86ualb2]

google is the best........... search engine ever....... that is for me ...............

[n01100110]

Yes im sure it is for all of us here... Im just still in shock at some of the information that is there.. /me pities web server admins

[ric-o]

n01100110 wrote :
Lets say if i own a web server and someone decides to try to data prawl my site.. How would you go about protecting this ? How would i hide the directories from google ?

Google tells you how to keep them from caching your site -> http://www.google.ca/webmasters/3.html#B3

[DÆMÔÑ]

Greetings every body,

I found a related topic on "google" at newoder.box.sk .It name is *Google: A Hacker's Best Friend*

[Shrekkie]

I must say ... very nice info.

Keep this one updated.

Greetz

[breakology]

Another intersesting thing you can do with google is use their translation service as an transparent proxy (not anonymous) try translating an english page back to english text here: http://www.google.com/language_tools

You end up viewing the page through a google cache with a URL that will look something like this : http://translate.google.com/translat...language_tools