Search Engine Spiders

[mikester2]

Can anyone explain (or have any links to sites explaining) how search engine spiders (bots, indexers, whatever you want to call them) work ? How to they find the sites? Where do they start looking ? etc etc

[Spyder32]

Well, they kinda work similiar to search engine's. They use context and word's in the meta tag's and word's that are on the site. For instance, AntiOnline has word's dealing with security on the front page and in it's source code. Spider's will go to AntiOnline then find a site with those meta tag's and header's in it's source and on the website itself.

[mikester2]

any idea how the search bot knows that antionline.com was here in the first place ?

[Negative]

All you ever wanted to know (and more) about search indexing robot spiders

[mikester2]

Thanks alot dude, that was extactly what i was looking for.

[Spyder32]

Damn you Negative Smart bastard..