These is still a vast world wide web left beyond the reach of today's search engines, which i prefer to call 'Deeper end of the Web'.


The deeper end of the web consists of web pages created dynamically in response to the user requests and huge databases linked to the sites. Though search engine querry can fetch you a static page html or else depending upon the metadata or whatsoever.. still your search querry in not able to fetch any response from the huge data residing in the public databases.. The search engines are yet to get capable of digging this much deeper in to the web yet.

Searching on the Internet today can be compared to dragging a net across the surface of the ocean. While a great deal may be caught in the net, there is still a wealth of information that is deep, and therefore, missed. The reason is simple: Most of the Web's information is buried far down on dynamically generated sites, and standard search engines never find it.

Traditional search engines create their indices by spidering or crawling surface Web pages. To be discovered, the page must be static and linked to other pages. Traditional search engines can not "see" or retrieve content in the deep Web — those pages do not exist until they are created dynamically as the result of a specific search. Because traditional search engine crawlers can not probe beneath the surface, the deep Web has heretofore been hidden.

The deep Web is qualitatively different from the surface Web. Deep Web sources store their content in searchable databases that only produce results dynamically in response to a direct request. But a direct query is a "one at a time" laborious way to search. BrightPlanet's search technology automates the process of making dozens of direct queries simultaneously using multiple-thread technology and thus is the only search technology, so far, that is capable of identifying, retrieving, qualifying, classifying, and organizing both "deep" and "surface" content.

This is the interesting paper on this topic.


P.S. posting slightly offtopic but interesting info in this post only to keep the spirit of the thread alive.


Peace