Many people have the naive expectation that they can find anything on the World Wide Web by using search engines like Google or Yahoo or Ask.com or Bing. The truth is that all of these search engines only index 10% of the entire web. The remaining 90% is called the “Invisible Web,” or “Cloaked Web,” or “Deep Web.”
This means that there are huge amounts of data that are publicly available, but remain hidden from the search engines that everyone knows about.
It may be hard to understand how billions of websites can't appear in Google's search results. But they do. The 'spiders' that crawl and archive the world wide web have limited capabilities.
To better understand, let's start with some numbers about the size of the services offered by: Google.com, Yahoo.com, Cyberatlas, and MIT. The statistics are from the summer of 2013:
Google.com has 40 billion public web pages in its archives. 100+ billion of them are static web pages and publicly available. These pages can be easily found by Google, but also other search engines.
11+ billion static pages are hidden from the public, having declared that they contain private content, or are on the intranet. These are corporate pages that are open only to employees of the specific companies.
450+ billion pages have databases that are completely invisible to Google. For example, government databases with tax information, etc.
Google is considered to have the best database in searches today. The company's spiders crawl millions of web pages every week.
So, if Google has only stored 8-10% of the World Wide Web and other search engines have even smaller databases, then where is the remaining 92% of the content on the internet hiding?
The “Invisible Web” (or “Deep Web” or “Cloaked Web”) is the content that is not displayed in search engines.
More specifically: the Invisible Web consists of 220+ billion web pages that have not been stored as static web pages. The Invisible Web consists of on-demand pages and databases. That is, pages that exist only as reports of changing data. As of August 2007, robot spiders had not advanced enough to read these private databases. Only humans can access them, and only if they have the knowledge.
Technical terminology:
“Spider”: An artificial intelligence program, or robot, that is sent to read millions of static web pages on the public Internet. The information collected by Spiders is stored in databases, which are used by search engines.
“Database-Driven Web Content”: Web pages that exist only temporarily, and are created only when readers request answers from a large database. These temporary web pages are dynamic, and usually cannot be saved in bookmarks. They usually have extremely long URLs.
The Invisible Web contains Dynamic Web Pages. This means that a database creates a temporary page for you to answer your question! Good, huh?
How can I use the Invisible Web?
There are many people asking exactly the same question. Let's look at some notable databases below.
Humanities
Voice of the Shuttle: Launched in 1994, it is one of the oldest and largest humanitarian databases on the Web.
Special US government bases
University of Michigan Government Documents Center: You'll find a wealth of data, research, statistics, and more from the highest levels of the U.S. government. Databases offered include Arts, Health Sciences, Social Sciences, and International Studies.
USA.gov: A one-stop portal for many agencies of the United States government. Includes government positions, services, and information on finding grants, loans, and financial aid.
Health and Science
PsycNET: Use the American Psychological Association's database to find excerpts and entire journals on various psychology topics.
Scirus: A search tool dedicated exclusively to scientific information. This amazing search tool has hundreds of millions of scientific and academic documents to help researchers from all over the world.
Healthfinder: Contains information from over a thousand different health databases on the internet.
RXList: If you're looking for reliable information about medications, then this database is for you.
Mega Portal
The University of California, Riverside maintains InfoMine, an incredible source of knowledge that at last count contained over 100,000 connections and access to hundreds, if not thousands, of databases.
There are many websites that have been set up to bring data from the Invisible Web to the surface. CompletePlanet.com is one of them. It contains “over 70,000 databases.”
Most of the information about the invisible web is maintained by academic institutions. There are “academic portals” that can help you find this information. To find almost any educational resource on the web, simply type the following term into your favorite search engine:
site:.edu “topic I am looking for”
Your search will only return results related to edu sites. If you want to search for something from a specific university, use the university's URL in your search:
site:www.penepistimio.gr “topic I am looking for”
This is just the tip of the iceberg. Everything we have mentioned in this article is just beginning to touch on the vast resources available on the Invisible Web. As time goes by, the Invisible Web becomes larger.
