Spiderman would be at home here! But for lesser mortals, the Invisible or Deep Web sounds rather daunting. A bit creepy too – if one has heard of it at all. So what’s the Invisible Web? We take you deeper into the Internet space.
Matters of size
As if the Web we know of isn’t already mind-bogglingly huge, it appears that there’s more to it than meets the eye. Back in 1994, Dr. Jill Ellsworth, Internet Consultant, coined the term “Invisible Web” to describe what many suspected – that a huge ocean of information lies out there, inaccessible to Messrs. Google, Yahoo and others of their ilk. To gauge its size, consider this: while the visible Web has around 250 billion pages (from which Google indexes around 8 billion), the Invisible Web could be 2 to fifty times bigger!
What’s in it? Why is it hard to access?
University libraries, rare manuscripts, government portals, archives, stock prices, and census data – these are but a few examples of what’s in there. Why are they so difficult to access? Internet search guru Chris Sherman explains: “The bulk of the invisible web is made up of databases ….To access information from a database, you need to interact with a form that tells the database what you want. Unfortunately, search engine crawlers cannot type. When a crawler runs into a web page that is the gateway to a database, it is as if it has run smack into the front door of a vast library with securely bolted doors.”
Some sites are blocked by publishers for business reasons. Specialized articles on scientific research or law, for instance, which may have a considerable cost attached to them, can only be accessed on payment.
Who needs it – and why?
We’ve all experienced the agony of sifting through pages of unauthenticated, irrelevant or dubious information on the Net. This is where Deep Web scores – it’s a primary source of high-quality information. Since academic institutions maintain most of what you’ll find here, it’s reliable and in-depth, offering insights that can’t be found in stuff thrown up on the more commercialized, visible Web. For students and all seekers of genuine information, this is akin to the Holy Grail.
Navigating the Invisible Web
Here are a couple of tricks to find your way around. If you were researching the topic “reproductive system of the three-toed sloth”, a regular Google search would throw up 15 pages. Way too much. Add “database” to the same search words and voila! You’ve tapped into the invisible Web, with around 8 pages of stuff from academic journals and university libraries – information that’s focused and citation worthy.
The query site: edu. will get you on to educational domains. Type site: edu Bollywood and treat yourself to an array of insightful, well-worded articles, which you’ll be hard pressed to find on the searchable Web (try it!).
Want a short cut?
Try some pathfinders. Gary Price and Chris Sherman’s Direct Search is highly acclaimed. There’s also InvisibleWeb.com, Resource Discovery Network from UK, the US Government’s FirstGov.gov, AlphaSearch, Librarians Index to the Internet. All of these serve as “gateways” to databases, helping you to narrow down your focus, save time and gain access to the best information there is on your subject of interest.
So what’re you waiting for? Get your Spidey suit on and leap in! Already an expert? Share with us how you extract treasures from the Invisible Web, your hits and misses.