You may think the Internet is open to all – but beyond the regular web addresses like Google and Facebook, there is the Deep Web, a vast expanse of information many times larger than the regular, so-called Surface Web.
This web contains anything from corporate databases and scientific journals, to websites that allow you to buy drugs or hire hitmen. And it’s all inaccessible through search engines, whether by design or by accident.
You can actually observe three different categories of this invisible Internet, (although the lines are often blurred): the Deep Web, Darknets, and the dark Internet.
To build their website directories, search engines used programs called spiders or crawlers, which go through websites, index the information they find, and then add any hyperlinks to the list of sites they will crawl next.
The Deep Web is defined as the sites that are publicly available, but are not picked up by these crawlers, according to a white paper by BrightPlanet, a company that specializes in finding information on the Deep Web.
Sites on the Deep Web include databases of corporate and consumer information, government documents, websites for internal use, and password-protected sites.
No reliable estimate exists of the current size of the Deep Web, but in another white paper by BrightPlanet, published back in 2001, estimated it to be 500 times as large as the surface web.
But according to a speech by then-CEO of Google, Eric Schmidt, the Internet was estimated to be about 5 million terabytes in 2004. Out of that, Google had indexed 170 TB, or 0.0034%.
Of course, as an immense network that is not centrally distributed, and which grows every day, it’s near impossible to chart out the whole Internet.
Probably closer to people’s idea of the Matrix-like Deep Web, darknets are where the controversy happens.
The term “darknet” was first popularized in 2002 by a paper on piracy written by four Microsoft engineers.
It refers to “a closed network [created] to communicate securely in a manner that defies detection or penetration by governments or corporations,” according to a 2011 paper by the University of Richmond’s Journal of Law & Technology.
They work by using non-standard protocols, sharing files and information on a distributed peer-to-peer network that is unavailable to the public, and are often very hard to detect.
This means that a lot of the networks are used for illegal purposes.
An infamous example was the Silk Road, a now defunct online marketplace for drugs where one could buy anything from weed to heroin and crack cocaine.
Another, still operational darknet, is Tor, which is short for “The Onion Network,” named after the layers and layers of encryption the network adds to any address distributed through the network.
A lot of shady services use the Tor network, including but not limited to hitmen for hire, gun stores, child pornography and bestiality sites, stolen credit card information, and fake IDs. Tor is indeed so shady, that Comcast has threatened to shut down users who utilize its often unscrupulous services.
A lot of the capacity of darknets are taken up by such illegal activities (which also include botnets, which are basically networks of computers taken over by viruses that can be used to send spam, denial-of-service attacks, and more), as reported by the MIT Technology Review (MITTR).
However, the point in MITTR’s piece notwithstanding, the networks are still being used to do good things, too, helping dissidents worldwide in countries such as China and Syria to communicate without being snooped on by their government.
The Dark Internet
Compared to the Deep Web and Darknets, the dark Internet is not actively being used. In fact, it’s more akin to a black hole.
While technically, the Internet is supposed to be all interconnected, there are some addresses that are only reachable through certain providers, and others that are not reachable at all, according to a paper by a University of Michigan professor.
Mostly, these addresses are either the result of competitive routing of networks by providers, or misconfigurations, but they can also be remnants of the old U.S. military network.
This dark space can also be used for malicious purposes, with hackers using the dark address space to launch DoS-attacks, according to the magazine Government Technology.
Have you ever accessed any of these hidden Internet services? Tweet us @curiousmatic