If you've been on the internet for long enough, chances are you've heard about the mysterious deep web. It's likely you've read or listened to stories of live-streamed killings, human trafficking, child pornography and anonymous marketplaces for drugs. From Ulbricht's Silk Road to Scully's Daisy's Destruction, these stories serve to create an aura of darkness and morbid curiosity around several privacy technologies.
The only problem with the stories that have caused us to buy a VPN subscription or cover our webcams is that most of them are fake. Most of the stories you'll hear on YouTube or read on r/nosleep are completely fabricated for the sake of entertainment. In fact, most of what the non-technical user and the mainstream media knows about the deep web is completely false.
There have already been many articles challenging many assumptions of what the deep web actually is and what you can find on it. Despite this fact, I'd like to cover the subject matter and hope to fill in anything these other articles may have missed. I also personally enjoy talking about this since I've had a keen interest in the deep web for over 5 years.
There are many urban legends and creepypastas written to make the internet seem a lot more mysterious than it is; much like how stories are told in the dark for the thrills and chills we crave. This revelation may be hard for you to accept, since it takes away a lot of the excitement and fascination surrounding the deep web legends. Personally, I wish the stories were true, because I love thrillers and horror. Unfortunately for both of us, it's all fake.
What is the truth behind the deep web mythos?
The Deep Web is literally only sites that are not indexed by search engines; nothing more, nothing less. We just called everything the web before we had a term for what was indexed and what wasn't; and that's really all it is. Until your site is indexed on Google or such, your site is technically part the deep unindexed web—from the definition I just gave. Many sites don't want some or all of their pages to be indexed by search engines for legitimate privacy concerns, such as their admin panel.
When you see statistics such as "Google has only indexed 0.4% of the content online", you have to think: "On what metric do we measure the internet?". The simplest way to measure the internet is saving and counting the pages. Websites commonly have anywhere from 10 to millions upon millions of pages. Sites such as Amazon have a page for every product, Facebook has pages for every person, and PornHub has a page for every video and account. Logically, the next questions that follow are inspired by the curiosity of what lies inside the other 99.6% of the internet.
According to this site, the part of the World Wide Web that is indexed by Search Engines as of the 26th of August, 2019 is 5.31 billion pages. There are many reasons why a search engine has not indexed or will not index parts of the internet—as opposed to voluntarily opting out with a "robots.txt". Google might have blacklisted a site because of illegal activity, the site might be dead or improperly configured, or the site might not have any backlinks. Since the majority of emails ever sent are spam, it is entirely possible that a lot of what isn't indexed is spam, scams or illegal porn.
Anyone who has done some research on the deep web have almost certainly seen the infographic that uses an iceberg to describe the internet. There are different variations of this infographic, some of which have 3, 5, or even 8 levels of the internet. Most commonly, the levels are along the lines of: the Surface web (aka the clearnet), the Deep Web, the Dark Web and sometimes The Marianas Web. Oh boy.
To put it bluntly, there are no levels of internet. The internet is not an ocean nor cave. Try asking around even on the most underground of hacking forums: "How do I set up a site on the Charter Web?", "How do I get a closed shell system?". No one can help you because a lot of what you see in those infographics do not exist. What the fuck even is a Gadolinium Gallium Garnet Quantum Electronic Processor or Polymeric Falcigol Derivation or The Primarch System? Quantum computing does not power the internet, and it never has.
These infographics have been created to confuse and troll people into thinking that the creepypastas are real or are made out of a lack of understanding of how the internet works. Whatever good intentions these infographics had originally to explain deep web intuitively, have been overshadowed by the damage they have caused as a result of being accepted as absolute fact. This has led to a fantastical mental image of the internet and will make you look like an idiot to other people online. Many of the terms that have been thrown into these infographics are just random kinds of content that usually shocks the public.
I advise thinking of the internet as a whole as a city. Any building in a city can be used as a commercial business, government owned property, a storefront for selling drugs, or a dungeon to rape and torture men and women. The criminal underground and black market is all around us. Anyone you know could be committing depraved acts: your neighbour, your father, your son, your daughter, your friends or anyone else. If you've ever taken a computer networking course or know even a little about networks, you'll understand why this analogy is intuitively plausible.
As these infographics fade into obscurity, we still see a widespread usage of the term "Dark Web". The dark web, that is understood by most people, does not exist; please stop using the term. Furthermore, if you've heard a lot of creepypastas, you've probably even heard about a deeper, darker web called "The Shadow Web". The shadow web does not exist and was made up for use in one creepypasta, sorry to crush your hopes and dreams.
What people are referring to when they use Deep Web, Dark Web, Dank Web, or Dark Net interchangeably is called a darknet. Darknets are overlay networks that require either special authorization or additional software to access. Darknets have existed since the late 90s to ensure freedom of speech worldwide. Naturally, people who had intent to commit illegal actions have used darknets since their inception. There is nothing inherently wrong with using a darknet, but illegal actions are unfortunately some of the more common uses.
The Onion Router, or TOR for short, is by far the most popular and streamlined darknet technology. TOR routes your internet traffic around and encrypts your TCP/IP packets with multiple layers of encryption, hence the similarity to an onion. You can route any Apache, NGINX, ExpressJS or any other Webserver through your TOR port to create a TOR hidden service accessible only through the TOR network.
Sites on darknets can be and are indexed, because they usually want to be. Don't ever let anyone tell you "Dark Web" sites can't be indexed, search engines for TOR actually do exist (eg. Torch and Ahmia). Selling drugs or guns anonymously online is a very lucrative business opportunity that needs customers, so webmasters of these sites will list their site's URL on The Hidden Wiki or other indexes. Many sites choose not to be indexed because they wish to remain exclusive to a small group of people. Some of these exclusive sites have been indexed but do not allow registration and will not allow you to access any content without an already existing account or voucher.
TOR or any other darknet is not completely anonymous and much of the illegal content has been taken down since the media deep web hypetrain. The NSA and FBI have their hands up TOR's ass. These organizations own nodes in the TOR network and do monitor activity that passes through them. There are also other darknets still in use such as I2P and Freenet.
I2P is essentially a peer-to-peer version of TOR. I2P is a darknet more frequently used for torrenting and chatting rather than selling illegal goods. I've heard people say that I2P is the real dark web since TOR is too mainstream; this statement is completely false. I2P users actually seem to be much more benign than TOR users, because of how every criminal and their mother knows about TOR while less know about I2P or Freenet. You can still find some banned movies such as Maladolescenza in P2P networks or darknets but the child porn has mostly disappeared.
Freenet was the most popular darknet before TOR became much more widely adopted. Freenet is now just shadow of what it used to be since there are nowhere near as many users as it used to have. A lot of child pornography, gore, and illegal information went through Freenet, but has since been lost to time. Unlike TOR, Freenet's limitations as a technology would never allow a web app with SQL or PHP, thus could never be used to create something like The Silk Road. If you go on Freenet today, you'll likely just find some early 2000s cyber-anarchy and cypherpunks.
In 2019, the hype is over. The golden age of hacking is long gone and the scary mysterious internet ship has sailed. We have entered into a new era, an era where we are all potentially thought-criminals. It is becoming more and more crucial to use the previously taboo technology in order to ensure your freedom and privacy online. Please support the TOR project and the Invisible Internet Project (I2P), your privacy online depends on them.
Illegal stuff can be found on Google:
Did you know that everything you can find on TOR and more can be easily found on the clearnet? The only difference is how long sites hosting illegal content stay online. Drugs are sold on Reddit, Discord and WhatsApp everyday; there are multiple pedophile communities on Wire and Reddit; sites such as BestGore showcase extreme images with gore; and sites like PornHub cannot always regulate child pornography. The realities of the darkness in humans is not contained to darknets that the general public can't normally access. The discussion of whether or not the staple urban legend of Red Rooms are real or not deserves an article dedicated to it.
In conclusion, I hope you now have a much better understanding of how the internet works at a high level. If you'd like to learn the nitty-gritty details of computer networking, I suggest looking at my Top 10 Essential Books for Self-Taught Computer Science article; I've recommended a very nice book on the subject. I wish you safe travels and happy hacking!