Google searches 1 trillion unique Web links at once

The first Google index had 26 million pages in 1998. In 2000 it reached 1 billion. Now when you search a keyword, Google will search 1 trillion unique URL's on the Web at once to get the best results.

Google's staggering index count shows how big the Web has become and how fast it is growing. To keep pace with the astounding growth, Google indexes all the unique links and sites on the Web and keeps it in its database. The number of sites Google has indexed has reached 1 trillion (1,000,000,000,000) unique links. When we type a word or words in Google, it searches those terms from the database stored in Google servers located around the world. According to Google, it searches the trillion links as follows:

We start at a set of well-connected initial pages and follow each of their links to new pages. Then we follow the links on those new pages to even more pages and so on, until we have a huge list of links.

Actual records showed that Google found more than 1 trillion individual links, but most of them were exact copies under different website names. Google eliminates all the duplicate content and keeps only the unique URL link. Google said the links are growing at an incredible pace, several billion pages per day. So, the next trillion links will reach faster. Google's main goals remains the same: to index the entire world's data and have the most comprehensive index of any search engine. Compared to its earlier days in 1998, the indexing by Google has become complex as well.

Back then, we did everything in batches: one workstation could compute the PageRank graph on 26 million pages in a couple of hours, and that set of pages would be used as Google's index for a fixed period of time. Today, Google downloads the web continuously, collecting updated page information and re-processing the entire web-link graph several times per day. This graph of one trillion URLs is similar to a map made up of one trillion intersections. So multiple times every day, we do the computational equivalent of fully exploring every intersection of every road in the United States. Except it'd be a map about 50,000 times as big as the U.S., with 50,000 times as many roads and intersections.

All this effort is just to get us the best result at the top of the page. It is no wonder that Google has a 70 per cent share in the search market.