You’re searching a search engine’s index of web pages. When you type a query into a search engine, you’re not directly searching the internet for matching results. This is essentially a digital library of trillions of webpages where Google’s search results come from. Indexing is where processed information from crawled pages is added to a big database called the search index. That said, some processing occurs before and after rendering-as you can see in the diagram.
Google has to render pages to fully process them, which is where Google runs the page’s code to understand how it looks for users. Nobody outside of Google knows every detail about this process, but the important parts for our understanding are extracting links and storing content for indexing. Processing is where Google works to understand and extract key information from crawled pages.
If you have a large website, it could take a while for search engines to fully crawl it. This is important because it means that search engines might crawl and index some of your pages before others. Google queues URLs for crawling based on a few factors, including: It’s important to note that Google doesn’t always crawl pages in the order they discover them. CrawlingĬrawling is where a computer bot called a spider (e.g., Googlebot) visits and downloads the discovered pages. Google also allows submissions of individual URLs via Google Search Console. If you submit your sitemap to Google, it may help them discover your website faster. Sitemaps list all of the important pages on your website. Our crawler is the second most active after Google,so you should see a reasonably complete view of your backlinks here.
You can see your website’s backlinks for free using Site Explorer with Ahrefs Webmaster Tools. If someone adds a link to one of your pages from one of those web pages, they can find it from there. Google already has an index containing trillions of web pages. Google discovers these through various processes, but the three most common ones are: From backlinks Step 1. URLsĮverything begins with a known list of URLs. There are other types of search engines like Amazon, YouTube, and Wikipedia that only show results from their website. The process below applies specifically to Google, but it’s likely very similar for other web search engines like Bing. Google is the search engine that most SEO professionals and website owners care about because it has the potential to send more traffic their way than any other search engine. If you can rank high for these queries, you’ll get more clicks and organic traffic to your content. Understanding how search engines find, index and rank content will help you to rank your website in organic search results for relevant and popular keywords. Why should you care how search engines work? More users means more ad clicks and more revenue. This is known as pay-per-click (PPC) advertising. You can pay to be here.Įach time someone clicks on a paid search result, the advertiser pays the search engine. Organic results from the search index.Search engines have two types of search results: That’s how they obtain or maintain market share-at least in theory. What is the aim of search engines?Įvery search engine aims to provide the best, most relevant results for users. Computer program(s) that rank matching results from the search index.Įxamples of popular search engines include Google, Bing, and DuckDuckGo. A digital library of information about web pages. Search engines are tools that find and rank web content matching a user’s search query.Įach search engine consists of two main parts: Before we get into the technical stuff, let’s first make sure we understand what search engines actually are, why they exist, and why any of this even matters.