Skip to content

Google indexing: what it is and why a page stays out of the index

Author: Matteo Pellegrini

Indexing is the stage in which Google analyses a page it has already downloaded and decides whether to store its information in its index, the database it draws results from when someone searches.

Almost every problem that gets called an "indexing problem" inside a company belongs to a different stage. Either the page has never been downloaded, or it is already in the index and Google has decided not to show it anyway. The two situations can be told apart in Search Console and are fixed in opposite ways, so it pays to separate them before touching the site.

Crawling, indexing and serving are not the same thing

The Google Search Central documentation describes three stages and warns that not all pages make it through each one. First crawling, meaning Googlebot downloading text, images and videos. Then indexing, meaning the analysis of that content and its storage in the Google index. Finally serving, meaning the choice of which pages to show for a specific query.

During indexing Google also does something few people expect: it groups pages with similar content found across the web and picks one as canonical. The others stay in the cluster as alternate versions. If you have three URLs saying the same thing, there is no guarantee the one you care about will be chosen.

StageWhat Google doesWhere to checkWhat blocks it
CrawlingDownloads text, images and videos with GooglebotCrawl stats, URL Inspection toolrobots.txt rules, server errors, pages behind a login
IndexingAnalyses the content, chooses the canonical page, stores it in the Google indexPage indexing reportnoindex meta tag, low quality, duplication
ServingDecides which pages to show for a queryPerformance report, impressionsPoor relevance, low quality, robots meta tag rules

Rendering deserves a line of its own. During crawling Google runs the JavaScript it finds, using a recent version of Chrome. If a page's main content only appears after the scripts have run, that page depends on one more step than a page served as HTML.

Indexed does not mean you will see it in the results

This is the point in the official documentation that gets quoted least: Search Console may report a page as indexed while you don't see it in search results. Google lists three causes: the content isn't relevant to users' queries, the quality is low, or a robots meta tag rule prevents it from being served.

The same document contains the premise that ought to end a lot of arguments: Google doesn't guarantee that it will crawl, index or serve a page, even if it follows the Search Essentials. Indexing is a decision made by the search engine, not a right you acquire by publishing.

What the Search Console statuses really say

The Page indexing report uses labels that look like synonyms and aren't. Two in particular are confused all the time, and lead to opposite actions.

Status in the reportWhat happenedFirst move
Discovered - currently not indexedGoogle knows the URL but hasn't downloaded it yet. Usually it tried and postponed the crawl because it seemed to be overloading the siteLook at server response times, not at the page content
Crawled - currently not indexedThe page was downloaded and Google decided not to store it. It may index it in the futureWork on the content. Google states there is no need to resubmit the URL
Indexed, though blocked by robots.txtThe block prevents crawling, not entry into the index: the URL got in through links pointing to it from other sitesRemove the block from robots.txt and put a noindex on the page
Duplicate without user-selected canonicalGoogle chose another page as canonical and doesn't serve this oneDeclare the canonical or genuinely differentiate the content

One limit of the report to know before building an audit on it: the list of example URLs stops at 1,000 items, and Google states that not every URL with a given status is guaranteed to be shown even when there are fewer than 1,000. On a 40,000-page e-commerce site that report is a sample, not an inventory.

How to request indexing, and what doesn't speed it up

For a few URLs you use the URL Inspection tool, which requires an account with full access to the property. For many URLs you submit a sitemap. On the dedicated page Google gives a wait of "a few days to a few weeks" and adds two things worth repeating in full: there is a quota for submitting individual URLs, and requesting a recrawl several times for the same URL won't get it crawled any faster.

It is the most common habit and the most pointless. Resubmitting the same address every morning for two weeks uses up quota without moving anything.

IndexNow exists, but Google isn't part of it

IndexNow is an open protocol that notifies search engines when a URL is added, updated or deleted. It works, it is free and it can be switched on from almost any CMS. The official documentation lists the endpoints of the participating search engines: the global IndexNow endpoint, Amazon, Bing, Naver, Seznam.cz, Yandex and Yep. Google is not on that list.

It is worth saying because you often read the opposite. Installing IndexNow in the hope of getting indexed faster on Google means optimising for the other search engines, which makes sense but is a different thing. For the record: the protocol accepts up to 10,000 URLs per POST request, and the verification key must be between 8 and 128 characters long.

Crawl budget concerns few sites, and Google says which

In its crawl budget guide Google says who it is for, and the list is short: sites with more than a million unique pages whose content changes about once a week, sites with more than 10,000 unique pages whose content changes daily, and sites with a large share of URLs classed as "Discovered - currently not indexed". Outside those three cases the guide says you don't need to read it.

Two mechanisms are still worth knowing below those thresholds. The crawl capacity limit drops when the server responds with 5xx errors or an HTTP 429, and rises again when response times stay stable: hosting that struggles is paid for in lost crawls. And supporting HTTP 304 (Not Modified) tells Google to reuse the cached version when the page hasn't changed, saving bandwidth on both sides.

When the problem isn't technical

We are often called in for an indexing problem and find every page properly in the index. The Macropix case, a Milan-based LED screen manufacturer, is the one we like to use: on the Italian keyword "monitor pubblicitario" (advertising display) the page went from position 88 to position 2. Position 88 is page nine of Google, meaning zero traffic, but the URL had been indexed for years. What moved it was not a line in the robots.txt file, it was a page written on a subject the company really had something to say about.

Today, on that group of keywords in Italy, macropix.it has a 25.55% share of voice and sits ahead of amazon.it, at 18.08% (Semrush, July 2026). That is why in an SEO project the technical diagnosis comes first, but it is almost never the part that moves the numbers.

Being in the index and being cited are no longer the same thing

For twenty years "being in the index" and "being reachable by searchers" were the same. With answers generated by AI engines the two conditions are separating: a page can be in Google's index and never be picked up in an answer, and a piece of content can be cited by an assistant that draws on sources the classic index treats as secondary. If you only check indexing status, you are watching the traffic light on one road, and traffic has started to go other ways too.

Frequently asked questions about Google indexing

How long does Google take to index a new page?

Google's documentation gives a wait of a few days to a few weeks after a crawl request. There is no guarantee: Google states that it doesn't guarantee it will crawl, index or serve a page.

What is the difference between indexing and ranking?

Indexing means the page is stored in the Google index. Ranking is about the position it holds when someone searches. A page can have been indexed for years and sit on page nine, where traffic is zero.

Does resubmitting the same URL several times speed up indexing?

No. Google says explicitly that requesting a recrawl several times for the same URL won't get it crawled any faster, and that there is a quota for submitting individual URLs with the URL Inspection tool.

Is robots.txt enough to keep a page out of Google?

No. The block prevents crawling but not entry into the index: if other sites link to that address, Google can index it using information from the linking page. To really exclude it, remove the block and add a noindex.

Does IndexNow help get pages indexed on Google?

No. The endpoints listed in the IndexNow documentation are the protocol's global one, Amazon, Bing, Naver, Seznam.cz, Yandex and Yep. Google doesn't take part, so IndexNow doesn't speed up indexing in Google Search.

Matteo Pellegrini

Matteo Pellegrini

I’m a Business Developer, and at Visilay I focus on developing data-driven SEO, Google Ads, and CRO strategies. I love historical museums, have been practicing Karate for as long as I can remember, and on weekends I enjoy exploring Italian villages in search of authentic local food.