Noindex is the rule that tells a search engine not to show a page in its results. It is written in one of two places: a meta tag in the HTML <head>, or an X-Robots-Tag HTTP header.
It works on one condition only, and it is the one that fails most often: the crawler must be able to fetch the page. If the same URL is blocked in robots.txt, Googlebot never reads the rule and the address stays in the index, sometimes for months. Almost every case of "I added noindex and nothing happened" ends up here.
The two ways to write it
The first is the meta tag, which applies to HTML pages:
<meta name="robots" content="noindex">The second is the HTTP header, the only option for PDFs, images, videos and any file that has no <head>:
HTTP/1.1 200 OK
X-Robots-Tag: noindexThe effect is identical, as the Google Search Central guide says. The details differ, and in practice they are what makes the difference.
The value of the name attribute decides who you are talking to. With robots you address all crawlers; to address Google only there are two tokens, and just two: googlebot for text results and googlebot-news for Google News. Any other value is ignored. If you write different rules for different crawlers, Google applies the sum of the negative rules, and in a conflict the most restrictive always wins.
Two shortcuts that hardly anyone uses and that save lines of code: none is equivalent to noindex, nofollow, and unavailable_after with a date removes the page from the results from that moment on, handy for promotions and events with a known end date. One last detail from the robots meta tag specification: Google does not require the tag to be in the <head> and will honour a robots meta tag found in the <body> too. Other engines do not guarantee the same, so the correct place is still the head.
Five ways to remove a page from Google, and what they really do
The list below is the one Google itself published in 2019 as the alternative to noindex written in robots.txt. Two columns matter, and they are different from each other: blocking crawling and removing from the index.
| Tool | Blocks crawling | Removes from the index | When it is the right choice |
|---|---|---|---|
| noindex (meta tag or X-Robots-Tag) | No | Yes, at the next crawl | The page must stay online but out of the results |
| Disallow in robots.txt | Yes | No: the URL can stay in the SERP without a description | Saving crawl on areas nobody needs to read |
| rel=canonical | No | No, it consolidates signals on another URL | Two addresses with the same content |
| 404 and 410 status codes | No | Yes, when the URL is recrawled | The page no longer exists |
| Password protection | The crawler gets an error and does not see the content | Yes | Content nobody should open |
| Removals tool in Search Console | No | Yes, but temporarily | Urgent cases, while one of the rows above takes effect |
A 404 or 410 removes a page from the index just as noindex does, with the difference that it declares the page dead. rel=canonical removes nothing from anywhere: it consolidates. It is the most expensive misclassification on the list, because anyone using it to deindex is waiting for an effect that will never come.
Why noindex and Disallow together do not work
The two rules operate on different levels. Robots.txt governs crawling, noindex governs the index, and the second sits inside the page the first stops the crawler from fetching. Google says so plainly on its debugging page: among the reasons it cannot see the tag, it lists the robots.txt file "blocking the URL from Google web crawlers, so they can't see the tag".
Until 2019 the confusion also had a technical root: you could write a Noindex: line inside robots.txt, never documented but honoured by Googlebot. On 2 July 2019 Gary Illyes announced the retirement of all the code that handled those unpublished rules, effective from 1 September 2019. The reason given in the post is a data point: analysing robots.txt files across the web while open-sourcing the parser, the use of those lines was contradicted by other rules in all but 0.001% of files.
The correct sequence for getting a URL out of the index has four steps, and the order is not negotiable: remove the Disallow covering that address, add noindex, wait for the crawler to come back, and only once the page has gone put the Disallow back if you need it for crawling. Anyone who swaps the first two steps is left with the page in the SERP and no way of noticing, other than looking at the Page indexing report.
How long a page takes to disappear
There is no guaranteed time, because noindex takes effect at the next useful crawl. Google's documentation is blunt about the worst case: "Depending on the importance of the page on the internet, it may take months for Googlebot to revisit a page." On a page linked from the homepage and updated often, a few hours may be enough.
Two tools shorten the wait and one confirms it. URL Inspection in Search Console lets you request a recrawl of a single page and, more usefully, see the HTML Googlebot actually received: if the tag is in your editor but not on that screen, the problem is further down the line, usually a cache or a plugin rewriting the head. The Page indexing report has a dedicated entry, "Excluded by 'noindex' tag", which is the only reliable list of what has really gone. If you are instead trying to work out why your site is not showing on Google, that entry is the first one to open.
Three things noindex does not do
It does not hide the page. The URL still returns 200 and anyone who knows it can open it: noindex speaks to search engines, not to the web. Content that must stay private needs authentication. It is worth reading how the IETF deals with the same misunderstanding about robots.txt: RFC 9309, which since 2022 has standardised the robots exclusion protocol Martijn Koster wrote in 1994, closes its introduction with a single line, "These rules are not a form of access authorization", and in its Security Considerations adds that listing a path in robots.txt makes it publicly discoverable.
It does not save crawl budget. For the rule to be read the page has to be fetched, so a noindexed URL uses crawl exactly like the others, at least until Google decides to visit it less often. Anyone adding noindex to thousands of filter pages in the hope of lightening the crawl is using the wrong tool.
It is not the answer to duplicate content. On two near-identical pages, noindex throws one away instead of combining the signals: for duplicate content the route Google recommends is still the canonical. Noindex makes sense when the page should not exist in the results at all, not when it is the less important copy of another.
Noindex does not speak to AI model crawlers
Google's specification says so in a line almost nobody quotes: the noindex rule is addressed to search engine crawlers, and to block crawlers that are not search engines you need rules aimed at that specific user agent. In practice, a robots meta tag has no effect on anyone collecting text to train or feed a language model, because those crawlers are controlled from robots.txt, which is a different file and a different level.
Google handles this with a separate token, Google-Extended, which does not even have its own user agent in HTTP requests: it exists only as a line in robots.txt and decides whether content already crawled can be used to train Gemini models and to ground answers in Gemini and Vertex AI. The list of Google crawlers states that Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. They are two separate dials, and turning one does not move the other.
OpenAI takes the same approach with three distinct bots, documented on its crawlers page: GPTBot for training, OAI-SearchBot for appearing in ChatGPT search answers, and ChatGPT-User for visits started by a user, where, since they are user-initiated, "robots.txt rules may not apply". None of the three reads your meta robots. Changes to robots.txt, OpenAI warns, take around 24 hours to be reflected in search results.
| What you want to prevent | Where to write it | What does not work |
|---|---|---|
| The page must not appear in Google Search | noindex in the meta robots or X-Robots-Tag | A Disallow in robots.txt on its own |
| The text must not be used as direct input for AI Overviews and AI Mode | nosnippet or max-snippet:0 | noindex, if you still need the page in the SERP |
| The content must not train Gemini models | user-agent: Google-Extended in robots.txt | Any meta tag on the page |
| The content must not train OpenAI models | user-agent: GPTBot in robots.txt | Any meta tag on the page |
| The site must not appear in ChatGPT search answers | user-agent: OAI-SearchBot in robots.txt | noindex |
The AI Overviews row deserves a note, because it is the only point where a meta tag really touches generative answers. Google's specification says that nosnippet and max-snippet apply to all forms of search results, AI Overviews and AI Mode included, and that they prevent or limit the use of the content as direct input for those answers. If you want to stay in the SERP and out of the generated summaries, that is where you work, not on noindex: it is one of the choices we deal with in our answer engine visibility projects.
A data point on the state of the discussion: on Google UK's first page for "noindex", checked on 26 September 2026, none of the seven results other than Google's own documentation mentions Google-Extended, GPTBot or OAI-SearchBot. Two of them name Bing, and Wikipedia also covers a Yandex tag, which shows how firmly the map of crawlers you find around is still stuck in 2019.
Where to set it without touching code
On WordPress the most dangerous switch is in Settings > Reading: the "Discourage search engines from indexing this site" box adds noindex to every page on the domain. It is meant for development environments and stays ticked after launch more often than seems plausible. If organic traffic collapsed suddenly after a release, that box is the first place to look, before any other theory about the traffic drop.
On a single page, Yoast SEO has the option in the Advanced tab, under the question "Allow search engines to show this Post in search results?": answering No adds noindex. Rank Math puts it in Titles & Meta with separate boxes for index, follow, archive and snippet, so with finer control. For non-HTML files you go through the server: one rule in .htaccess on Apache or in the configuration file on NGINX applies X-Robots-Tag to every PDF on the site in a single line.
The periodic check is done by a crawler: Screaming Frog, or any spider that exports the indexability column, shows at a glance which URLs have noindex and which also have a Disallow above them. It is one of the fixed steps of an SEO audit, and one of the few that finds serious errors in minutes.
One last ratio, which says something about the priorities of people studying this subject. In the UK "robots.txt" is searched 1,600 times a month on average, "noindex" 140: more than eleven to one in favour of the tool that removes nothing from the index (Google Ads data via DataForSEO, UK, twelve-month average to August 2026). On the sites we end up checking the ratio shows: robots.txt files tended line by line, and nobody who has ever opened the Page indexing report to find out which pages have really left the results. It is the kind of imbalance that half a day of SEO work fixes and that few people bother to look at.
Noindex FAQs
No. Noindex says not to show the page in search results, nofollow says not to follow the links on the page. They are independent rules and can be combined: the value none in the robots meta tag is exactly equivalent to writing noindex, nofollow.
It depends on when Googlebot comes back to crawl it. Google's documentation warns that for a less important page it can take months. With the URL Inspection tool in Search Console you can request a recrawl of the single page and check the HTML the crawler received.
No. The page keeps responding normally and anyone who knows the address can open it. Noindex removes it from search results, not from the web. Private content needs authentication, password protection or moving the files out of the public folder.
They do two different things: Disallow blocks crawling, noindex removes from the index. A URL blocked in robots.txt can stay in the results without a description, because Google knows it exists from links. On the same URL the two rules cancel each other out: the blocked crawler does not fetch the page and so never reads the noindex.
No. The robots meta tag is addressed to search engine crawlers. Language model crawlers are controlled from robots.txt with dedicated tokens: Google-Extended for training and grounding Gemini answers, GPTBot and OAI-SearchBot for OpenAI. To limit the use of your text in AI Overviews while staying in the SERP, use nosnippet or max-snippet instead.
