Robots.txt is a plain text file that sits in the root of a website and tells search engine crawlers which URLs they may download and which they should leave alone.
What it does not do is decide what appears in search results. A URL blocked in the file can still turn up on Google, with no description under the title, if someone else links to it. To keep a page out of the index you need noindex, which is a different instruction and lives in a different place.
Only one possible location
The file is called robots.txt, all lower case, and it applies only to the exact combination of protocol, host and port it was fetched from. https://example.com/robots.txt says nothing about https://shop.example.com/, http://example.com/ or https://example.com:8181/: each of those addresses needs its own file. Google's specification lists the cases one by one, punycode and FTP included.
A robots.txt inside a subfolder does not exist as far as crawlers are concerned: none of them looks for it there. The format is plain UTF-8 text, and Google discards the lines it cannot read rather than throwing away the whole file. The size limit is 500 kibibytes, beyond which the rest is ignored, and the cached copy is kept for 24 hours at most.
The four lines Google actually reads
The syntax is field: value, one directive per line, with # for comments. Google's crawlers interpret four fields, and everything else is skipped.
| Directive | What it does | Example | |
|---|---|---|---|
| user-agent | Opens a group and says which crawler it is for | User-agent: Googlebot | Yes |
| disallow | Path that must not be downloaded | Disallow: /basket/ | Yes |
| allow | Exception inside a blocked path | Allow: /basket/guide.html | Yes |
| sitemap | Absolute URL of a sitemap, valid for all bots | Sitemap: https://example.com/sitemap.xml | Yes |
| crawl-delay | Seconds to wait between two requests | Crawl-delay: 5 | No |
| host, noindex | Preferred domain, exclusion from the index | Host: example.com | No |
Field names are not case-sensitive, values are: Disallow: /pricelist does not block /Pricelist. Two wildcards are allowed, * for any sequence of characters and $ to anchor the end of the URL, so Disallow: /*.pdf$ only catches paths that really end in .pdf and lets /datasheet.pdf?v=2 through.
When two rules contradict each other, the longer one wins, meaning the one that matches more characters of the URL. If they are the same length, the standard says to prefer allow. This is where hand-written files most often behave differently from what their author expected.
Then there is the case we see in half of all audits: a specific group beats the generic one, but does not inherit from it. If you write three Disallow lines under User-agent: * and further down open a User-agent: Googlebot block with a single line, Googlebot reads that one line and ignores the other three. The rules have to be repeated inside the specific group.
Blocking crawling is not removing from results
Google makes the point in its introduction to robots.txt: the file "is not a mechanism for keeping a web page out of Google". If other pages point to a blocked URL, Google can index it without ever downloading it, and shows it in the results with no description.
The reverse is trickier. If you put noindex on a page and then block the same URL in robots.txt, the crawler does not download the page, so it never reads the noindex, and the address stays in the index, sometimes for months. The two instructions do not add up: the second cancels the first.
The third effect is the one people notice much later: links inside a blocked page are not discovered. If an article can only be reached from a page under Disallow, as far as Google is concerned that article does not exist.
What happens when the server responds badly
This part is missing from almost every guide out there, and it covers the most expensive incident a robots.txt can cause.
| HTTP response to /robots.txt | Crawler behaviour |
|---|---|
| 2xx | The file is processed and the rules applied |
| 3xx | Google follows at least five redirects, then treats the request as a 404 |
| 4xx, except 429 | As if the file did not exist: no crawl restrictions |
| 429 | Temporary error, not a missing file |
| 5xx | Rules undefined: the crawler must assume the whole site is disallowed |
The row that matters is the last one. If the server returns a 500 or 503 to the request for /robots.txt, the standard requires the crawler to behave as if the entire site were disallowed, because the rules are undefined. This is not a partial block on a few folders: the whole site stops being crawled until the file becomes reachable again, or until enough time passes (the RFC gives thirty days as an example) for the crawler to treat it as simply absent.
That is why a robots.txt generated by a plugin, a middleware or a firewall rule that fails under load can stop crawling without anyone touching a line of configuration. When organic traffic drops and the site stops showing up with no obvious cause, the robots.txt status code is one of the first checks in a technical audit, before anyone looks at the content.
From 1994 to a real standard
The file was born in June 1994 out of a discussion on the robots-request mailing list, after a crawler had brought a server to its knees. The proposal came from Martijn Koster. For twenty-eight years it remained a convention: everyone followed it, nobody had written it down in a formal standard.
In 2022 the IETF published RFC 9309, Robots Exclusion Protocol, on the Internet Standards Track. Since then the protocol has had a formal ABNF syntax and mandatory rules on error handling, caching and parsing limits, which must be at least 500 kibibytes. The same document makes clear in one line what robots.txt is not: "These rules are not a form of access authorization".
A good deal of the material still in circulation predates it. robotstxt.org, which many guides still point to as the reference, reflects the 1994 version: no wildcards, no Allow, no Sitemap.
Who reads it today, besides Googlebot
Crawler traffic no longer comes only from search engines. On the Cloudflare network in May 2025, Googlebot accounted for 50% of requests in this category, up 96% in a year, and OpenAI's GPTBot had risen to 7.7%, up 305%. It is a global sample, but the order of magnitude shows which way things are going.
The adoption figures are less flattering. On 6 June 2025 Cloudflare found a robots.txt on 3,816 of the top 10,000 domains, just under four in ten. Of these, 546 (around 14%) contained a rule aimed explicitly at AI bots. The most blocked token was GPTBot, on 312 domains, followed by CCBot and Google-Extended.
Google-Extended needs a technical note, because it causes confusion: it is not a user agent you will see in your server logs. It is a token that only tells Google whether content may be used to train its models, while the requests keep coming from Googlebot. Anyone trying to match that name between robots.txt lines and access logs will never find it. llms.txt is a different matter: it is a separate proposal and replaces none of this.
How to check it now the tester has gone
The tool many guides still mention has not existed for years. Google retired the robots.txt Tester in November 2023 and replaced it with the robots.txt report in Search Console, under Settings.
The report shows the files found for the top 20 hosts in the property, when they were last fetched, their size in bytes and any parsing errors. It also keeps the versions fetched in the last 30 days, which is the most useful part: when someone uploads an old copy of the file by mistake, you can see the exact moment it happened.
To find out whether a single address is blocked, use URL Inspection. To test rules locally, Google publishes the open source robotstxt library on GitHub, the same parser used in Search; Screaming Frog lets you load a custom file and see what it would block before you publish it. Look at the report when you change the file, not every week: in our SEO checklist it sits among the quarterly checks.
For many sites, the best file is no file
A missing robots.txt returns a 404, and a 404 means no restrictions: Google crawls everything. For a forty-page site that is exactly the behaviour you want, and a badly written file does more damage than a missing one. You need one when there is really something to exclude: faceted navigation filters that multiply URLs on an ecommerce site, internal search results, baskets, comparison pages generated on the fly. In those cases the real work is deciding which addresses to cut, which is the first thing we look at when we take on a site's SEO; writing the lines takes five minutes.
One thing people rarely mention: the file is public and people read it too. Writing Disallow: /private-client-area/ does not hide that folder, it announces it to anyone who opens the address. NIST has said this for years in its server security guidelines, and it applies here to the letter: the security of a system cannot depend on the secrecy of its components.
Robots.txt FAQs
No. If there is nothing to exclude from crawling, not having the file is perfectly fine: Google reads it as no restrictions and crawls everything. A badly written robots.txt does more damage than a missing one.
No. It blocks the page from being downloaded, not from appearing in the results. A blocked URL that other sites link to can still show up, with no description under the title. To keep it out of the index you need the noindex meta tag or the X-Robots-Tag header, and the page must stay crawlable so the crawler can read them.
No. The only fields Google's crawlers interpret are user-agent, allow, disallow and sitemap. Other engines, including Bing and Yandex, still recognise it, each with its own interpretation. To reduce Googlebot's crawl rate you go through the Search Console settings.
Yes, and the same goes for every protocol and every port. The rules only apply to the exact host, protocol and port the file was fetched from: https://example.com/robots.txt does not apply to https://blog.example.com/ or to http://example.com/.
Usually less than 24 hours, which is the maximum lifetime of the cached copy. If you need it sooner, you can request a recrawl of the file from the robots.txt report in Search Console.
Under RFC 9309 the crawler must treat the rules as undefined and behave as if the whole site were disallowed. A prolonged server error on that single file can therefore stop the whole site being crawled, even if no directive has changed.
