Crawling is the process by which a search engine discovers the address of a page, requests it from the server and downloads its content. It is the first of the steps that take a page into the search results.
The thing to know before you touch anything: a page that has not been crawled cannot be indexed, but a page that has been crawled does not automatically enter the index. They are two separate problems, they show up in two different Search Console reports and they are fixed in different ways. Mixing them up is why you sometimes see 200-page websites busy optimising their crawl budget while the real problem lies elsewhere.
What happens when Googlebot visits a site
Googlebot is the generic name for two crawlers: Googlebot Smartphone and Googlebot Desktop. For almost every site Google indexes the mobile version of the content, so most requests come from the mobile crawler and only a minority from the desktop one. Both obey the same token in robots.txt, so you cannot write different rules for each of them: you do come across files that try, and they do not work.
Three separate things happen within a single visit. The crawler discovers a URL, in most cases by following a link on a page it has already crawled. It requests that URL from the server and downloads the response. It then hands what it has collected to the indexing queue, where the page may be rendered, that is, opened with its JavaScript executed, and evaluated. Rendering is a later stage with its own queue: that is why a page can show as crawled and still contain parts Google has never read. For who actually visits your site, and how to check that a request really comes from Googlebot and not from something pretending to be it, see the entry on web crawlers.
The crawl limits Google puts in writing
These are not estimates from the SEO community: they are in the Google Search Central documentation and in RFC 9309, the IETF standard that has defined the Robots Exclusion Protocol since 2022. Two values were updated in February 2026, so they are worth checking again even if you think you know them.
| Limit | Stated value | Where it is written |
|---|---|---|
| Portion of an HTML file fetched | first 2 MB, the rest is not downloaded | Google Search Central, Googlebot page |
| Portion of a PDF fetched | first 64 MB | Google Search Central, Googlebot page |
| Average access frequency | no more than once every few seconds, for most sites | Google Search Central, Googlebot page |
| Portion of robots.txt parsed | 500 KiB, the rest is ignored | How Google interprets the REP |
| robots.txt cache lifetime | up to 24 hours, longer after timeouts or 5xx errors | How Google interprets the REP |
| Redirects followed on robots.txt | at least 5, then the file is treated as a 404 | How Google interprets the REP |
The 2 MB limit applies to uncompressed data and applies separately to each resource referenced in the HTML: a JavaScript bundle or a stylesheet above that threshold is truncated exactly like a page. When the limit is reached, Googlebot stops the download and sends only the part it has already fetched on to indexing.
How Google discovers a new URL
The documentation is clear: Googlebot finds new URLs mainly through links on pages it has already crawled. The sitemap is a support, not a substitute. A page that is in the sitemap but that no other page on the site links to is discovered with low priority and can sit there for weeks. This is why internal links are first of all a crawling lever, and only after that a ranking lever.
On the lastmod tag there is a simple rule: Google recommends including it, but a lastmod that changes with every deploy while the content stays the same stops counting as a signal. On sites that change often there is a more neglected lever, the 304 (Not Modified) status code. It tells Google the page has not changed since the last request, lets it reuse the copy it already has and frees up requests for the URLs that really have changed. In the Crawl Stats report, 304 appears among the valid response codes, not among the problems.
When robots.txt does not respond, Google stops crawling
The weakest point in the whole chain is the file that is supposed to govern it. Google accepts three kinds of response for robots.txt: a 200 with any file at all, even an empty one or one with syntax errors, and a 403, 404 or 410, which simply mean the file does not exist. Everything else is a connection problem, and the Search Console documentation describes in detail what happens next.
| How long robots.txt has not returned a valid response | What Google does |
|---|---|
| Less than 24 hours since the last successful fetch | uses the cached file and crawls normally |
| First 12 hours of unavailability | stops crawling the site, keeps requesting the file |
| From 12 hours to 30 days | resumes crawling using the last robots.txt fetched successfully |
| Over 30 days, home page reachable | behaves as if robots.txt did not exist and crawls without restrictions |
| Over 30 days, home page unreachable | stops crawling the site |
The last two rows deserve attention: a robots.txt that stays broken long enough does not block the site, it opens it up. All the rules you thought you had set stop applying and the sections you had excluded become crawlable again.
How widespread is the problem? Data from the HTTP Archive Web Almanac 2025, collected on millions of sites worldwide, shows that 85% of robots.txt requests return 200, 13% return 404, about 1% time out and 0.1% return a 5xx error. That 0.1% is a small share, and it is exactly the case in which crawling stops without anyone noticing, because the site keeps working for users.
The other brakes are less dramatic and much more common. 5xx errors and 429 responses on pages lower the crawl capacity limit Google calculates for the site. Redirect chains use up requests, and in the report each hop is counted as a request of its own: if page 1 redirects to page 2, which redirects to page 3, Google records three requests. A worsening average response time reduces the number of pages fetched in the same window.
Crawl budget matters to far fewer sites than talk about it
Google states precisely who its crawl budget guide is for: sites with more than a million unique pages whose content changes about once a week, sites with more than 10,000 unique pages whose content changes daily, and sites with a large share of URLs classed as "Discovered - currently not indexed". For everyone else the same page suggests keeping the sitemap up to date and checking the Page indexing report. Nothing more.
The threshold stated for the Crawl Stats report is even lower: Google writes that if a site has fewer than a thousand pages, you should not need to look at crawling at that level of detail. The report is also only available for root-level properties, so a property set up on a subfolder does not show it at all.
The budget is the sum of two components. The capacity limit depends on server health: it rises if the site responds reliably and falls if it slows down or returns errors. Crawl demand depends on popularity, update frequency and the perceived quality of the pages. You work on the first through infrastructure, and on the second by reducing the number of useless URLs exposed to the crawler, starting with filters, parameters and duplicate content. The order in which to tackle this work is in the technical SEO guide.
Three things people keep repeating that the documentation contradicts
First: Google does not support the crawl-delay directive in robots.txt. Four fields are recognised, user-agent, allow, disallow and sitemap, and the documentation says explicitly that the others are ignored. If you need to reduce the crawl rate for a few hours or a couple of days, Google says to return 500, 503 or 429, and warns against keeping it up for more than two or three days because the signal becomes permanent.
Second: returning 403 or 404 to Googlebot to slow it down does not work. 4xx codes other than 429 have no effect on the crawl rate and instead lead to the content being removed from Search. The reverse also holds: a 404 on a page that has really been removed is the right response, as the entry on the 404 error explains.
Third: blocking a URL in robots.txt does not take it out of the index. If the page is blocked, the crawler cannot read the noindex you have put on it, and the address can keep appearing in the results without a description. The two directives act on different stages: one on crawling, the other on indexing.
One figure from the Web Almanac describes how crawling is managed better than any guide: 97.5% of robots.txt files are under 100 bytes, 77% contain only the wildcard, and googlebot is named explicitly in about 6% of files. For almost every site, then, nobody is managing crawling: it is whatever the CMS generated on installation day, in a file nobody has opened since. If your site is large enough for this to affect revenue, opening that file and reading it is the first check we make in every SEO project.
Crawling FAQs
Crawling is the crawler fetching the page. Indexing is the later decision to store it in the index. Without crawling there is no indexing, but crawling on its own guarantees nothing: in Search Console the status Crawled, currently not indexed describes exactly this case.
Google writes that for most sites Googlebot should not access the site more than once every few seconds on average. The actual rate depends on the capacity limit calculated from server health and on crawl demand. The Crawl Stats report shows the trend for the site over the period selected.
The URL Inspection tool in Search Console lets you request a crawl, with no guarantee on timing. In practice it matters more to link the page from URLs Google already visits often and to include it in the sitemap. Google states that you cannot request an increase in crawl rate.
According to Google's guide it concerns sites with more than a million unique pages updated weekly, sites with more than 10,000 pages updated daily and sites with many URLs in the Discovered, currently not indexed status. Below these thresholds Google itself advises simply keeping the sitemap up to date and checking the indexing report.
No. The supported fields are user-agent, allow, disallow and sitemap; the others are ignored. To reduce the crawl rate for a short time Google says to return 500, 503 or 429, without keeping it up for more than two or three days.
