A web crawler is a program that downloads a site's pages one after another by following the links it finds, to build a search engine's index or the dataset a language model is trained on. It is also called a spider, robot or bot.
If you run a website, the first thing you need to know is that the user agent a crawler presents proves nothing. Anyone can write Googlebot in the header of an HTTP request, and in part of the traffic you see in your logs someone actually does. Verification is done through DNS, in two steps, and we explain it further down.
From seed set to index
A crawler starts from a list of URLs it already knows, the seed set, usually taken from the previous index and from sitemaps. It downloads those pages, extracts their links and puts them in a queue. The structure that holds the queue is called the crawl frontier, and that is where it is decided what gets visited first, what gets revisited and what sits untouched for months.
No crawler downloads everything. The queue is ordered using signals that can be computed quickly across billions of documents: how many other pages link to a URL, how much traffic it gets, how often it had changed the last time it was visited. That is why a new page linked only from the menu can stay out of the index for weeks while the homepage is fetched again every day.
Coming back regularly is half the job, and it is expensive. Google handles it with HTTP caching: it sends If-None-Match or If-Modified-Since headers built from the previous response, and if the page has not changed the server answers 304 without sending the body again. Google's documentation recommends ETag over Last-Modified, because the date has to be formatted in a precise way and a hash does not.
The crawlers you will see in your logs
Google splits its clients into three families, and the difference is not cosmetic. Common crawlers, such as Googlebot, always follow robots.txt rules. Special-case crawlers, such as AdsBot, may ignore the user-agent: * group because there is an agreement with the advertiser. User-triggered fetchers, such as the site verification tool, ignore robots.txt on purpose: a person asked for the request.
| Crawler | Operator | robots.txt token | What it is for |
|---|---|---|---|
| Googlebot | Googlebot | Search index, smartphone and desktop versions | |
| GoogleOther | GoogleOther | Internal research and development, active since 2023 | |
| Google-Extended | Google-Extended | Control only: says whether content may train Gemini. No effect on ranking | |
| Bingbot | Microsoft | bingbot | Bing index |
| GPTBot | OpenAI | GPTBot | Collecting content to train models |
| ChatGPT-User | OpenAI | ChatGPT-User | Fetching a page in response to a prompt |
| ClaudeBot | Anthropic | ClaudeBot | Training and updating Claude |
| Meta-ExternalAgent | Meta | Meta-ExternalAgent | Data collection for Meta models |
| Bytespider | ByteDance | Bytespider | Data collection for the group's models |
Google-Extended deserves a line of its own because it is the only one on the list that does not exist as a string in HTTP requests. It is a token that lives only inside robots.txt and expresses a preference about training: the traffic keeps arriving with the normal Google user agents. Looking for it in your logs is a waste of time.
Who really visits your site, in numbers
Cloudflare sees a large slice of web traffic and publishes the counts. Comparing May 2024 with May 2025 across a cohort of more than thirty search and AI crawlers, Googlebot went from 30% to 50% of that cohort's requests, a 96% increase in absolute volume. GPTBot rose from 2.2% to 7.7%, which in raw requests means up 305%. Bytespider, the second most active AI crawler a year earlier, lost 85% of its requests.
These are global figures: Cloudflare does not publish a per-country breakdown of this series, so read them as an order of magnitude, not as a picture of your own domain. The number that applies to everyone is a different one, again from Cloudflare Radar: around 30% of global web traffic comes from bots, and in some countries it exceeds human traffic.
On the purpose of crawling, in the first week of August 2025 around 80% of requests from the AI bots monitored by Cloudflare, which publishes a breakdown by purpose, were for training. Fetching tied to a user action, the kind triggered when someone asks ChatGPT something and the model goes and reads a page, accounted for less than 5%. Anyone looking at AI-generated results and expecting visits in exchange for crawling starts from a very lopsided ratio.
How to verify it really is Googlebot
Google states that its crawlers identify themselves in three ways: the user agent in the request, the source IP address and the hostname you get from that IP with a reverse DNS lookup. The first can be faked by typing a string, the other two cannot.
The manual check is two host commands. The first resolves the IP you find in the logs and must return a name under googlebot.com, google.com or googleusercontent.com. The second resolves that name and must return exactly the starting IP. The double step is needed because a reverse DNS zone is controlled by whoever owns the IP and on its own can be manipulated: the round trip is what closes the loop.
For checks at volume, Google publishes its IP ranges in CIDR format in regularly updated JSON files, split by family: common-crawlers.json for Googlebot and similar, special-crawlers.json for AdsBot, three more for user-triggered fetchers. The comparison can be automated in a few lines.
A detail that confuses plenty of people: Google crawls mostly from US IP addresses, and if it detects that a site blocks requests from the United States it may try from other countries. A Googlebot with an American IP on a UK site is normal, not an anomaly to block.
Robots.txt is still the first file a compliant crawler downloads, and since 2022 its rules have been a real IETF standard, RFC 9309, which formalises the protocol Martijn Koster wrote in 1994. The RFC also says something worth keeping in mind: these rules are not a form of access authorisation. They keep out those who have chosen to respect them. We have a dedicated entry on robots.txt covering how to write it and what it cannot do, and one on noindex, the different instruction you need when the problem is the index rather than crawling.
The technical limits almost nobody reads
Google's documentation on the technical properties of its crawlers is short and contains at least three numbers that change how you build a page.
| Property | Google crawler behaviour |
|---|---|
| Maximum size fetched | First 15 MB per file, the rest is ignored. Individual crawlers may have lower limits (2 MB) or higher ones for PDFs |
| Protocols | HTTP/1.1 and HTTP/2, switching between sessions based on previous statistics. Also FTP and FTPS, very rarely |
| Accepted compression | gzip, deflate, Brotli, declared in the Accept-Encoding header of every request |
| Caching | Only ETag with If-None-Match and Last-Modified with If-Modified-Since. Other caching directives are not supported |
| Geographic origin | Mostly US IP addresses, spread across many data centres |
| Reducing crawl rate | Done by returning 429, 500 or 503. There is no way to ask for an increase |
The 15 MB ceiling applies to the HTML, not to linked resources: images, CSS and JavaScript are fetched separately and counted separately. Still, a document that exceeds the threshold is cut off halfway and assessed on whatever was read. It happens more often than you would think on pages with hundreds of products rendered server-side.
When crawling becomes a cost
Google names three recurring causes behind a sudden spike in crawling, and they are nearly always URL problems: faceted navigation generating endless filter combinations, a calendar producing a URL for every date, a dynamic search ads target. Before doing anything about the crawler, look at the access logs and work out how many distinct URLs the site is serving.
If you genuinely need to reduce the load for a few hours, the documented route is to answer crawl requests with a 429, 500 or 503 instead of a 200. When Google sees a significant number of URLs with those codes it reduces the rate across the whole hostname, and raises it again by itself when the errors drop. It is a tap, not a switch, and it should be turned down for hours or days, not weeks: a prolonged 503 ends up removing pages from the index. If your response time is the real bottleneck, the tap solves nothing.
The opposite mistake is more common and quieter: serving a 200 for pages that no longer exist, the so-called soft 404s, or piling up chains of 301 redirects that use up requests and lead nowhere. They are the two quickest ways to make a crawler spend its time on your site and get nothing back. In technical SEO they are among the first things to check.
Crawlers and scrapers do different jobs
A crawler explores breadth first: it starts from a list, follows links and does not know in advance where it will end up. A scraper starts from URLs already chosen and goes after specific fields, the price, the availability, the phone number. The first produces an index, the second a database.
The practical difference is manners. The major search engine crawlers read robots.txt and pace their requests so as not to bring the server down. Scrapers often do neither, and it is no coincidence that they arrive from residential IPs with browser user agents. If your logs show thousands of requests to a single type of page, all at regular intervals, you are looking at a scraper, not a crawler.
The ratio that is changing the decision
For thirty years letting a crawler in was an automatic choice, because the trade was clear: you let me read, I send you visits. Since July 2025 Cloudflare has published the number that measures that trade operator by operator, the ratio between crawl requests and visits referred back. In the first week of August 2025, across all monitored sites, Anthropic was at around 50,000 requests per visit, OpenAI at 887, Perplexity at 118. On news and publishing sites the same ratios dropped to 2,500, 152 and 32.7.
The interesting point is not the size of the numbers but the fact that they now exist and vary by sector. It means "allowing crawling" has stopped being a single decision and has split into one decision per operator, with an infrastructure cost on one side and a measurable return on the other. Anyone working on visibility in answer engines will have it on the table much sooner than they expect, and the right answer for a publisher will not be the right one for an industrial manufacturer.
Web crawler FAQs
It is an automated client that downloads resources identified by a URL and follows their links recursively. The formal definition is in the IETF's RFC 9309, which describes it as an automated client that traverses links recursively for indexing. Outside computing the same word covers a baby, a tracked vehicle or a radio-controlled rock crawler, which is why a UK search for crawler on its own mixes dictionaries, an RC model shop and a film listing with the technical pages (DataForSEO, Google UK, September 2026).
Crawling is downloading the page; indexing is the decision to keep it and its recording in the index. They are two separate stages and can fail independently: a crawled page may not be indexed because it is a duplicate or judged to be of little value, and a URL that was never downloaded can still appear in the results if other sites link to it.
The Crawl Stats report in Google Search Console shows requests, bytes downloaded and average response time for Google's crawlers, broken down by file type and purpose. For everything else you need the server's access logs, which are the only source that also sees non-Google bots. Before reading them, run the reverse and forward DNS check, or you will count as Googlebot requests that Googlebot never made.
In the case of Google-Extended, no: Google states that it does not affect inclusion in Search and is not a ranking factor. Blocking Googlebot, on the other hand, removes the site from the index and therefore from Search. With other operators the calculation is different and concerns visibility inside assistants, not positions on Google.
In SEO the terms are used interchangeably: crawler, spider, robot and bot all describe the same kind of program. Bot is the broadest, since it also covers scrapers and other automated clients that do not follow links. Google's own documentation talks about crawling and crawl rate rather than spidering.
