A search engine is two things put together: an archive of web pages and a program that decides which of them to show to someone asking a question. Between the archive and the answer there are five steps. Most explanations you will find cover three.
Crawling, indexing, ranking: that is the model used by BBC Bitesize, by most agency blogs and by the AI Overview Google itself places above the results when you search "how do search engines work" in the UK. It isn't wrong, it is incomplete in two places. Between crawling and the index there is rendering, which is why many modern sites are only half read. And on top of ranking, for a couple of years now, there has been a layer that writes an answer instead of listing links.
A few numbers to set the scale. Google's index covers "hundreds of billions of webpages" and is "well over 100,000,000 gigabytes in size", according to Google's public documentation. In the UK that engine handles 91.75% of searches, against 5.67% for Bing and 1.25% for Yahoo (StatCounter, August 2026). Understanding how Google works means understanding almost all UK search traffic.
The five stages, in order
This table is the map for the rest of the article. The third column is the one that matters to site owners: it shows what can break at each step.
| Stage | What the engine does | What blocks it |
|---|---|---|
| 1. Discovery and crawling | Finds URLs and downloads their content with a crawler | robots.txt, server errors, pages reachable only through an internal search form |
| 2. Rendering | Runs the JavaScript in a Chromium instance and gets the final HTML | Content that appears only after a click, blocked resources, a long rendering queue |
| 3. Indexing and selection | Analyses text, images and video, chooses the canonical version, stores it | noindex, duplication, pages too similar to each other, low quality |
| 4. Ranking | Interprets the query and orders the eligible pages | Search intent different from what the page satisfies |
| 5. Answer generation | Writes a summary citing some indexed sources | nosnippet, data-nosnippet, max-snippet, page not indexed |
1. Discovery and crawling
Google's crawler is called Googlebot, the generic name for two types of crawler: Googlebot Smartphone and Googlebot Desktop. The first simulates a phone, the second a computer, and for Search it is mainly the first that counts.
Googlebot discovers addresses in three ways, which the documentation lists without ranking them: revisiting pages it already knows, following a link from a known page to a new one, or reading a sitemap you have submitted. Before downloading anything it reads the robots.txt file, which says which parts of the site it may visit.
How much it crawls depends on two variables. The first is the capacity limit: Google measures how quickly your server responds and cuts back its requests if the server slows down, because the time and resources it can spend on any single site are limited. The second is crawl demand, which grows with the size of the site, how often it is updated and the quality of its pages.
Google also says when this is worth worrying about, and the thresholds are higher than people tend to assume: more than a million unique pages that change weekly, or more than ten thousand unique pages that change daily (crawl budget management guide). Below those figures, crawl budget is not your problem. On the business sites we analyse, the bottleneck is almost always in the next two stages.
2. Rendering, the step almost nobody mentions
The AI Overview Google shows in the UK for "how do search engines work" describes three stages and never mentions rendering. It is the biggest gap, because it is the point where a site built in React or Vue can disappear from the results while looking perfectly fine in the browser.
Google puts it like this: "Googlebot queues all pages with a 200 HTTP status code for rendering", and then "a headless Chromium renders the page and executes the JavaScript". The words that matter are queues and once resources allow. A page can sit in that queue for a few seconds or for longer (Understand the JavaScript SEO basics).
The practical consequence: if the text on your product page comes from a JavaScript call, Google sees it later, and until it does, the page is judged on what was in the initial HTML. Often that is a title and a menu. Most of the technical SEO work on a modern site revolves around this gap.
3. Indexing: crawled does not mean indexed
After rendering, Google analyses the text, images and video files and decides whether the page deserves a place in the index. This is the step beginner guides skip, because it undermines the comforting idea that publishing is enough.
The sentence Google repeats in its in-depth guide to how Search works is blunt: "Google doesn't guarantee that it will crawl, index, or serve your page, even if your page follows the Google Search Essentials." It is not boilerplate. At the same stage Google groups pages with similar content and picks the one that is most representative of the group: this is the mechanism that decides which canonical URL is shown and which is left out, and the reason duplicate content costs rankings even when nobody has copied anything.
How strong is the filter across the web? Ahrefs measured its own index of around 14 billion pages and found that 96.55% get no traffic from Google, and another 1.94% get between one and ten visits a month (Ahrefs study, 2023). The sample leans towards the better part of the web, as the authors themselves admit, so the percentage should be read as an order of magnitude rather than a UK figure. But that is the order of magnitude: publishing a page and seeing it in the index are two different events.
The two tools you need here are the noindex tag, to keep out what shouldn't get in, and the Page indexing report in Google Search Console, which gives the reason for every excluded URL. It is the first screen we open when a client says their site "doesn't show up on Google".
4. Ranking: what happens after you hit enter
When someone hits enter, Google doesn't search the web: it searches its own index, the ordered copy of the web built in the three previous stages. It then orders the eligible pages using signals of relevance, quality, language, location and device. The full list isn't public, but the families of signals are documented and we have collected them in our guide to ranking factors.
The signal that outweighs all the others combined is also the least technical: search intent. Google shows the kind of page its users open and stay on. If the first page for your keyword is made of guides, a product page won't get in, however well optimised it is. It is also why the same question returns different results in different countries: the Italian SERP for the equivalent query is full of tech-brand glossaries, while in the UK the first page for "how do search engines work" mixes agency blogs with threads from Reddit, LinkedIn and Quora (DataForSEO, Google UK, September 2026).
5. The generative layer: when the engine answers instead of listing
A fifth step has appeared above the ten blue links. Google calls it AI Overviews and, in its conversational version, AI Mode. It uses a technique Google calls query fan-out, issuing multiple related searches across subtopics and data sources, and then writes a summary citing the pages it found.
The part that surprises people in the industry is that there is nothing new to implement. Google is explicit: "There are no additional requirements to appear in AI Overviews or AI Mode" (documentation on AI features). A page needs to be indexed and eligible to appear in Search. The fifth step draws on the same index as the first four, so everything above applies unchanged.
What does change is who comes knocking. Besides Googlebot, other crawlers now visit, with separate permissions:
- Google-Extended controls whether crawled content can be used to train future generations of Gemini and for grounding in the Gemini apps. Google states that it "does not impact a site's inclusion or ranking in Google Search".
- OAI-SearchBot is the crawler OpenAI uses to show websites in the search results of ChatGPT's search features. Block it and you drop out of those answers.
- GPTBot collects content for model training, a separate choice from the one above.
- ChatGPT-User is triggered when a person asks ChatGPT to open a URL, so it doesn't crawl automatically.
Anyone who adds a hasty line to robots.txt to "block AI" risks dropping out of ChatGPT's answers while staying in the training data, or the other way round. The rules are in Google's documentation on common crawlers and in OpenAI's documentation on its bots, and they don't match. How to get into those answers is covered in our guide to appearing in ChatGPT answers.
How far it has got in the UK: our test on five queries
On 26 September 2026 we ran five UK searches about how search engines work on Google UK, desktop, without personalisation, using DataForSEO's SERP scraping. We wanted to know one thing: how many of these results pages already put a generated answer above the first organic link.
| Query | Generated answer at the top | "People also ask" box | First organic result |
|---|---|---|---|
| how do search engines work | Yes | Yes | bluewater.digital |
| what is a search engine | Yes | Yes | rhacc.ac.uk |
| difference between browser and search engine | No | Yes | ionos.co.uk |
| how does googlebot work | Yes | Yes | reddit.com |
| how to get my website on google | Yes | Yes | search.google.com |
Four queries out of five have an answer written by the engine above the first result. The one without is the comparison query, "difference between browser and search engine". In the equivalent Italian test the only query without a generated answer was the practical one (how to get a site indexed); in the UK, "how to get my website on google", with 880 monthly searches the most searched practical query of the five, has one too (DataForSEO, Google UK, September 2026). So the generative layer is not limited to definition questions: it is taking how-to searches as well. The same reasoning applies to zero-click searches.
The most instructive detail, though, is in the sources cited. For "what is a search engine" (1,900 monthly searches in the UK), Google's generated answer takes its definition from a glossary on BDC.ca, the site of a Canadian business bank, and also cites a YouTube video, Britannica, an email marketing platform's glossary and Study.com. Google Search Central, the primary source on this subject, doesn't appear. The engine doesn't pick the most authoritative source: it picks the one that answers most directly and is well indexed. Anyone writing technical content in English has a real opening here, and few are using it.
A search engine and a browser are not the same thing
Google shows this question in the "People also ask" box for the two broadest searches in our test, and it gets 260 searches a month in the UK on its own, so it is worth settling in two lines. The browser is the program installed on the device: Chrome, Safari, Firefox, Edge. It opens pages and turns code into text and images. The search engine is a service that lives on a website and helps you find which pages to open.
You always use a browser to reach a search engine. You don't need a search engine if you already know the address. The confusion comes from the fact that Chrome opens on Google and that both carry the same brand name.
Which search engines really matter in the UK
A recurring question is whether it makes sense to optimise for anything other than Google. The UK figures answer it.
| Search engine | UK share |
|---|---|
| 91.75% | |
| Bing | 5.67% |
| Yahoo! | 1.25% |
| DuckDuckGo | 0.71% |
| Ecosia | 0.30% |
| Yandex | 0.28% |
Google accounts for more than nine searches in ten. Bing and Yahoo together don't reach seven points, and Yahoo takes much of its results from Bing, which narrows the real variety further. If you have the time and budget for only one engine, the choice is made. Alternative search engines are still interesting for reasons other than share, such as privacy or foreign markets.
What changes for site owners
The five stages can also be read backwards, as a diagnostic checklist. When a site doesn't bring in enquiries, the problem sits in one of those five points, and it pays to check them in order, because fixing the fourth without solving the first achieves nothing.
- Can the pages be reached through an internal link, and are they not excluded by robots.txt?
- Is the content in the HTML, or does it arrive later, via JavaScript?
- Does Search Console say they are indexed, or list them as excluded?
- Does the page type match what fills the first page for that search?
- Does the text answer the question directly enough to be cited in a generated answer?
The fourth point moves the numbers more than any other, and the clearest example we can show is public. For Macropix, a Milan-based LED screen manufacturer, the Italian keyword "monitor pubblicitario" (advertising monitor) went from position 88 to position 2, and the domain now holds a 25.55% share of voice on the LED wall cluster in Italy, ahead of amazon.it at 18.08% (Semrush, July 2026). It wasn't a technical fix: it was a page built around the type of answer that search was asking for. The full story is in our case study on SEO for manufacturing, and our other projects are on the projects page.
If you want the five-point diagnosis run on your own site rather than by hand, it is the work we start every SEO project with; the fifth layer, visibility inside generated answers, is covered by our dedicated AI search optimisation service.
Frequently asked questions
The browser is the program installed on your device, for example Chrome, Safari, Firefox or Edge, and it opens web pages. The search engine is a service that lives on a website and helps you find which pages to open. You always use a browser to reach a search engine, while you do not need a search engine if you already know the address of the site.
According to StatCounter, in August 2026 Google handled 91.75% of searches in the UK, Bing 5.67%, Yahoo! 1.25%, DuckDuckGo 0.71%, Ecosia 0.30% and Yandex 0.28%. Yahoo takes much of its results from Bing, so the real variety is even smaller than the shares suggest.
There is no guaranteed time. Google states explicitly that it does not guarantee it will crawl, index or serve a page. After crawling, the page enters a rendering queue, where it can stay for a few seconds or longer. The URL Inspection tool in Google Search Console shows the real status of that page and lets you request indexing.
The most common causes, in order of frequency: the page is blocked by the robots.txt file, has a noindex tag, cannot be reached from any internal link, shows its content only after JavaScript runs, or is treated as a duplicate of another page and replaced by the canonical version. The Page indexing report in Search Console gives the exact reason for each excluded URL.
They have not replaced them: they have added a layer on top. Google generated answers draw on the same index as Search, and Google says there are no additional requirements to appear in them. ChatGPT uses its own crawler, OAI-SearchBot, and sites that block it drop out of its search answers. In our test of 26 September 2026 on five UK queries, four SERPs showed a generated answer above the organic results, which were all still there.
One thing the industry rarely says. The five stages don't describe Google alone: the paragraph that gets a page into a generated answer is the same one that gets it cited by ChatGPT and Perplexity, because all three look for text that answers without preamble and stands on its own, outside the context of the page. A business that can explain a technical subject well therefore has an advantage no optimisation tool can replicate, and the UK first page for "what is a search engine" shows what the competition looks like: a college course listing, a 2019 GOV.UK blog post and an agency's SEO services page, while the generated answer borrows its definition from a Canadian bank's glossary.
