SEO for AI agents is the work needed so that an agent can read a website, understand it and complete an action on it: find a price, fill in a form, add a product to the basket. It's the second meaning of agentic SEO, the one about visitors rather than about who does the work.
The difference from classic SEO lies in the reader. Googlebot downloads the page and indexes it. An agent uses it: it clicks, types, goes back, and if something doesn't work it tries another site.
What an agent does on a website
An agent gets a task from the user ("find me three outdoor LED wall suppliers in the Midlands and ask each for a quote") and breaks it into steps. For each step it opens a page, reads it and decides what to do.
How it reads the page depends on the agent. Slobodan Manic, writing in Search Engine Journal, identifies three approaches:
- accessibility tree: the agent reads the structure the browser prepares for screen readers, with the role and label of every element. This is how tools such as Playwright MCP work;
- screenshots: the agent looks at an image of the page and decides where to click. This was how the old ChatGPT agent worked, using a virtual browser based on screenshots;
- hybrid: a mix of the two.
In its FAQ for publishers and developers, OpenAI wrote that its browser "uses ARIA tags, the same labels and roles that support screen readers" and recommends following the WAI-ARIA guidelines. Put simply: an accessible website is one that agents can use.
The trouble is that hardly any websites are. According to the WebAIM Million 2026 report cited by Manic, 95.9% of the home pages analysed have WCAG errors.
Agentic browsers today
At the end of September 2026, these are the agents that actually browse websites:
- Perplexity Comet, free to use since October 2025 and on iOS since March 2026;
- Claude in Chrome, available on all paid Claude plans since 26 August 2026;
- Chrome auto browse with Gemini, in preview in the US only for AI Pro and Ultra subscribers. It hasn't reached the UK: in July 2026 Google kept it US-only, with no date for other markets;
- ChatGPT Work and the ChatGPT extension for Chrome, which replaced the ChatGPT Atlas browser, shut down on 9 August 2026;
- Opera Neon, a paid product since December 2025.
According to HUMAN Security, in July 2026 Comet generated 47.1% of the agentic traffic it measured, Claude in Chrome 24% and Atlas 15.5% before it closed. 76% of activity was product browsing and research.
How to spot them in your logs
AI agents and crawlers declare a user agent. The main ones, from the official documentation:
| User agent | Owner | What it's for | Follows robots.txt |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Indexing for ChatGPT search | Yes |
| GPTBot | OpenAI | Collecting training data | Yes |
| ChatGPT-User | OpenAI | Actions started by a user | Not always |
| PerplexityBot | Perplexity | Indexing for search | Yes |
| Perplexity-User | Perplexity | Visits started by a user | Usually not |
| ClaudeBot | Anthropic | Data collection | Yes |
| Claude-User | Anthropic | Visits started by a user | Not always |
| Google-Agent | Actions by Google agents at a user's request | Check case by case |
Sources: OpenAI, Perplexity, Google. Google-Agent was added to the documentation on 20 March 2026.
One detail changes the job: the "-User" user agents are visits made on a person's behalf, in real time. OpenAI writes that for ChatGPT-User "robots.txt rules may not apply", and Perplexity that Perplexity-User "generally ignores robots.txt". Blocking them in robots.txt often achieves nothing. If you really want to block them, you have to do it at firewall level, at the risk of losing customers who are using an assistant.
We looked at the logs for visilay.com, a site whose audience is mostly Italian, from 31 August to 27 September 2026: around 199,000 requests. Claude-User made 9,340 requests, but a good share of them are ours, since we use Claude every day to work on the site. The rest is more interesting: ChatGPT-User made 834 requests across 70 different URLs, 402 of them to the home page. PerplexityBot made 100 and Perplexity-User 58. The full breakdown, bot by bot, is in our article on AI crawlers.
Signed agents: telling them apart from malicious bots
A user agent can be faked. That's why, since 2025, there's been Web Bot Auth, a system in which the agent signs its requests with a public key. Cloudflare introduced "signed agents" in August 2025, based on the HTTP message signatures of the RFC 9421 standard.
OpenAI signs its agent's requests with the header Signature-Agent: "https://chatgpt.com" and publishes the keys in a dedicated directory (documentation). Cloudflare and Akamai recognise it. Google published its own Web Bot Auth documentation, still experimental, in May 2026.
If your site uses Cloudflare or another anti-bot system, it's worth checking how it treats signed agents. It's one of the most common causes of an "agent blocked at the first step".
Semantic HTML and the accessibility tree
The fixes that help most, in order of impact:
- Buttons that are buttons. A
divwith a click event has no role in the accessibility tree. Abuttondoes. - Labels on form fields. Every input has its own
label. A placeholder isn't enough, because it disappears as soon as you type. - Alt text on images that carry information, such as price tables saved as images. Better still, put them in HTML. Our guide to alt text explains how to write it.
- Headings in order. One H1, then H2s and H3s with no skipped levels. Agents use headings to find their way around, just as screen readers do.
- Content in the HTML, not only in JavaScript. Some agents run JavaScript, others don't. Prices and key information need to be there without it.
- Links with descriptive text. "Learn more" repeated ten times doesn't tell the agent where each one goes.
Forms, buttons and checkout
This is where agents get stuck most often.
- CAPTCHAs. A CAPTCHA on every submission also stops the agent of a genuine customer. Invisible checks, or checks limited to suspicious cases, work better.
- Hidden steps. Fields that only appear after an action, pop-ups covering the button, cookie banners that block everything until you click.
- Unclear errors. "There's an error in the form" without saying which field. The agent retries at random or gives up.
- Visual-only confirmations. A green tick with no text. The agent can't tell whether the submission went through.
Chrome auto browse asks the user to confirm before purchases and posts. The other agents also tend to stop before irreversible steps. A clear checkout, with a readable final summary, gets the agent as far as that point.
robots.txt, llms.txt and structured data
- robots.txt: decide which AI indexing bots to let in. Blocking OAI-SearchBot removes the site from ChatGPT search answers. Blocking GPTBot or ClaudeBot only affects training. Our guide to robots.txt has the syntax.
- llms.txt: a text file listing your main pages, intended for models. Google has said it isn't needed for its AI features. It costs little and some agents read it. Ours is at visilay.com/llms.txt, with more than 600 links; how to put one together is explained in llms.txt.
- Structured data: Organization, Product, Offer, FAQ. It helps the agent find prices, availability and contact details without interpreting the design. See schema markup.
- WebMCP: the next step, in which the site explicitly declares the actions an agent can take. It's still experimental in Chrome. We explain it in WebMCP and MCP.
How to test your website with an agent
The most useful test takes a few minutes.
- Pick a real task: "request a quote for X", "find the price of Y and add it to the basket".
- Give it to an agent, for example Claude in Chrome or Comet, without helping.
- Watch where it stops, what it gets wrong and how many attempts it makes.
- Repeat with a second agent, because they read pages differently.
- Fix the problems and test again.
Alongside that, check your server logs to see which agents are already arriving and on which pages. If the logs show lots of ChatGPT-User visits to a page that doesn't convert, that's the page to fix first.
Checklist
- Buttons and links built with the correct HTML elements and descriptive text
- A label on every form field
- Key information (prices, contact details, availability) in HTML, not only in images or JavaScript
- Headings in hierarchical order
- No compulsory CAPTCHA on the main forms
- Error and confirmation messages written as text
- A robots.txt that lets in the AI search bots you care about
- A firewall that recognises signed agents
- Organization structured data and, if you sell online, Product and Offer
- A test with at least two different agents
Further reading: Agentic commerce: how to prepare an online shop for AI agents.
FAQs
Through the page's accessibility tree, meaning the structure of roles and labels prepared for screen readers, through screenshots, or with a mix of the two. OpenAI has said that its browser uses ARIA tags.
It depends on what you want. Blocking OAI-SearchBot removes the site from ChatGPT search answers. Agents acting on a user's behalf, such as ChatGPT-User or Perplexity-User, often don't follow robots.txt, so blocking them would take a firewall rule, with the risk of losing real customers.
From your server logs, by looking for agent user agents and the pages visited after the first one, and with a manual test: give an agent a task and watch where it stops.
Yes. Many agents read the page through the same accessibility tree that screen readers use. Buttons with no role, fields with no label and information that only exists in images are the most common reasons agents get stuck.
