AI visibility data is reliable as a frequency estimate across many answers, not as a snapshot of a single answer. In our test on 27 October 2026 we asked ChatGPT and Gemini the same question five times each, on two topics, in Italian: all 20 brand lists we got back were different, and the two engines did not cite a single website in common.
So the useful question isn't "is the tool accurate?". It's how many answers sit behind the number you're reading.
On Google you never had to ask. Take Sunscape, which develops solar plants across Italy: from March 2026, in six months, organic impressions went from 329 to 5,809 a month and clicks from 46 to 195, as we explain in how long SEO takes. Search Console gives you a number, and it's the same number tomorrow morning.
AI answers don't work like that. And one of the two topics in our test is Sunscape's own sector.
The mistakes we see most often when a company reads AI visibility data:
- trusting a screenshot: one answer, forwarded to the managing director, deciding whether "we're in" or not;
- reading the position in the list as a ranking, when the first name changes from one answer to the next;
- comparing two tools that use different prompts, engines and repeat counts, then being surprised they disagree.
In this article you'll find the full test with a table you can cite, what changes and what stays put between answers, how many repeats you need before trusting a percentage, and six questions to ask of any report.
How we ran the test (20 answers, under 50 cents)
It's a repeated measurement: the same question, word for word, sent several times to the same engine to see how much the answer moves.
- Two questions in Italian, one consumer and one B2B: which solar panel brands to choose for a home in Italy, and which CRM to choose for a small B2B company in Italy.
- Two engines: ChatGPT (gpt-5.4-mini) and Gemini (gemini-3.5-flash requested, served as gemini-3.6-flash), both with web search enabled.
- Five runs per question per engine, on 27 October 2026 between 17:03 and 17:06 UTC, through the DataForSEO LLM Responses endpoint.
- What we counted: brands named in each answer, the first name in the list, the domains cited as sources, and whether the engine actually searched the web.
Total cost of the 20 answers: about $0.45 in API fees.
Is 20 answers too few? To estimate a percentage, yes. To check whether two identical answers exist, no. And they don't.
How accurate are brand visibility insights across different LLMs? (it depends heavily on the engine)
They're accurate for telling you which brands an engine tends to name, not for predicting what the next user will see. And stability varies a lot by engine: on the solar panel question, Gemini named 7 brands in all five answers, ChatGPT just one.
| Question and engine | Distinct brands in 5 answers | Brands in 5 out of 5 | List length | First name in the list | Web search used | Distinct domains cited |
|---|---|---|---|---|---|---|
| Solar panels, ChatGPT | 11 | 1 (REC) | 3 to 9 | 4 different names in 5 answers | 5 of 5 | 8 |
| Solar panels, Gemini | 14 | 7 | 9 to 12 | Maxeon 5 of 5 | 4 of 5 | 13 |
| CRM, ChatGPT | 9 | 2 (Pipedrive, Zoho CRM) | 4 to 6 | Pipedrive 5 of 5 | 5 of 5 | 6 |
| CRM, Gemini | 14 | 3 (HubSpot, Pipedrive, Zoho CRM) | 5 to 7 | HubSpot 4 of 5 | 1 of 5 | 6 |
Three numbers sum it up: 0 identical lists out of 20; 14 of 48 brands (29%) appeared in only one answer out of five; 0 domains in common between ChatGPT's and Gemini's sources, out of 33 domains cited in total.
We're not the first to see it. The SparkToro and Gumshoe study published by Rand Fishkin in January 2026 (600 volunteers, 2,961 runs on ChatGPT, Claude and Google AI, US market) found less than a 1 in 100 chance of getting the same list twice, and roughly 1 in 1,000 of getting it in the same order. Our test confirms it on a small scale, in Italian.
What changes between answers (and what stays put)
Mostly the length of the list and the names at the bottom. A core of brands stays, when there is one.
The starkest case is ChatGPT on solar panels. The first answer listed 9 brands, the second 3: Trina, LONGi, REC. Same question, same model, 33 seconds apart. And the first name? Aiko, then Trina, then Maxeon, then REC twice.
If a report tells you "you're first on ChatGPT" after a single run, it's telling you something the next question may contradict. That's why, among the AI visibility metrics, position counts for less than mention rate.
On CRM it's the opposite: Pipedrive came first in 5 out of 5 ChatGPT answers. On Gemini, HubSpot came first in 4 out of 5. So "position" exists, but it belongs to the engine, not the market. The leader depends on who's answering.
Another way to look at it: of the 29 brands named in the first run of each series, only 13 (45%) were still there in all five. AirOps' September 2025 report on 800 queries found lower numbers: 30% of brands stayed visible from one answer to the next, and only 1 in 5 from the first run to the fifth.
One more detail for local brands. In ChatGPT's five answers on solar panels for an Italian home, no Italian manufacturer appeared. In Gemini's, FuturaSun appeared 5 times out of 5, Trienergia 4, Peimar 2.
Sources change more than brands (and ChatGPT reads in English)
Cited sources, the AI citations most brands want, are the least stable part of monitoring. Across both questions ChatGPT and Gemini cited 33 different domains with none in common.
On solar panels, ChatGPT cited 8 domains, none of them Italian: which.co.uk, energysage.com, solarreviews.com and manufacturers' own sites. Gemini cited 13, all with Italian-language pages (12 sites plus a translated Reddit thread). On CRM, ChatGPT cited only vendor websites: pipedrive.com, salesforce.com, zoho.com, hubspot.com, brevo.com, teamleader.eu.
Does Google ranking predict any of this? Barely. We compared the sources with Google Italy's top 10 organic results for the Italian query "migliori marche pannelli fotovoltaici" on the same day: 4 of Gemini's 13 domains were on page one, none of ChatGPT's 8. The reason is query fan-out: the engines search with their own queries, and ChatGPT's were almost all in English ("best solar panel brands Europe 2026" and similar), even for an Italian question.
Google's AI Overviews behave similarly. Ahrefs tracked more than 43,000 keywords for a month (Louise Linehan, November 2025): the text changes on average every 2.15 days, and between two consecutive versions only 54.5% of cited URLs stay the same. The meaning barely moves, with a semantic similarity of 0.95 out of 1.
Gemini sometimes doesn't search at all (5 answers out of 10)
Turning web search on doesn't mean the model searches. In our test Gemini used Google Search in 5 answers out of 10: 4 of 5 on solar panels, 1 of 5 on CRM.
That's not a bug. Google's developer documentation says it plainly: the model analyses the prompt and decides whether a Google Search can improve the answer.
The four CRM answers without search named brands the searched answer didn't include (vteNEXT, vTiger, ActiveCampaign, Sellf, Fatture in Cloud) and had no sources at all.
If your Gemini citation rate is calculated over answers where Gemini never searched, half the sample couldn't cite you by design. You look invisible, but the engine never looked. That's where budgets leak: you rewrite pages to fix a number that has nothing to do with your pages.
A badly calculated number costs more than no number.
You've just seen that half of Gemini's answers in our test never searched the web. If your report mixes them with the rest, you end up changing pages based on data that has nothing to do with them. We separate answers with and without search, repeat the questions, and tell you how wide the margin is on every percentage.
Book a strategy callHow many repeats you need before trusting a percentage (more than 5, fewer than 100)
A credible mention rate needs dozens of answers per group of questions, not five. With 5 answers, a brand named 3 times has a "true" rate anywhere between 23% and 88%.
We calculated the margin with the 95% Wilson interval, the standard method for proportions on small samples:
| Answers analysed | Measured mention rate | Plausible range (95%) | Width of the margin |
|---|---|---|---|
| 5 | 60% (3 of 5) | 23% - 88% | 65 points |
| 10 | 60% (6 of 10) | 31% - 83% | 52 points |
| 20 | 60% (12 of 20) | 39% - 78% | 39 points |
| 50 | 60% (30 of 50) | 46% - 72% | 26 points |
| 100 | 60% (60 of 100) | 50% - 69% | 19 points |
Read it like this: with 20 answers, going from 60% to 70% in a month proves nothing, because it sits inside the margin.
So do I need 100 runs per prompt? No. Answers add up: 25 prompts on the same intent, run 4 times, give you 100 answers. We cover how to build that sample in our guide to prompt tracking. It's also Fishkin's conclusion: visibility percentage across dozens or hundreds of prompts run several times is a reasonable metric; ranking position isn't.
API or interface: what a tool really measures (and what it doesn't)
Almost every tool queries the models through an API or simulates a user with no history. So it measures a "clean" answer, not the one your customer sees with their account and their chat history.
Our test used APIs too, and it's fair to say so: a real user would have seen different lists. The SparkToro study, run with real people on the actual interfaces, found the same instability.
What can you still rely on?
- Comparison with competitors, if everyone is measured on the same prompts, with the same method, on the same day: that's the basis of share of voice.
- The trend over several months, if the sample doesn't change.
- Clear presence or absence: a brand that never appears in 100 answers has a problem, whatever the margin.
What you can't rely on is comparing different tools: prompts, engines and scoring formulas differ between vendors, as we showed when we looked at the AI visibility score.
How to read an AI visibility report without being fooled (6 questions to ask)
Before you act on a report, ask whoever produced it these six things.
- How many answers is each percentage based on? Below 20 per group of questions, the margin is wider than 39 points.
- Is each prompt repeated, or run once? A single run is a sample, not an average.
- Did the engine search the web? Answers without search should be separated when you measure citations.
- Are the prompts the same as last month? Changing the questions moves the number more than any change to your site.
- Which engines and which models? In our test ChatGPT and Gemini had zero sources in common: an average of the two hides everything.
- Are competitors measured the same way? Without that, the percentage has nothing to compare against.
That's why we built ScryRank, the monitoring tool developed by Visilay: it asks the same questions every month, with identical settings, on ChatGPT, Claude, Gemini, Perplexity and AI Overviews, and keeps the history of every survey. ScryRank should be judged by the same six questions as any other tool, the ones we use in our guide on how to choose an AI visibility tool.
If you'd rather someone else read the data, that's what our AI monitoring service does; tool prices are in our comparison of AI visibility tool pricing.
What this means for your chances of being cited by AI (the part you control)
You can't remove the variability, but you can shift the odds. The brands that appeared 5 times out of 5 in our test had one thing in common: they show up in many of the pages the engines read. FuturaSun appeared in every Gemini answer and is named in the Italian guides Gemini cited (quotalo.it lists it); on ChatGPT, which read English-language sources, it never appeared.
So the lever isn't the single answer, it's the sources: pages on your own site that answer the question directly, and third-party pages (industry guides, comparisons, reviews) that name you. For Google's AI features, Google's own documentation on AI features in Search says SEO best practices still apply. Nobody can guarantee you a citation; you can increase the number of places where the engine finds you. That's the work behind our GEO service.
The full picture is in our guide to AI visibility; turning these numbers into a monthly document is covered in the AI visibility report.
FAQ
It depends on how many answers sit behind it. A percentage based on 100 answers has a margin of about 19 points; on 5 answers the margin is over 60. In our test on 27 October 2026, 20 answers from ChatGPT and Gemini to two repeated questions were all different.
Yes. They use different prompts, engines, models and repeat counts, and their own scoring formulas. Comparisons only make sense within the same tool, on the same prompts, over time.
No. In our test ChatGPT listed 9 solar panel brands in one answer and 3 in the next, 33 seconds later. One answer tells you what happened once, not how often it happens.
It depends on the engine and the topic. On CRM, Pipedrive came first in 5 of 5 ChatGPT answers, but HubSpot came first on Gemini. On solar panels ChatGPT put 4 different brands first in 5 answers. Mention rate is a better guide.
In short
20 answers, 20 different lists.
Gemini steadier on brands, ChatGPT steadier on the top CRM.
Zero sources in common between the two engines.
Half of Gemini's answers with no web search.
Below 20 answers per group of questions, the margin is wider than 39 points.
AI visibility data looks more like an opinion poll than a ranking. And nobody publishes a poll without saying how many people were asked and what the margin of error is. Next time you get a report, ask for the same thing: how many answers, and how much margin. If the vendor doesn't know, treat the number as an impression, not a measurement.
