SEO Data Bench

SEO Data Bench/Guides/ai-visibility-runs-per-prompt

How many runs per prompt does an AI visibility score need? 319 answers measured

Field guidecohort v1US · enrun 2026-09-15includes a Treg route

Eight buyer prompts, ten runs each, through ChatGPT, Google AI Mode, Gemini and Perplexity's API in one hour: how much of the brand list and cited URLs one, three and five runs see, and how far two identical panels sit apart.

By Shirley Lou, ResearchPublished 2026-11-04Facts re-checked 2026-11-04Last material change 2026-11-04

Written for people asking: How many times should you run each prompt for an AI visibility score?Is 3 runs enough to measure which sources an AI answer cites?Why does the same prompt name different brands every time in AI answers?

Five runs per prompt is a defensible minimum for a brand list; three is not, and no number tested settles a cited-URL list. On 4 November 2026 eight buyer prompts went ten times each through four answer routes. On the three interface routes (ChatGPT, Google AI Mode, Gemini), one answer named 42% to 59% of the listed vendors the ten runs named, three runs 67% to 79%, five runs 81% to 88%. Cited URLs did not converge: five runs saw 63% to 69% of the ten-run list, the tenth run still added one to four new URLs per prompt, and no domain was cited in every run on any prompt. For one brand on one prompt, two separate five-run panels still sat 15 to 19 points apart on average. Averaged across all eight prompts, as a tracker's score is, that gap fell to 3 to 3.5 points. Perplexity's API was steadier, with a shorter capped answer. ChatGPT's interface route answered 17 of 79 times with no sources and returned no search flag. Five runs of eight prompts on ChatGPT settled at $0.11.

The question comes from people building their own trackers. One r/TechSEO poster runs each prompt five times per engine because “the same prompt names different brands every time”. Another in r/SEO_LLM asks “is 3 runs enough, or does the noise floor need more?” This run measures the noise floor within one hour. The same prompts repeat on about 4 December for the 30-day half of that second question.

The run

Eight unbranded buyer prompts from this network’s frozen panel (fixture), ten runs each, round by round, through four routes on 4 November 2026 between about 22:30 and 23:10 UTC, including a retry of failed calls. Each answer was checked against a fixed list of 65 vendor names. Cited URLs had tracking parameters removed.

What k runs see

Share of the ten-run list that k runs see, averaged over every possible set of k runs and all eight prompts.

Route Brands: 1 run 3 runs 5 runs Cited URLs: 1 run 3 runs 5 runs New URLs from run 10, per prompt
ChatGPT web interface (Cloro) 59% 79% 88% 22% 50% 69% 3.9
Gemini web interface (Cloro) 46% 73% 85% 19% 44% 63% 1.2
Google AI Mode (Cloro) 42% 67% 81% 19% 44% 63% 1.1
Perplexity API, fast preset, 400-token cap 78% 91% 95% 67% 85% 91% 0.1

Brand lists flatten: the tenth run added 0.15 to 0.26 new listed vendors per prompt on the interface routes. Cited-URL lists keep growing. ChatGPT’s list reached 84 URLs per prompt on average, 54% of them cited in only one run.

How far apart two identical panels sit

The next test avoids comparing a sample with itself. Split the runs into two separate panels of k, and compare each panel’s mention rate for the same brand. Every split was counted for one brand on one prompt; the last three columns draw 2,000 random splits per route and score each brand across all eight prompts, for brands named in at least 10% of answers.

Route One brand, one prompt: 1 run 3 runs 5 runs Score across 8 prompts: 1 run 3 runs 5 runs
ChatGPT web interface 26 pts 19 pts 15 pts 6.0 pts 3.8 pts 3.0 pts
Gemini web interface 33 pts 23 pts 19 pts 7.0 pts 4.4 pts 3.4 pts
Google AI Mode 31 pts 23 pts 19 pts 7.3 pts 4.6 pts 3.5 pts
Perplexity API 19 pts 13 pts 11 pts 2.6 pts 1.8 pts 1.4 pts

Reading the two tables:

  • A week-on-week move on one prompt is usually noise. Two five-run panels from the same hour differed by 15 to 19 points on a typical brand, so a 20-point swing on one prompt is not evidence of anything. The five-run gaps run slightly high because two five-run panels use up all ten runs: a vendor named once is always exactly 20 points apart.
  • Prompts do as much work as runs. Across eight prompts, single runs per prompt put two scores 6 to 7 points apart on the interface routes, and five runs took that to about 3. That is far closer than five runs on one prompt, though the two figures measure different things: a score over a prompt set, and one brand on one prompt.
  • Some brands are stable and most of the list is not. On Gemini, “cheapest way to run weekly keyword and SERP checks” named DataForSEO in 10 of 10 runs, Serper in 7, SerpApi in 5 and Semrush in 4. On AI Mode, “cheapest way to enrich company domains” named 15 listed vendors across ten runs, none in every run and seven only once.
  • Sources move more than brands. Two runs of one prompt shared 43% to 63% of their brands on the interface routes, 20% to 23% of their domains and 14% to 17% of their URLs. Not one domain appeared in every run of any prompt on those three routes. A “we are cited” badge from one capture says little.
  • Perplexity’s steadiness is partly its setup. This route is Perplexity’s API with one search step and output capped at 400 tokens, the highest value the endpoint accepted among those tried. It reported openai/gpt-6-luna as its model. It is not perplexity.ai, and shorter answers have fewer places to differ.

Log whether the answer searched

A commenter on the r/TechSEO thread said a run from memory and a run that searched measure two different things. Cloro’s interface routes return no search flag (its searchQueries field came back empty on 20 of 20 calls in the October run and was not requested here), so cited sources are the nearest proxy.

ChatGPT’s interface route cited no sources on 17 of 79 answers, spread over all eight prompts and up to 4 of 10 on two of them. Those answers named about as many vendors (5.4 against 5.1), but one vendor’s mention rate differed by 18 points on average between the two groups on the same prompt. That comparison rests on few unsourced answers per prompt. AI Mode cited nothing on 1 of 80, Gemini and Perplexity on none.

So split ChatGPT runs by whether they carry sources before averaging, or at least store the count. The live-answers guide covers the routes that do report a flag.

What it cost

Route Settled per call Five runs of one prompt Median latency
ChatGPT web interface $0.0028 $0.014 37 s
Gemini web interface $0.0024 $0.012 14 s
Google AI Mode $0.0024 $0.012 10 s
Perplexity API $0.00135 $0.007 4 s

Costs are charges settled on the ledger; the 319 answers came to $0.71. ChatGPT’s list price is $0.0036, and it settled lower. Seven ChatGPT calls failed on the first pass, six of them timing out at 180 seconds. Six succeeded on retry and one prompt stayed at 9 of 10 runs. The failed calls left no settled charge.

Who this fits

  • You read a paid tracker’s score: ask how many runs sit behind each prompt, and treat a one-prompt change under about 20 points as noise unless the tracker has many more than five runs.
  • You build your own: five runs per prompt across as many prompts as you can afford, stored answers and sources per run, and citation frequency instead of a citation yes or no. ChatGPT is where a timeout budget and a retry matter.
  • You track citations for a page: count domains and URLs as rates across at least ten runs, and expect that list to keep growing.

What this run cannot tell you

One hour, one country, eight prompts in three data-API markets, so nothing here shows drift over days; that is the 30-day repeat. Ten runs is a small reference, and the “share of the ten-run list” figures flatter k because the true list is longer. Brands outside the fixed list were not counted. Panel gaps are for vendors named at least once, so the many vendors a prompt never names would pull a whole-market average down.

Two planned route types are missing. DataForSEO’s ChatGPT API and Perplexity routes, which report a search flag, returned a provider daily spend-limit error through Treg during the run. Cloro’s Copilot and Perplexity interface routes answered that logged-out access was unavailable. So this run has no comparison against a provider-reported search flag and no perplexity.ai interface.

Next step

Every call ran through Treg’s CLI from one prepaid balance; Cloro and Perplexity are also callable directly with their own keys.

Questions this page answers

How many times should you run each prompt for an AI visibility score?

At least five if you read results per prompt. On 4 November 2026, across ChatGPT, Google AI Mode and Gemini, five runs saw 81% to 88% of the vendors ten runs named, and the tenth run added about 0.2 new vendors per prompt. Five runs do not make one brand's rate on one prompt precise: two five-run panels sat 15 to 19 points apart on average. The score across eight prompts is steadier, 3 to 3.5 points apart.

Is 3 runs enough to measure which sources an AI answer cites?

No, and neither were ten. On the three interface routes three runs saw 44% to 50% of the URLs that ten runs cited, and the tenth run still added one to four new URLs per prompt. No domain was cited in all ten runs of any prompt on those routes. Two runs of one prompt shared 14% to 17% of their URLs and 20% to 23% of their domains. Report how often a source is cited over many runs, not whether it appears in a list.

Why does the same prompt name different brands every time in AI answers?

Each identical call produced a different answer: 0 of 319 were word-for-word repeats of another run of the same prompt. The vendors at the edge of the answer move most. On one prompt, AI Mode named 15 listed vendors across ten runs, none in every run and seven only once. On ChatGPT's interface route, 17 of 79 answers cited no sources at all.

Relationship: the people who run this site also work on Treg, one of the routes compared on this page. Every Treg figure here is a call we paid for, with the raw response downloadable, and the same fixture is run against the alternatives where their terms allow.


Spotted an error or a changed price? Email corrections@seodatabench.com with the page URL and a source. Corrections are logged on the corrections page.