SEO Data Bench/Guides/ai-visibility-tracker-live-answers
How to check an AI visibility tracker is pulling live answers: seven answer routes, measured
Ten prompts, four of them about September 2026 events, through ChatGPT, Perplexity, Google AI Mode and Gemini, and through the ChatGPT model API with web search off and on. Which answers were live, which came from model memory, and what a tracker has to store to prove the difference.
Written for people asking: How do you verify an AI visibility tracker is actually pulling live answers?Does ChatGPT answer from training data instead of searching the web, and how can a tracker tell?What should an AI visibility tracker store for each answer?
Ask it about something from the last few weeks, then check what it stored. On 8 October 2026 ten prompts went through seven answer routes; four asked about September 2026 events that no model's training data holds. The ChatGPT model API with web search off got all four wrong or unknown and cited nothing on any of its ten answers. The five routes that answer as the live products do (ChatGPT web, with and without its search-off option, Perplexity, Google AI Mode, Gemini) got three or four of the four right and cited sources on 49 of 50 calls. Two results matter for trusting a tracker. First, asking the ChatGPT API for web search does not make it search: it searched on 6 of 10 calls, and its own web_search field said so. Second, a cited answer can still be wrong: AI Mode and Gemini both denied an API-limit change Ahrefs had announced, with sources attached. A tracker worth trusting stores the full answer, the cited URLs, the fetch date and a record that search ran. Only DataForSEO's LLM Responses routes returned all four in the response; on the interface routes the tracker has to record the date itself.
The question, as one poster on r/SEO put it: “How do you verify an AI visibility tracker is actually pulling live answers?” (r/SEO, 27 September 2026). The worry is that without web search, ChatGPT and Gemini answer from training data that can be a year old, and a tracker’s mention score could be measuring that memory rather than what buyers see today.
The test: questions only a live answer can get right
The fixture has ten prompts (download). Four ask about things that happened in September 2026, each tied to a dated source:
- F1 What is the multimodal filter in Google Search Console? (introduced in September 2026, Google)
- F2 Did Ahrefs change its API limits in September 2026? (doubled on paid plans, r/ahrefs)
- F3 What is OpenSEO, and what data source does it use? (DataForSEO)
- F4 What is Jev from TypeSafe AI? (a decision model launched in September 2026)
The other six are unbranded buyer prompts from this network’s frozen prompt panel, the kind a tracker counts brand mentions on. Each prompt went once through seven routes on 8 October 2026: 70 calls, all of which returned an answer.
What came back
| Route | Fresh questions right | Calls with cited sources | Model, timestamp, search flag in the response | Median latency | Cost per call |
|---|---|---|---|---|---|
| ChatGPT web interface (Cloro) | 4 of 4 | 10 of 10 | None | 28.9 s | $0.0036 |
ChatGPT web interface, disableWebSearch: true (Cloro) |
4 of 4 | 10 of 10 | None | 29.4 s | $0.0036 |
| Perplexity (Treg’s routed endpoint, served by DataForSEO’s LLM Responses API) | 4 of 4 | 10 of 10 | All three; web_search: true on 10 of 10 |
5.3 s | $0.0058 to $0.0064 |
| Google AI Mode (Cloro) | 3 of 4 | 9 of 10 | None | 10.4 s | $0.0024 |
| Gemini web interface (Cloro) | 3 of 4 | 10 of 10 | None | 42.3 s | $0.0024 |
ChatGPT model API, web_search: true (DataForSEO, gpt-4.1-mini) |
3 of 4 | 6 of 10 | All three; searched on 6 of 10 | 5.5 s | $0.029 to $0.030 when it searched, about $0.001 when it did not |
ChatGPT model API, web_search: false (DataForSEO, gpt-4.1-mini) |
0 of 4 | 0 of 10 | All three | 6.5 s | $0.0007 to $0.002 |
Costs are the charges settled on the ledger for these calls. Grades are by hand against the source for each fact; the rubric and a note on every wrong answer are in grades.json.
Reading the table:
- Model memory reads as confident, not as blank. With search off, the model said it had no information about the Ahrefs change and made up the other three. It described the Search Console filter as a rich-results and video filter, gave OpenSEO’s data source as Search Console, and called Jev a code assistant. None of its ten answers cited a URL. If a tracker’s stored answers carry no sources, this is what it is measuring.
- “Web search on” is a permission, not a guarantee. The ChatGPT API asked to search did so on 6 of 10 prompts. On the other four the response said
web_search: false, carried no annotations, and cost about a thirtieth of a searched call. One of those four was F1, which it then got wrong in the same way as the model-only route. A tracker that sets the flag and never reads it back cannot tell these apart. - The interface route’s off switch did nothing visible. With
disableWebSearch: true, the ChatGPT interface route still cited sources on all ten prompts and got all four fresh questions right. It cited the September 2026 Google post for F1. Treat a route option as unverified until the response shows its effect. - Live is not the same as right. AI Mode answered F2 “No, Ahrefs did not make a general change to its core API rate limits” with two sources, and Gemini said the same in other words with five. Sources prove the answer was fetched, not that it is correct, so a tracker that stores them lets you check which pages the claim came from.
- A searching model can still carry a stale date. On one panel prompt the API’s search query, which it chose itself, ended in “2024”. Stored search queries show what the engine went looking for, which is the second thing to check after the sources.
Does model memory change the brand count?
On the six panel prompts, the live ChatGPT interface answer named 25 vendors from a fixed list of 48 data-API and ad-tool names, and the model-only API answer named 26. They shared 14. A score built on model-only answers would have counted a different set of vendors on every prompt. Both answers named three to five listed vendors on five of the six prompts, so a mention rate alone would not show the difference. Two routes and one capture each is too small to say more than that. The per-prompt lists are in scored.json under panel_brand_overlap.
What a tracker should store, and which routes give it
| Field | Why it matters | Returned by |
|---|---|---|
| Full answer text | The only way to read what was said, not just whether a name appeared | Every route |
| Cited URLs | Proof a fetch happened, and where a claim came from | Every route when sources exist; none on the model-only route |
| Fetch time | Answers change week to week; an undated answer cannot be compared | DataForSEO routes (datetime); the tracker must record it on the Cloro routes |
| Search flag or search queries | Separates a searched answer from a memory answer when both look fluent | DataForSEO routes (web_search, fan_out_queries); the Cloro ChatGPT route returned an empty searchQueries list on all 20 calls even when it was requested |
| Model name | Model changes move scores | DataForSEO routes (model_name); not on the Cloro routes |
Treg’s routed Perplexity endpoint labels its result source_kind: model_api because it calls Perplexity through an API rather than the website. That label does not mean the answer came from memory: the provider payload underneath said web_search: true on all ten calls. Read the provider’s own flag, not a wrapper’s label.
Who this fits
- You pay for a tracker: ask the vendor for the stored answer, sources and fetch date behind one score, then run two or three fresh-event prompts through it. If the answers carry no sources, or get last month’s news wrong without citing anything, the score is measuring model memory.
- You build your own: the interface routes are what a buyer sees; the API routes give you a model name, timestamp and search flag to audit. Running one of each on the same prompts, as here, lets you check them against each other.
What this run cannot tell you
One capture per prompt per route, US, on one day. Four fresh-event prompts is enough to show the difference between memory and search, not to rate the routes against each other: AI Mode’s and Gemini’s one miss each was the same Ahrefs question. Whether the Cloro ChatGPT route’s disableWebSearch option works on other prompts, or with other parameters, was not tested beyond these ten. Brand counts use a fixed name list, so vendors outside it were not counted.
Next step
All 70 calls ran through Treg’s CLI from one prepaid balance, $0.38 in total; the Cloro routes are also callable directly (its free plan grants 500 credits a month, about 55 ChatGPT or 83 AI Mode calls), and DataForSEO’s with an account of its own.
- Run the same check:
scripts/run-ai-answer-check.py evidence/fixtures/ai-answer-check-v1.json evidence/ai-answer-checks/$(date +%F), then grade the four fresh prompts and runscripts/score-ai-answer-check.pyon the folder. Runner · scorer · fixture. Replace the four fresh prompts with events from the last month before you run it; these will be in training data eventually. - Output: scored.json holds every answer, its sources and the fields above; raw responses are kept off the site.
- The Google AI Overview half of the same question: Is there an API for Google AI Overviews?
Questions this page answers
How do you verify an AI visibility tracker is actually pulling live answers?
Add three or four prompts about events from the last few weeks to the prompt set and read what the tracker stored for them. On 8 October 2026 the ChatGPT model API with search off got 0 of 4 such prompts right and cited no source on any of its ten answers, while the five routes that answer as the live products do got 3 or 4 of 4 right with sources on 49 of 50 calls. An answer with no cited URLs and a wrong or invented account of a recent event came from model memory.
Does turning on web search guarantee a live answer?
No. The ChatGPT model API with web search enabled searched on 6 of 10 calls; on the other 4 its response reported web_search false and no sources, and the call was billed about $0.001 instead of $0.03. The reverse also held: the ChatGPT interface route's option to disable web search changed nothing visible, and all 10 of its answers cited sources.
What should an AI visibility tracker store for each answer?
The full answer text, the cited URLs, when it was fetched, and evidence that search ran (a web-search flag or the search queries). A percentage alone cannot be checked. In this run only DataForSEO's LLM Responses routes returned the model name, timestamp and search flag in the response; the Cloro interface routes returned answer and sources only.
Relationship: the people who run this site also work on Treg, one of the routes compared on this page. Every Treg figure here is a call we paid for, with the raw response downloadable, and the same fixture is run against the alternatives where their terms allow.
Spotted an error or a changed price? Email corrections@seodatabench.com with the page URL and a source. Corrections are logged on the corrections page.