Reading Variance in AI Answers Before You Call It Movement

Ask ChatGPT the same question in two clean sessions and, as the sampling and web search behavior in OpenAI’s own API documentation would predict, you can get two different vendor lists. That is a property of the instrument, and any measurement of AI visibility has to be designed around it rather than wished away. This post goes through where the variance comes from, what the Index protocol of three runs per prompt per engine on at least two days can and cannot bound, what the consistency metric is for, and how to decide whether a rank change is movement or noise.

Four places the variance comes from

Retrieval is the first. OpenAI’s documentation for the web search tool in the Responses API describes the model deciding whether to search, writing its own queries, and composing the answer from the pages those queries return, so which pages come back depends on the query the model wrote, the state of the search index at that moment, and the location signal on the request. The app works the same way and reports less about it: the Index reads ChatGPT from chatgpt.com now, which shows the links in the answer and not the queries or the pages behind them. In the one run in the September 2026 preview, taken through the API on 3 September 2026 before that change, the ChatGPT answer to prompt mvn-best-01 cited 124 sources, 38 of them on reddit.com. A different set of threads surfacing on a different afternoon produces a different answer with nothing else having changed.

Search fan-out is the second, and it compounds the first. Google has described AI Mode as issuing several related queries on the user’s behalf, a technique it calls query fan-out, and the Gemini API’s grounding metadata returns the list of web search queries the model ran for a given answer. Each of those sub-queries is its own retrieval with its own variance, and an answer built from four searches has four chances to draw a different page.

Sampling is the third. OpenAI’s API reference documents temperature and top_p as parameters that control how the next token is chosen, and the Index leaves every engine that exposes them at its defaults rather than forcing them to zero. That is a deliberate choice, and on ChatGPT it is not even a choice: the app exposes no sampling controls at all. A property manager in the app does not set temperature, so measuring at zero would produce a tidier number about a situation no buyer is in. Even at zero, the retrieval step above still moves.

Model updates are the fourth, and the one most often mistaken for a vendor’s own doing. The run of 3 September 2026 stored gpt-5.5-2026-04-23 as the model identifier the API returned. OpenAI’s model documentation describes dated snapshots behind a product alias, and the alias can move to a new snapshot without the consumer app saying so. A vendor that drops three places the month a snapshot changes may have done nothing at all. That is why a model identifier is stored per run and shown per edition wherever a capture returns one. The ChatGPT app returns none, so on that engine this source of variance is real and invisible, and the Index says so rather than implying the model held still.

What three runs on two days bound

The methodology sets the protocol. Three runs per prompt per engine, each in a fresh session with its own session id. Runs spread across at least two days, with the scheduler firing on the 2nd, 4th and 6th of the month at 14:00 UTC, one run index per day. ChatGPT is the exception, because a browser session cannot run unattended on a server: its three runs are captured by hand on three days at least two days apart, and imported with the time each answer was collected. A verification step checks that every prompt and engine cell has three runs, on two or more distinct days, in distinct sessions, before an edition may publish.

That design bounds three things. It bounds a single bad afternoon at one engine, because the other two runs are on other days. It bounds session contamination, because no run shares conversation state with another. And it bounds the single retrieval draw, because a vendor has to be named across separate searches to score a full mention rate.

It does not bound everything, and what it leaves should be stated. Three draws is a small sample. A vendor named in two of three runs has a mention rate of 0.67 on that cell, and the honest confidence interval around that number is wide. A model snapshot can change between the 2nd and the 6th, inside the window. And three runs say nothing about the personalized answers real accounts get, since the Index measures the unpersonalized baseline on purpose. Three is a cost decision. At 25 prompts, four engines and three categories, an edition is 900 responses, and the number of runs is the lever that multiplies that.

What the consistency metric is for

Consistency is the share of prompt and engine cells, with at least two runs, where all runs agreed, meaning the vendor was named in every run or in none. A value of 1.0 means the engine gave the same verdict on that vendor every time it was asked. A low value means the engine flips between runs. The metric is displayed on every row and never hidden.

Its job is to tell you how much to trust the mention rate next to it. Two vendors can carry the same mention rate of 0.5 and be in different situations. The first has consistency near 1.0, so it is named on half the prompts every time and missed on the other half every time. That vendor has a prompt problem, and the fix is to find which prompts it is absent from and why. The second has consistency near 0.2, so most of its cells are coin flips. That vendor has a retrieval problem, and the fix is in which sources its mentions ride on, which the source dependency field shows.

In the September 2026 preview the consistency field is empty, because one run cannot agree or disagree with itself. That blank is the metric working as designed.

Telling movement from noise

The movement field is the prior edition’s rank minus the current one, positive when a vendor moved up. Before reading it as a story, run four checks.

Check the engine and version record first. If the model identifier changed between editions, movement across every vendor in the category is suspect until the pattern shows otherwise. Check the changelog second, for a prompt set version change, since a retired prompt and its replacement can shift who gets named. Check for degraded engines third. A degraded engine is excluded from the denominators, so a blended mention rate over three engines is a different number from one over four, and the rank built on it is too. Check consistency last. A rank change where both editions carried high consistency is more likely to mean something than one built on cells that flipped.

Then look at engine spread. A vendor that moved on all four engines moved. A vendor that moved on one of four may have caught a different forum thread on that engine. And look at the source dependency. If a vendor’s mentions rode on one thread and that thread dropped out of the results, the drop has a mechanical explanation before it has a strategic one.

One rank on one edition is not a trend. Two editions in the same direction, with the checks above clean, is the earliest point at which we would say a vendor’s position changed. The preview has no prior edition, so movement is null for every row, and it will stay null until October.

What to do with a single flip

If your own quick check flips, the temptation is to act on the flip. Someone runs the question, the product is missing, and a meeting gets scheduled. That check is one draw from a distribution nobody has seen, which is the same problem as a screenshot that shows the product present, pointed the other way.

The useful response is the protocol in miniature. Put the question in a fixed prompt set. Run it three times in fresh sessions on two days, on each engine you care about. Keep the raw answers. Then read the rate and the consistency together, and only then decide whether anything changed. The AI visibility measurement guide lays out that setup for a single vendor, and the rest of the field notes are on the blog.