# One Screenshot of an AI Answer Is Not Evidence > Why a screenshot of one ChatGPT answer proves nothing about AI visibility, what a defensible claim needs, and how to read a vendor's number-one post. Canonical: https://webpossible.com/blog/one-screenshot-is-not-evidence/ Source: https://webpossible.com/blog/one-screenshot-is-not-evidence/ Format: Markdown version for AI agents. The canonical HTML page is at the source URL above. Last verified: 2026-09-04 --- Blog # One Screenshot of an AI Answer Is Not Evidence By Ryan York · last verified 2026-09-04 A screenshot of a ChatGPT answer with your product in it feels like proof. It gets pasted into a Slack channel, then a board deck, then a LinkedIn post that says the company is number one in ChatGPT for maintenance software. It is one observation, taken under conditions the image does not show. This post covers what that observation can and cannot tell you, what a claim about AI visibility needs before it counts as evidence, and the questions to ask the next time a vendor publishes one. ## What one answer is a sample of When you type a question into ChatGPT and read the answer, you are drawing one sample from a distribution you cannot see. The sample depends on at least five things the screenshot leaves out. Session state comes first. OpenAI’s help documentation describes a memory feature that carries details from earlier chats into later ones, and a temporary-chat mode that turns it off. A marketer who has spent a month asking ChatGPT about their own product is not a neutral sampler. The account has been told, repeatedly, that this product exists and matters, and the answer may be repaying the favor. Location comes second. OpenAI’s API documentation for the web search tool accepts an approximate user location and says results can depend on it. A property manager in Phoenix and one in Boston may get answers built from different forum threads and different vendor pages. Model version comes third. The single run in the September 2026 preview was taken through the OpenAI API and stored the model identifier it returned, gpt-5.5-2026-04-23, a dated snapshot rather than a product name. The app shows the product name and, in a capture, not even that: the Index reads ChatGPT from the app now, and no model id comes with it. The snapshot behind it changes without an announcement, and a screenshot from June and a screenshot from September may have been produced by different models. Phrasing comes fourth. “Best maintenance software” and “maintenance coordination software for a property management company with around 300 doors” are different questions and get different answers. The screenshot usually crops the question, or the question was the third message in a conversation whose first two set the answer up. Day comes fifth. Ask the same question tomorrow from a clean session and the answer may differ with none of the above having changed. We wrote up [where that run-to-run variance comes from](/blog/reading-variance-in-ai-answers/) separately. The short version is that retrieval, search fan-out and sampling each add noise, and a single answer cannot tell you how much. ## What the screenshot cannot rule out Put those together and the list of things one screenshot cannot exclude is long. It cannot rule out that the product appeared because the account had asked about it before. It cannot rule out that a slightly different phrasing produces a list without it. It cannot rule out that the same question ten minutes later omits it, or that the question shown is one no buyer asks. It cannot tell you what the other three engines said, because it shows one. None of this means the screenshot is false. The product was in that answer. What the screenshot cannot support is the sentence people write under it, which is a claim about a rate: that the product is usually named, or named first, or named more than a competitor. A rate needs more than one draw. ## What a defensible claim needs Four things, written down before the runs happen. A prompt set, meaning the exact text of every question, frozen so nobody rephrases the ones that gave a bad answer. The engines, meaning which ones and through what capture method, since an API call with web search and a consumer app with memory on are different instruments. The runs, meaning how many per prompt per engine, each in a fresh session. The dates, spread over more than one day, so a bad afternoon at one engine does not become a month’s result. The Index protocol is one way to meet that bar and the [methodology page](/ai-shortlist-index/methodology/) states it in full. Twenty-five prompts per category. Four engines: ChatGPT from its app, Perplexity and Gemini through their APIs, Google AI Overviews through a licensed data provider. Three runs per prompt per engine, in separate sessions, on at least two days. The model identifier stored per run wherever the engine gives one, and recorded as absent on ChatGPT, where it does not. Every raw capture written once to a file that is never edited, so a third party can check the extraction against it. A vendor named in the prompt text does not get credit for being named in the answer. You do not need the Index’s volume to make a defensible claim about your own product. You need the shape: fixed prompts, named engines, repeated runs, dates, and the raw answers kept. ## Our own preview is a screenshot We hold the Index to the same standard, which means saying plainly what the September 2026 preview is. It is one prompt, mvn-best-01, sent to ChatGPT through the OpenAI API on 2026-09-03, one time. The answer named Property Meld first and Latchel second and cited 124 sources while doing so, with reddit.com the most-cited domain at 38 citations. The row on the category page shows Property Meld at rank 1 and Latchel at rank 2, and if you stopped reading there it would look like a result. Keep reading and the row shows what it does not know. The consistency field is empty, because a single run cannot agree or disagree with itself. Three of the four engine columns are marked missing rather than zero. That is the difference between a row and a screenshot. The row carries its own uncertainty on its face. The [notes from that first run](/blog/how-to-measure-ai-visibility-for-property-ops-software/) go through the four metrics and why one run does not fill them. ## How to read “we’re #1 in ChatGPT” When a vendor posts that claim, the questions are the same ones the protocol answers. What was the exact prompt, and was your brand in it? How many times did you run it, and on which dates? Which engine, through the app or the API, and which model? Was memory on? Can I see the raw answer, and what did Perplexity and Gemini say to the same question? A post that cannot answer the first three is a screenshot. It may be a true observation. It is not a measurement, and it should not move anyone’s budget. If a screenshot lands in your own Slack channel, the useful move is to treat it as a prompt candidate. Add the question to a set, run the set properly, and see whether the answer holds. The [AI visibility measurement guide](/ai-visibility/) covers how to set that up, and the rest of the field notes are on the [blog](/blog/). Part of [the AI visibility measurement pillar](/ai-visibility/).