2026-08-24
2 min read
Why I stopped trusting one screenshot
A screenshot of an AI answer is one observation under one set of conditions. Here is what it took to learn that, and what a reading looks like instead.
A screenshot of an AI answer is one observation under one set of conditions. It is not a ranking, not a trend, and not a diagnosis, and for longer than I like to admit I treated it as all three. Someone would send me an image of ChatGPT naming a competitor and not naming them, and I would do what everyone does with a screenshot: read it as the state of the world.
The state of the world is much harder to see than that. One screen records one surface, one date, one phrasing of the question, one location, and whatever the session already knew. Change any of those and the answer changes, sometimes completely. When Hendricks sent the same 17 buyer questions to three engines in one archived run, the pattern was not a competitor standing above anyone. On the run published 2026-08-21 (run id 2026-08-20-110653), ChatGPT cited any source at all on 2 of 17 questions, the run logged 449 citations in total, and no single source was cited by all three engines. On the probe baseline dated 2026-08-18, hendricks.ai was cited in 1 of 45 cells, and only 21 of those cells cited any source at all, so the denominator that actually mattered was 21. Kevin Indig's H1 2026 halftime report (2026-07-27) found 91 percent of citations appearing in one engine only. The picture is wide and shallow, and a single screenshot is a random tile pulled out of it.
That is why I stopped trusting one screenshot, and why the rule I now hold every reading to is short: absence is not yet a diagnosis. A single answer screen is one observation under one set of conditions. The word yet is the whole argument. It says an absence can become a diagnosis, but only after repeated runs under defined conditions, and it says a screenshot cannot get you there on its own.
If you cannot state the number of runs, the dates, the engines, the customer contexts, and the denominator behind the evidence you are about to act on, you are not reading a result. You are reading a screenshot. The reading is the work; the screenshot is what happens before the work starts.
One finding, one reading, one thing I built. Every Thursday.
One email a week. Unsubscribe in one click.
Until the list is live, subscribing is one email. Send it and you are on the first issue.