AI visibility tests show inconsistent results

A recent analysis has found that AI visibility tests, which check how often a brand is mentioned by AI assistants like ChatGPT, Perplexity, Claude, and Gemini, often give different results when repeated under what seem to be the same conditions. This means that percentage changes in visibility metrics might not reflect real progress or decline.

Same test, different outcomes

In one example, a test conducted in August on the same website with the same questions and AI engines showed that only two of five competitors were mentioned in both runs. On a website not cited by any engine, the "not cited" result was consistent 45 times out of 45, but the list of domains cited changed significantly, with only one of nine domains appearing in both runs.

Zero does not mean absence

A test run on October 22 found that none of the answers named the brand on any of the four AI engines. However, the true citation frequency could range from 0% to 16%, meaning that a measured zero does not necessarily mean the brand was not mentioned at all. This highlights the need for more accurate data representation.

Confidence intervals improve clarity

The statistical solution is to provide each number with a confidence interval. Instead of stating "35%," one might say "35%, somewhere between 20% and 50%." This helps distinguish between real changes and random fluctuations in the data.

Confidence intervals build trust

For marketing agencies, using confidence intervals helps maintain client trust by avoiding misleading claims about visibility. It shifts the focus from guaranteed results to honest comparisons and makes real progress more credible when it happens. Confidence intervals should be included in initial reports, not added later.

AI model changes affect results

Changes in AI models can also influence test outcomes. For example, ChatGPT updated its model between two readings spaced a month apart, which could affect the results. These changes are not captured by confidence intervals and should be communicated to clients as a potential factor in the data.