AI visibility tracking data is not entirely reliable. Because generative models often elicit different responses, the citation shares and rankings on your dashboard are just snapshots of an ever-changing target, not hard facts.
A difference between you and a competitor may be real or simply relate to a variation between measurements. A new IQRush paper due to be published next week (we had access to the pre-publication) offers a way to differentiate and shows that no fixed amount of data can definitively settle the question.
The article is by Ron Sielinski, co-founder of IQRush, a company that sells software that measures AI visibility in the way the article claims. The reason it’s worth your time is that a separate team published a similar result of repeated measurements in April, so IQRush isn’t alone in making this case.
How much these numbers move
Querying SearchGPT, Gemini, or Perplexity repeatedly with the same question may result in different sources each time. They are constructed to add some randomness to each answer, so that each citation is just one of many possible URLs it could have retrieved. A previous article by the same author examined this variability and showed that, for example, when testing SearchGPT on running gear, Tom’s Guide accounted for about 9.5% of citations, while Runner’s World accounted for about 6.0%. Tom’s Guide appeared on the dashboard more often, but the numbers overlapped due to the large margin of error. With only one sample, it was incorrect to say that Tom’s Guide outperformed Runner’s World, as the 3.5 point difference was within the margin of error. The new paper aims to prevent this mistake by answering a simple but often overlooked question: How much data is needed for rankings to be truly meaningful?
If a ranking is trustworthy
The answer has two parts, and both must be true for a ranking to be reliable. First, the order must stop changing.
In the beginning, rankings can change frequently as new answers are added, as no site yet has a clear lead. Only after enough answers have been collected do the top pages stand out clearly and the order stabilizes. It is also important that the top locations are far from each other. If they are very close, the rankings may not be meaningful, as close competition doesn’t really show who is really ahead. The article examines whether the difference between the top locations is greater than the margin of error for each. If this is the case, the ranking reflects a real difference. If it doesn’t, it’s probably just statistical noise. Both conditions must be met at the same time; neither alone is sufficient. Across 30 platform topic tests, the number of responses required to satisfy both conditions ranged from 33 to 94, considering only responses with quotes.
Three out of 30 failed to reach this point even after 125 questions, all on SearchGPT, where the top sites were too similar to distinguish. There is no uniform limit that applies everywhere; What works for one platform and topic may not work for another.
We circled this
In January, I discussed SparkToro’s discovery that more than 99% of the time, when you ask the same question, AI tools return a different list of recommended brands. This article left one question unanswered: How many times do you need to check before the results stabilize? This paper provides the clearest answer I have ever found.
Rand Fishkin, who led this study, offers some helpful advice. Before you spend money on tracking AI visibility, he suggests making sure your provider “does their math.” The IQRush paper is a great way to do this because it includes a simple stopping rule so you don’t have to just rely on your intuition about how many runs are enough.
It also fits in with a number of studies SEJ has conducted over the past year, each of which reports AI citation counts as if they were fixed. He turns around, examines the measurement itself and asks whether these numbers are even stable enough to be compared.
What this changes for your reporting
The number on your dashboard is just a single example. Before trusting it, check whether your tracker runs the same check repeatedly and reports a range, or whether it fetches the data once and returns a clean value. The clean number may actually be a warning sign rather than a reassurance.
A win after a content change can easily be misinterpreted. For example, a three-point increase in your SearchGPT citation share might seem like evidence that your efforts were worthwhile, but such a change may be within the natural variability of successive runs, according to the original paper’s data.
To win, measure before and after more than once each. A single before and after reading cannot distinguish your change from the usual noise.
The platform you are measuring changes the amount of data you need, and not in the way you would think. What matters is how much independent information each answer contains, not how many citations it gives you. Gemini stacks quotes from the same few sites in a single answer, so many of those quotes will tell you the same thing. SearchGPT reports fewer citations per answer, but spreads them out so that each answer contains more independent information than the raw number suggests. The same number of responses on two search engines does not buy the same trust, and a budget that takes Gemini into account can leave you guessing on SearchGPT.
Sometimes the honest answer is that you can’t tell yet. Three of the 30 tests never cleanly separated their top sites within budget. For these, the right decision is to hold and not publish a ranking that the data cannot support. A tracker that can tell you that “there is not enough data” is worth more than one that gives a safe instruction every time you ask.
The top of the leaderboard is the part you can defend the most. With enough answers, the leaders retreat from the middle and the end, although even they are not accurate. The margins of error expand quickly below the top until the neighboring positions are just a coin toss, and even the top 10 was not flawless, with the typical margin of error on a top 10 page being around five positions and one in five being wider than 10. Trust the leaders, treat the middle and bottom as rough, and don’t give exact positions beyond the top of the list.
What the paper doesn’t prove
None of this comes from a finished, peer-reviewed study. It is a preprint based on 30 platform topic tests in three engines and uses ChatGPT-generated questions instead of real user searches in a single collection section. The exact numbers cannot be clearly transferred to your topics. Therefore, consider them as the form of the problem and not as a lookup table.
These counts only include answers that contained citations, which is most important in SearchGPT because a portion of questions return no citations at all. For one topic, 125 questions resulted in 104 usable answers, which corresponds to an error of 17%. So you would need to submit more questions than this total suggests.
The method is also tested internally. The paper compares a ranking it gives at the beginning with the final ranking of the same collection, not with an external ground truth. This checks whether the stopping rule is consistent with itself, which is why the unaffiliated team’s matching result does real work here. The authors of this April publication, Julius Schulte, Malte Bleeker and Philipp Kaufmann, are researchers at the University of St. Gallen. They ran a separate data set and came to the same conclusion, that a single reading is unreliable and you have to sample an engine repeatedly to trust what it says.
Where this leads
The paper falls short of what most people want, which is a way to know your running budget before you start collecting. Sielinski leaves this for later work, noting that the number depends on the shape of each platform’s citation pattern, so there is unlikely to be a consistent overall budget.
The bigger change is that AI viewability reporting is moving toward the way ads and analytics reporting already did, toward numbers that have a margin of error instead of a false decimal point. This happens while the basics are still missing because Search Console still doesn’t tell you which clicks come from AI. Until then, the onus is on you to run the check more than once and report the range, not the single number your dashboard shows you.
More resources
Featured Image: Sticky/Shuttertstock
Follow us on Facebook | Twitter | YouTube
WPAP (907)
