← Back to GEO Academy
Comparisons

If a Page Is Highly Relevant, Did the AI Actually Use It? GEO's Attribution Illusion

Compare BM25, semantic similarity, and a causal leave-one-out probe in a new generative-search study, then apply an evidence ladder that prevents GEO reports from overstating attribution.

Published 08/28/2026 7 min read
GEO attributionevidence utilizationcausal probinggenerative search

If a Page Is Highly Relevant, Did the AI Actually Use It? GEO's Attribution Illusion

A page can resemble both the question and the answer without having caused or supported the generated wording. Semantic proximity establishes potential relevance; it does not establish generative reliance. Presenting a similarity score as content contribution is an attribution error in a GEO report.

A paper released August 24, 2026, *The Laws of Context Allocation*, built same-query hard negatives: passages that were topically dense and relevant to the query but did not contain or entail the target answer. With easy off-query distractors, BM25 and embedding measures appeared nearly perfect. With same-query hard negatives and a three-document context, BM25 fell to an AUC of 0.444 and query–document cosine similarity to 0.484—roughly chance discrimination.

The paper's deletion leave-one-out probe achieved 0.876 in that setting and 0.824 at 24 documents. It held the generated response fixed, removed one context document at a time, and measured the reduction in token-level likelihood of that same response. If removing a document makes the realized answer materially less likely, that is stronger evidence of reliance.

Keep three signals separate

Retrieval relevance asks whether a document fits the query. A visible citation asks whether the system displayed it as a source. Counterfactual dependence asks whether the same answer would be harder to sustain without it. All three may align, but they frequently do not.

In an internally controlled RAG system, a team can hold the model, prompt, and realized answer constant, ablate documents, and record likelihood or material-claim changes. The validation set should include hard negatives from the same query, not only obviously unrelated pages.

For closed products such as ChatGPT, Gemini, or Perplexity, an external brand normally cannot observe the complete context pool or token probabilities. A practical report should use an evidence ladder instead: topical similarity; visible citation whose page supports the claim; stable co-occurrence across repeated runs; and, only in a controlled system with a predefined ablation, cautious causal-dependence language.

Build an attribution table that does not overclaim

For every material fact, preserve the question, complete answer, visible sources, page version, capture time, claim-to-source support, and degree of experimental control. Put “possibly relevant,” “displayed as a citation,” “supports this claim,” and “changed under ablation” in separate columns rather than compressing them into a precise-looking influence score.

GEO Radar at https://www.georadar.top can observe brand mentions, platform differences, citations, and historical answers, supporting the first three evidence levels. It provides an auditable observation layer; semantic similarity or co-occurrence alone should not be described as proof that a page caused an answer.

The study used controlled RAG systems, open-weight models, and question-answering benchmarks. It does not show that every commercial answer engine uses the same context orchestration. Its immediate value is methodological: harder negatives and more disciplined attribution language.

Sources for this article