Repeated facts can distort document evidence tests

When a document repeats a fact, should a system get credit for finding a valid copy, and can removing one marked passage show whether an answer depends on that fact? Single-location scoring and masking can give misleading answers when another sufficient instance remains in the document.

The authors annotate repeated fact locations and test how scoring and masking change when valid copies count as evidence.

To test whether marking one evidence location per fact distorts evaluation, the authors annotated independently sufficient locations and compared single-gold scoring with evidence-aware scoring.

On 46 MMLongBench-Doc questions with strict copies across pages, evidence-aware scoring raised top-page recall by 13.0 to 23.9 percentage points across four retrievers; the difference was smaller across all 208 applicable questions.

The same issue affected citation comparisons: under revised scoring, 11.4% of pairwise orderings reversed among 65 MMDocRAG systems with original scores of at least 0.5. Masking tests also showed that a surviving copy can hide fact-level dependence. Together, these results support scoring evidence as a set when facts have strict copies.

The annotations may miss copies visible only in page images, and one language model produced all labels. Independent reviewers found missed copies, but their review covered selected subsets, so the reported rates may undercount redundancy.