In plain words
The paper asks why unrelated text can weaken a language model’s correct answer even when that text gives no evidence and suggests no rival answer. The researchers tracked information about the question’s subject and the model parts that contribute to its answer.
Subject information stayed present, while some attention heads shifted toward the unrelated text and gave less support to the answer.
A fixed mask that blocked those heads from reading the context partly restored accuracy across four tested English tasks. The result applies to the models and task settings studied.
The research
A correct answer can lose support even when added text says nothing about the question, so removing all context could also discard useful evidence.
These findings make a narrower choice plausible: restrict reads from selected heads while keeping other heads unchanged, with partial recovery in the tested tasks.
The paper traces how irrelevant context weakens answer support, then tests a fixed mask that blocks context reads in selected attention heads.
In paired factual-recall runs, the researchers tracked subject information and head contributions with and without unrelated text. Measured subject enrichment stayed effectively unchanged, but selected heads shifted attention toward the prefix and contributed less to the correct answer.
A fixed mask on those heads partly recovered distractor-related accuracy loss across four downstream tasks. This supports a selective interference pathway; broad restoration of middle-block writes also removed some benefit from helpful context.
The tests covered open-weight, decoder-only base models and controlled English tasks. Larger or different architectures, other languages, multi-turn dialogue, and adversarial inputs were not tested; the mask also requires internal attention access and model-specific head selection.
What the evidence shows
- Separate lost facts from interference
The paper’s mechanistic account explains how irrelevant context can weaken a correct answer without providing evidence or proposing a competing answer. Its key question is whether the model simply loses information about the query’s subject, or whether the answer computation changes while that information remains available.
The distinction matters because a lower answer score alone does not identify what in the computation changed.
- Subject information remains available
In paired factual-recall runs, measured enrichment for selected subject-associated attributes stayed effectively unchanged when unrelated text was added, even as support for the correct answer could fall. The selected attention heads, however, shifted some attention toward the prefix and contributed less to the answer.
This contrast supports an account in which measured subject information remains available while its contribution to the final answer weakens.
- Trace where answer support falls
The source-resolved analysis found that prefix-derived writes promoted distractor-related content, but contributed almost no direct support change for the correct answer on average. The answer-support reduction came mainly from weaker non-prefix contributions, including query positions.
That distinction means distractor promotion alone does not account for the measured loss of support for the correct answer.
- Broad restoration also removes benefits
Restoring middle-block writes from a context-free run recovered support under irrelevant prefixes, but the same broad change also removed some benefit from helpful context. The result shows that this intervention site can affect the answer computation, while also showing why broad restoration is not selective. The authors therefore tested a fixed mask on context reads by selected heads.
- A sparse mask partly recovers accuracy
Across four downstream tasks, the fixed mask recovered a mean 56.9% of distractor-induced accuracy loss for the tested 7B-scale models; the percentage is the recovered share of the clean-to-distracted accuracy drop, averaged across tasks.
A matched-clean residual patch recovered 97.6% on average, but required a clean pass. The comparison supports partial recovery from a sparse mask, with the clean-informed patch serving as a diagnostic reference.
Deep
- Original paper authors
- Dikshant Kukreja, Gautam Gupta, Avni Mittal, Avinash Anand, Kritarth Prasad
- Submitted to arXiv
- License
- CC BY 4.0