Some models spot reliable sources without using that cue
In plain words
The paper introduces a controlled benchmark: 500 fictional text worlds that test how 13 language models handle conflicting reports. It asks two separate questions: can a model identify which source or evidence is more informative, and does that cue actually change its final choice?
In one source-history test, Qwen3.8-27B recognized the stronger source in 99.5% of cases, but switching the histories changed its choices by 0.0 percentage points.
Adding an explicit evidence-weighting rule later changed some models’ cue use, while answers backed by a trusted record stayed 100% correct in the tested cells.
The research
When evidence conflicts, a changed answer is ambiguous: did the model notice a relevant source history, let that history guide its judgment, or follow another cue such as report order?
A single answer-change score cannot separate those possibilities, making it difficult to tell whether a model’s apparent updating reflects the evidence researchers intended to test.
The authors separate recognizing a source’s reliability from letting it affect a judgment, then show prompts can reshape that influence for some models.
The authors built a 500-world evaluation with matched fictional scenarios, separate identification questions, and final support choices.
In matched pairs, one target cue—such as a source’s past record—changes while specified scenario features stay fixed. For unresolved choices there is no unique correct state, so influence is measured as the percentage-point change in the share choosing state B.
The result separates recognition from use: Qwen3.8-27B recognized the stronger source in 99.5% of unequal-history cases but showed a 0.0-point history effect, while GPT-6 Sol reached 100.0% on both measures.
For Qwen3.5-27B, correct identification still left the effect at 0.68 points versus 0.67 overall, so weak influence was not simply missed recognition.
These findings come from researcher-constructed English text scenarios, and the intervention tests only three Qwen checkpoints. Cue sensitivity here is behavioral and does not establish factual correctness, better real-world decisions, or trustworthy judgment.
Analysis
Deep
- Authors
- Jinchang Zhang, Kaijun Jiang, Guoyu Lu
- License
- CC BY 4.0