Chiosa
HCI & Social

Some models spot reliable sources without using that cue

2 min ·

In plain words

The paper introduces a controlled benchmark: 500 fictional text worlds that test how 13 language models handle conflicting reports. It asks two separate questions: can a model identify which source or evidence is more informative, and does that cue actually change its final choice?

In one source-history test, Qwen3.8-27B recognized the stronger source in 99.5% of cases, but switching the histories changed its choices by 0.0 percentage points.

Adding an explicit evidence-weighting rule later changed some models’ cue use, while answers backed by a trusted record stayed 100% correct in the tested cells.

The research

When evidence conflicts, a changed answer is ambiguous: did the model notice a relevant source history, let that history guide its judgment, or follow another cue such as report order?

A single answer-change score cannot separate those possibilities, making it difficult to tell whether a model’s apparent updating reflects the evidence researchers intended to test.

The authors separate recognizing a source’s reliability from letting it affect a judgment, then show prompts can reshape that influence for some models.

The authors built a 500-world evaluation with matched fictional scenarios, separate identification questions, and final support choices.

In matched pairs, one target cue—such as a source’s past record—changes while specified scenario features stay fixed. For unresolved choices there is no unique correct state, so influence is measured as the percentage-point change in the share choosing state B.

The result separates recognition from use: Qwen3.8-27B recognized the stronger source in 99.5% of unequal-history cases but showed a 0.0-point history effect, while GPT-6 Sol reached 100.0% on both measures.

For Qwen3.5-27B, correct identification still left the effect at 0.68 points versus 0.67 overall, so weak influence was not simply missed recognition.

These findings come from researcher-constructed English text scenarios, and the intervention tests only three Qwen checkpoints. Cue sensitivity here is behavioral and does not establish factual correctness, better real-world decisions, or trustworthy judgment.

Analysis

Deep