Decision models catch some hallucinations, but confidence can mislead
Decision Models as Judges for Hallucination Detection · arXiv:2610.37229
In plain words
This paper evaluates decision models, judges that choose between preset labels such as “target error present” and “target error absent,” for spotting five kinds of hallucination. It compares five such models with a generative judge, using existing dataset annotations to check their verdicts.
The top model varies by error: decision models nearly match the generative judge on context conflicts and slightly exceed it on fabricated facts, but trail on reasoning contradictions and instruction failures. Their confidence only moderately signals whether a verdict is right.
The research
When software is used to judge model answers, can it spot hallucinations without depending on another generative judge, and can its confidence show when it is wrong? A judge that produces unsupported verdicts or stays sure while making errors undermines the check it is meant to perform.
The study compares five models that choose preset verdicts with a generative judge, then tests whether routing uncertain cases among them improves detection.
Five decision models and a generative judge received the same responses and target-error criterion; researchers compared their present-or-absent verdicts with dataset annotations. Balanced accuracy means average recall for responses with and without the target error.
The strongest decision model varied by error type: it nearly matched the generative judge on conflicts with supplied context and slightly exceeded it for fabricated facts, but trailed on contradictions within a response and failures to follow instructions. This supports type-specific performance, not a universal winner.
These pooled comparisons cannot isolate error type from dataset: context and instruction inconsistency each come from one source. Cascade replay cross-fits rate choice, but its model members and order were selected using the full sample, and no live or unseen-domain test was run.
What the evidence shows
- Define the target error
The paper’s empirical study evaluates five decision models as judges that spot hallucinations in model responses, comparing them with the generative judge gpt-6-sol. Researchers turn existing annotations into five separate present-or-absent tasks, one for each target error type.
An “absent” label means only that the named error is absent; a response can still contain another kind of hallucination.
- Compare verdicts fairly
Each judge sees the same response and task criterion, plus the original instruction when available and source context for items about context conflicts. The researchers score decisions against dataset labels.
Balanced accuracy is the average recall for responses with and without the target error, so it does not simply count total correct answers. The best decision model varies by type: it nearly ties the generative judge on context conflicts and slightly exceeds it for fabricated facts, but trails on within-response contradictions and instruction failures.
- Check confidence separately
The models’ confidence is a separate question from whether their labels are right. Researchers found calibration slopes below 1 in 21 of 25 decision-model-by-error-type pairs, a sign of overconfidence.
A ranking measure asks whether a correct verdict is more confident than an error, counting ties as half; 0.5 is chance. Averaged across five types, it ranges from 0.67 to 0.73 for the six judges.
High-confidence errors remained: 10–30% of four decision models’ errors and 47% of gpt-6-luna, a decision model, and 69% of gpt-6-sol, the generative judge, had confidence of at least 0.9.
- Route uncertain items
The paper also tests a confidence cascade: the strongest of three decision models decides first and sends its least-confident items to the next model; the third handles items passed onward.
In cross-fitted replay, cascades improve on the first model for factual and context errors, but their measured differences from gpt-6-sol have intervals containing zero for those three types.
- Bound the cascade result
That benefit is not universal. For contradictions within a response, the cascade makes no deferrals and equals its first model, remaining 3.85 percentage points below gpt-6-sol.
For instruction failures, it gains 0.36 points over its first model, with an interval spanning zero, and remains 7.43 points behind. Cross-fitting lowers the selected peak by no more than 0.52 points, but checks rate selection only, since model membership and order still came from full-sample scores.
Deep
- Original paper authors
- Haozheng Liu, Zhaoqian Xue, Zhongyin Wang, Junting Mu, Zihui Ma
- Submitted to arXiv
- License
- CC BY 4.0