Most tested models match an older language model at classifying search conflicts
In plain words
Search results can disagree for different reasons: several answers may be valid, a topic may be contested, information may be old, or a source may be wrong. Each reason calls for a different kind of answer.
The paper tests models that choose among five reasons before an answer is written. On 406 expert-labeled queries, eight of 11 models were statistically tied with Gemini 2.5 Flash, though one did significantly worse. The test supports these models as classifiers in this setup, not as answer writers.
The research
When search results disagree, an answer generator may need to combine valid answers, present opposing views, prefer current information, or discount a false source. The question is whether a separate classifier can identify the right kind of conflict reliably enough to guide that response.
The paper tests typed decision models as conflict classifiers and finds that most match an older language model on expert-labeled queries.
The Conflicts test contains 406 queries whose search results have expert-assigned conflict types. The authors ask whether typed decision models can identify those types as well as a language model, using the same queries and input content for both.
Eight of 11 decision models are statistically tied with Gemini 2.5 Flash, which classifies 61.3% correctly. A cascade that sends each model’s least-confident quarter of queries to Gemini 3.8 Flash is tied with that model alone for eight cascades, at 25–37% of its cost.
These results support the classifiers as a lower-cost decision step in this setup, while the answer test measures judged behavior rather than factual grounding.
These results concern conflict labels on a small test set, including only four natural misinformation queries. The answer test used one generator and judges that could not see the search sources, so it measured expected behavior rather than factual grounding.
What the evidence shows
- Identify the disagreement
The paper studies a step that happens before a search-based answer is written: identifying why retrieved pages disagree. The reason matters because several valid answers, a contested issue, outdated information, and a false source call for different responses.
The authors test typed decision models, which choose among five conflict types, as classifiers for this task.
- Compare on labeled queries
The authors use Conflicts, a test of 406 queries with expert-assigned conflict labels and retrieved search results. Each decision model receives a query and its results and selects the most probable of five types.
The main comparison measures the share of queries classified correctly against Gemini 2.5 Flash rerun on the same queries, allowing paired comparisons of each system’s predictions.
- Most match the earlier model
The 11 decision models range from 50.0% to 67.0% accuracy; Gemini 2.5 Flash scores 61.3%. Eight models are statistically tied with Gemini, two are significantly more accurate, and one is significantly less accurate.
Through the tested gateway, decision models also have lower billed cost and mean latency, though models and interfaces differ across the comparison.
- Route uncertain cases
The authors test whether confidence can reserve a stronger language model for uncertain queries. A cascade sends the least-confident quarter of each decision model’s predictions to Gemini 3.8 Flash.
Eight of 11 cascades are statistically tied with Gemini alone at 25–37% of its cost. This is evidence of a cost-quality tradeoff in this test, not proof of equal accuracy.
- Probe formats and downstream use
Two additional tests qualify the result. Breaking the five-way choice into yes-or-no questions does not reliably improve classification.
On constructed cases with one rewritten false source, top-label detection is often weak, although the models’ probabilities usually rank altered cases above clean ones. In an answer-generation test, results for predicted types depend on which language model judges whether answers followed the expected behavior.
Deep
- Original paper authors
- Parv Khatri, Harshul Jain, Mehak Kapoor, Tanya Sah
- Submitted to arXiv
- License
- CC BY 4.0