When to trust an LLM judge
Does agreement between judges establish validity?
Agreement between evaluators, agreement with experts and sensitivity to a prompt are different measurements. Name the property you need before deciding whether a judge’s score answers your question.
- Judge agreement
- Do evaluators give similar scores?
- Independent reference
- Do scores agree with expert labels?
- Sensitivity
- Do wording or visible labels alter the result?
Three questions to take into your reading
- Is the target judge agreement or agreement with independent labels?
- What information and labels can the judge see?
- Would another evaluator or response budget change the conclusion?
Follow the evidence
Match judging evidence to the outcome and its cost
Distinguish the outcome a judge measures from its cost.
A panel of judges can give more consistent scores after its instructions are clarified, yet show no detectable improvement against expert ratings. A response can also look relevant without answering the request. These cases make “better judging” an incomplete goal until the outcome is named. The studies support choosing evidence for the claim at hand: clarify criteria to study consensus, inspect an answer’s substance to assess request matching, and observe an application in use when its behavior matters. Each choice has a tested scope and a cost; a budget planner depends on how well its pilot represents deployment.
Judge disagreement clarifies rubrics but not human alignment
Examine a case where judge consensus and agreement with experts are measured separately.
Disagreement-guided rubric edits raised judge agreement on news summaries, with no detectable gain in expert match.
Visible labels can mislead language model judges
Inspect how visible decision labels affect a judge in entity alignment.
Tests find that visible system labels can distort language model judgments of entity alignment decisions in the studied conditions.
Jailbreak rankings shift with evaluators and attempt settings
Check how different success rules alter a reported comparison.
A study finds jailbreak attack rankings shift with evaluator rules and attempt protocols across tests of three small open-weight models.
Where the conclusion stops
Results from a rubric study, entity alignment and jailbreak evaluation belong to different settings; one does not validate the others.