Match judging evidence to the outcome and its cost
Questa pagina non ha ancora una traduzione verificata. È mostrata la versione inglese. Leggi la versione inglese
A panel of judges can give more consistent scores after its instructions are clarified, yet show no detectable improvement against expert ratings. A response can also look relevant without answering the request. These cases make “better judging” an incomplete goal until the outcome is named.
The studies support choosing evidence for the claim at hand: clarify criteria to study consensus, inspect an answer’s substance to assess request matching, and observe an application in use when its behavior matters.
Each choice has a tested scope and a cost; a budget planner depends on how well its pilot represents deployment.
Cosa comprenderai
How can agreement among judges differ from agreement with human ratings?
What does reconstructing a request from an answer measure, and where can that proxy fail?
How do observed application behavior and a fixed computation budget affect a judging choice?
A panel can agree without matching experts
Imagine using several language models to grade news summaries. Each receives the same rubric, a set of scoring instructions, yet the judges can read a clause differently and assign different scores.
If an editor makes their scores more alike, the panel becomes easier to use. But a reader who cares about the quality of the summaries needs another answer: do those scores resemble expert ratings? The rubric study treats agreement within the panel and agreement with experts as separate outcomes.
On SummEval, a benchmark of news summaries, the rubric study compared the same evaluation items under the original instructions and an edited version. Across five judges, it measured agreement with Krippendorff’s alpha, a statistic for agreement beyond chance.
Averaging the paired alpha differences equally over four scoring dimensions, the edited version scored 0.069 alpha units higher than the original; the reported 95% interval ran from 0.046 to 0.092. This selected comparison measures how consistently those judges scored, not whether their scores were right.
The study also compared the panel’s scores with existing expert ratings on SummEval and found no detectable change after the edits. That finding does not establish that expert agreement stayed exactly the same; it says the study did not detect an improvement in that outcome.
It also makes the practical question sharper. If consistency, agreement with people, and correctness are different targets, what evidence should a judge receive, and how much work should be spent gathering it?
Come prosegue la lezione
Change the judgment, then check its target
Give the judge evidence from use
Account for the cost of gathering evidence
Keep the decision tied to its tested setting