Sauron checks evidence before judging agent tasks

When an agent completes a long task, evidence of what happened may sit in its final environment state or in the record of its actions, while a rubric judge may see only a claim. Can a verifier check those records closely enough for its verdicts to track the evidence?

Sauron checks claims against cited state and trace evidence, then accepts rubric verdicts only when verified claims support them.

To test whether rubric judgments can stay grounded when evidence is scattered across an agent’s response, final state, and action record, the authors evaluated Sauron on personal-assistant tasks. How well does checking cited evidence translate into judgments that agree with human labels?

On 100 simulated episodes, Sauron scored higher than Gandalf against human judgments with GPT-5.6 Sol: 0.96 versus 0.92 F1, a combined precision-and-recall measure pooled across 2,586 criteria. Both judged the same trajectories with the same model, but their procedures and trace access differed. The result supports Sauron’s overall approach, not trace access alone.

Evidence checking does not settle every disagreement about what a rubric criterion means: in a manual review of 65 PA100 disagreements, 48 involved Sauron reading criteria more strictly than annotators. The evaluation also does not test agents optimized to exploit Sauron as a reward.