A kit makes local coding-agent comparisons clearer
Contain, Conform, Compare (Extended Version): A Red-Teamed Evaluation Kit for Small Coding Agents and the Full Record of Its Experiments · arXiv:2611.16485
In plain words
A single competition score can make two coding agents look different just because the hidden tasks vary. drydock-eval is a kit for checking local agent runs and comparing versions more carefully.
It runs jobs in restricted containers, checks whether local requests match the competition setup, and compares agents on the same tasks.
In one test, four prompt differences for the E4B model produced no detected change in solve rate on 231 tasks from 15 repositories. That result applies to this model, agent bundle, and task panel; it does not show the differences have no effect everywhere.
The research
A few solved tasks can separate coding agents on a leaderboard, yet one score on hidden tasks is noisy enough to blur such differences.
A local result can guide comparisons only when requests match the scorer and candidate versions face the same tasks; this kit checks those conditions and makes the comparison explicit.
drydock-eval combines isolated agent runs, scorer-request checks, and repository-aware paired comparisons to make local coding-agent tests easier to interpret.
The kit was used to test four differences between local E4B prompts and the competition scorer’s prompts. A checker verified each job, and researchers compared task-weighted solve rates across four repeats on each of 231 Berth tasks from 15 repositories.
No difference was detected for any prompt change. The largest estimate was a 2.1-point drop when history thoughts were removed, but the uncertainty interval still allowed a larger drop. This supports a bounded conclusion about these conditions, not a claim that the prompts are interchangeable.
These findings cover one competition, the tested models and agent bundles, and the specified task panels and machines. Local solve rates did not predict hidden-task scores, and the third candidate’s confirmation reused a task panel and followed earlier results.
What the evidence shows
- A noisy score needs checks
drydock-eval is an evaluation kit for small coding agents. It combines a container runner, checks that compare local serving requests with the competition’s scorer path, and paired comparisons that account for tasks from the same repository tending to succeed or fail together.
The kit addresses a concrete problem: a single score on hidden tasks can be too noisy to separate agents that differ by only a few solved tasks.
- Three parts of local evaluation
The kit’s modules address separate questions. Contain runs agent jobs in fresh, restricted containers; Conform checks whether the local system’s requests match the scorer configuration; Compare tests candidate and baseline on the same tasks and repeats.
Together, they make local comparisons more interpretable, though a result still describes the task panel and conditions that were actually tested.
- More solved tasks did not settle it
The paired comparison asks whether each configuration solved more of the same tasks than BF5; it detected no difference. Because some totals combine revisions across task sets and SH6 was developed while results were being examined, these comparisons do not isolate a single change.
- Four prompt changes, no detected shift
Against the scorer-path setup, the four paired prompt comparisons found no detectable change in E4B solve rate under the preregistered test.
This does not show the prompts are interchangeable: removing history thoughts had the largest estimated drop, with uncertainty that still allows a meaningful decrease, and that arm also timed out more often.
- A screen lead needs confirmation
The researchers also screened variants and then checked selected candidates on separate task panels under a frozen pass rule. None of three candidates passed.
One estimated gain was 4.10 percentage points, but its p-value missed the required threshold, and its confirmation reused a panel after an earlier result was known. A screen leader is therefore a candidate to check, not evidence by itself that an agent improved.
Deep
- Original paper authors
- Nguyen Minh Thang
- Submitted to arXiv
- License
- CC BY 4.0