Tests reveal gaps in models that build coding-agent software
Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents · arXiv:2610.20344
In plain words
Doc2Harness tests whether a language model can build the software around a coding agent. This software, called a harness, manages tools, permissions, saved session state and recovery from failures.
A model builds a candidate, then a scripted stand-in model exercises the frozen result in test sessions. In the 19-model study, even the highest-scoring model’s harnesses failed many behavior tests on average across three builds. This measures the built software’s behavior, not success on real coding tasks.
The research
When coding-agent tests keep the surrounding software fixed, they leave one concrete question unanswered: can a model build that software well? The harness manages tools, context, permissions, session state and recovery, all of which shape what the agent can do. Doc2Harness makes that construction task testable.
Doc2Harness is a two-stage test: models build coding-agent harnesses, then scripted sessions score the share of behavior tests each frozen build passes.
Each model built three candidates from a development document and an interface-only skeleton. A scripted model drove each frozen candidate in test sessions, and assertions checked behavior. Pass rate means the fraction of cases passed, excluding positive controls.
Across 19 models, GPT-6 Astra’s candidates averaged a 53.05% pass rate and Hy3’s 27.44% across 328 cases and three builds. This spread, with failures even for the leader, supports the authors’ conclusion that the suite distinguished this cohort without saturation.
The candidates were frozen after a form-only check, so the scores do not show whether models could improve a harness using behavioral feedback. A scripted mock model drove the tests, so they do not measure success on real coding tasks.
What the evidence shows
- The untested construction skill
Doc2Harness is a two-stage benchmark of whether large language models can build the software around coding agents. Its score is the share of behavioral test cases passed by each built harness, an indirect measure of the model’s construction ability.
Existing coding-agent tests generally keep this software fixed and measure task completion, leaving the construction skill untested.
- Build from behavior documents
During construction, a model works inside a fixed builder setup and receives a development document describing candidate behavior, plus a starting skeleton that contains interfaces but no implementation. It writes a complete candidate in an isolated container.
A form check confirms that the candidate compiles and installs, has required commands and interfaces, and contains no placeholders; it does not run behavioral tests. A passing candidate is frozen.
- Test full scripted sessions
In the second stage, a scripted mock model drives each frozen harness through a complete session for each test case. Assertions inspect requests sent to the model, files on disk, process exit code and command output.
Some cases have positive controls that check whether the behavior and its effects can be observed. A failed control makes its case fail, and controls do not count toward the pass-rate denominator.
- Scores show a broad spread
The authors evaluated 19 frontier models from 11 vendors, building three independent candidates per model and scoring them on 328 test cases. Pass rate is the share of cases passed, excluding positive controls from its denominator; each model’s rate is the mean across its three builds.
GPT-6 Astra led at 53.05%, versus 27.44% for Hy3, so even the leader’s candidates failed nearly half the cases. The spread shows the suite could distinguish this cohort without saturation.
- One overall order misses variation
The overall order does not hold across every part of the suite. Across six main categories, each with at least 20 cases, model rankings shifted by as many as 16 positions.
GPT-6 Astra led four categories; Grok 4.7 and Qwen3.8 Max led the other two. Since the categories use different test cases and have different sizes, these shifts qualify the overall ranking but do not isolate category-specific ability.
- More modules accompany lower scores
A separate source-code analysis covered the 57 scored candidate harnesses. After statistically accounting for code lines, the authors found a partial correlation of −0.423 between module count and pass rate.
Repeating the analysis while leaving out each model in turn kept the association negative, between −0.484 and −0.309. More modules tended to accompany lower scores in this set, but the analysis does not show that module count caused the difference.
Deep
- Original paper authors
- Yuyao Ge, Shenghua Liu, Peizhong Gao, Chendi Ge, Chenxu Liu
- Submitted to arXiv
- License
- CC BY 4.0