Coding-agent harness choice changes lightweight models’ results
Pairing Matters: Performance, Failures, and Efficiency in Lightweight Coding Agents · arXiv:2610.32131
In plain words
A coding model can work with different software agents. Each agent, called a harness, gives it repository information and tools, then runs and checks its proposed code changes.
The authors tried six harnesses with three lightweight models on the same coding tasks. For a given model, the share of tasks resolved differed by as much as 44 percentage points across harnesses. The result applies to these tested pairings and tasks.
The research
When a coding model performs poorly, is the model itself the problem, or could a different software agent change what it accomplishes?
Earlier reports often tested fixed model–harness pairings, leaving limited evidence about how harness choice affected the same model’s resolution, failures, and token cost.
The authors compare every pairing of six agent harnesses and three lightweight models on the same coding tasks to measure how the harness changes each model’s results.
SWE-bench Verified tests whether software agents resolve coding tasks in repositories; its grades show whether tasks were resolved.
The authors paired six open-source agent harnesses with three lightweight coding models, and each pairing attempted the same 100 tasks under common serving, metering, and grading conditions.
For the same model, resolution rates differed by as much as 44 percentage points across harnesses, while harness rankings changed across models.
Because the task set and model were shared in the within-model comparison, the results show that harness choice changed measured performance in this evaluation.
These findings describe three lightweight coding models, six open-source harnesses, and the 100 SWE-bench Verified tasks tested under the study’s shared conditions. They do not establish how other pairings or task sets will perform.
What the evidence shows
- What the comparison measures
This paper is an empirical evaluation of lightweight coding-agent pairings. A coding-agent harness manages repository context, tool access, execution, validation, and patch submission for a coding model.
The question is whether changing that harness changes the same model’s results. SWE-bench Verified tests whether software agents resolve coding tasks in repositories; its grades indicate whether each task was resolved.
- A shared set of tasks
The authors paired six open-source harnesses with three lightweight models. Each of the 18 pairings attempted the same 100 SWE-bench Verified tasks, producing 1,800 graded runs under common serving, metering, and grading conditions.
Comparing harnesses for the same model on the same tasks makes harness choice visible within this evaluation.
- Results vary by pairing
For the same model, resolution rates differed by as much as 44 percentage points across harnesses, and harness rankings changed across models. The result means that a model’s measured task-resolution rate depended on the harness in this study; it does not mean every harness changed rank for every model. The authors therefore treat the model and harness as a joint selection decision.
- Failures differ too
The authors also classified 1,100 unsuccessful runs using an evidence-based taxonomy of five behavioral failure mechanisms. Failure patterns differed across pairings, so resolution rates were not the only measure that changed with the pairing. The reported evidence does not specify the category names or their separate rates.
- Efficiency depends on pairing
Among model-directed loop harness pairings, charged tokens per resolved task varied by more than 20-fold. This measure counts charged tokens against tasks resolved, and the reported variation applies to that subset of harnesses.
Together, the results support comparing pairings on task resolution, failure patterns, and token use within the tested models and tasks.
Deep
- Original paper authors
- Lu Zhao, Christopher M. Poskitt, David Lo
- Submitted to arXiv
- License
- arXiv non-exclusive