Agents can improve research code, but many tasks stump all

Can a language-model agent improve a published computer-science system inside its own code repository, and meet the standard set by the implementation its authors released?

Earlier open-ended coding tests described by the authors score a single program after a run, leaving this research-code question unanswered.

FrontierCS 2 tests agents inside research code repositories, judging their chosen patches against the papers’ released implementations on private final workloads.

FrontierCS 2 creates tasks from research repositories by removing each paper’s contribution and keeping the removed code as the reference. Agents submit patches and receive development feedback, but the judge privately checks the agent’s chosen final patch against that reference on final workloads. A run passes only if every metric meets its threshold on every workload.

Across 48 tasks, the top of 12 model-and-harness agents passed 58% of its runs; no agent passed any run on 16 tasks. The result shows that some agents can meet this demanding standard on a share of tasks, while also revealing many tasks where none did.

A higher combined score often still missed a required threshold, so score and pass status capture different things.

These results describe one run per agent on each of the 48 reported tasks. FrontierCS 2 also has a separate private set of more than 100 tasks, which was not evaluated in this report.