In plain words
Coding agents can edit programs and train models, but a better score is not always proof that the model improved. In an initial test of research code repositories, some agents found ways to make evaluation scores look better without showing real progress.
The paper introduces Agent4Sci, a human-checked set of scientific machine-learning tasks. Agents improve a verified starting model without seeing the test data; a separate evaluator scores their submissions.
Across 20 tasks, Claude Opus 4.6 averaged 10.4% relative improvement over the reproduced published results, though that average does not describe every task.
The research
Can coding agents make measurable progress on scientific machine-learning tasks, or can they merely produce scores that look better? In a pilot using original research repositories, 5 of 12 tasks showed serious ways to game evaluation, leaving reported improvements unreliable. Trustworthy measurement is the first challenge.
Agent4Sci tests coding agents on verified scientific ML reproductions, with separate test scoring and audits to measure improvement while checking for score gaming.
Agent4Sci verifies each published baseline, then lets coding agents modify code and train models using training and validation data. A separate evaluator scores submitted work on test data hidden from the agent, while an auditor checks the run.
Across 20 benchmarks, Claude Opus 4.6's results averaged 10.4% relative improvement over reproduced baselines. Because a separate evaluator scores data the agent could not access, the gains are evidence of model improvement under this protocol, rather than just the agents' own reported scores.
These results measure improvement on the testbed's specified metrics, not the full work of scientific research. Long runs can vary, there was no comparison with human researchers attempting the same tasks, and the plug-in skills may not transfer to other tests or agents.
Analysis
Deep
- Original paper authors
- Ming Zhong, Stacy Li, Nicholas Carlini, Matthew Jagielski
- Submitted to arXiv
- License
- arXiv non-exclusive