ReproBridge turns recovered experiment steps into reusable guides
ReproBridge: Building Reproducibility into ML Research Artifacts · arXiv:2611.08452
In plain words
Machine-learning papers and released code can leave out practical steps, such as which settings to use or how to turn run outputs into a reported score. ReproBridge is a workflow that records those decisions in reusable code, setup instructions, experiment launchers, and a guide.
In tests on selected experiments from 36 papers, four coding agents had higher average reproduction success with these artifacts than with released code alone. Success meant matching a verified numerical reference under a separate audit rule, so it did not certify the science itself.
The research
A reproduction can depend on decisions that are scattered across a paper, code, and the work of getting an experiment to run. Whether those decisions survive for the next attempt changes what a successful code release alone can reasonably be expected to provide. In this test, the reusable handoff was associated with higher average reproduction success.
ReproBridge packages recovered experiment steps and decisions for reuse, then tests whether fresh coding-agent sessions reproduce selected measurements more successfully.
Fresh sessions from four coding agents attempted the same 1,029 verified measurements from 36 papers, using either released code or a refined artifact, with the same 24-hour limit per paper. Success required a result within a fixed interval around human reference measurements and no confirmed violation found by the primary Auditor.
Mean success was 76.4% with raw code and 92.6% with the artifact, averaged equally across the four models and over three evaluations. This supports the value of the complete handoff in this test, but does not show that every attempt improved.
The evaluation covered selected tasks from 36 papers and four coding agents, with success defined by numerical agreement and an audit rule. Agreement does not certify that an experiment is scientifically valid or fully faithful.
What the evidence shows
- Preserve recovered procedures
ReproBridge is a workflow for making recovered experiment procedures reusable. Its four-part handoff contains refined code, environment setup, experiment launchers, and a guide that records both executable steps and reasons for settings or repairs.
The question is whether that preserved work helps a later, independent attempt reproduce measurements, compared with giving the reproducer the released codebase.
- Compare fresh attempts
With the same selected tasks and scoring rule, the complete handoff increased average success across the four tested agents.
This supports a benefit from the full construction-and-handoff intervention in this benchmark, but numerical agreement and passing the audit do not establish scientific validity.
- Count gains and reversals
The paired-outcome check shows that recoveries outnumbered reversals in the first evaluation, but the average gain did not mean every task-model attempt improved. Some attempts changed from success with raw code to failure with the artifact.
- Probe the handoff
The ordinary-output control suggests that keeping code, scripts, and logs alone did not account for the full result; because the handoffs came from separate upstream runs, the comparison tests construction and handoff together.
Component-removal tests also reduced success, but those losses depend on the other materials left available.
Deep
- Original paper authors
- Yubo Wang, Rui Meng, Jaehyun Nam, Tomas Pfister, Jinsung Yoon
- Submitted to arXiv
- License
- CC BY 4.0