ProgramGym trains agents to rebuild programs from behavior
In plain words
ProgramGym is a framework for training software agents to recreate programs from observed behavior. It turns repositories into practice tasks with a runnable reference, instructions, and checked tests. Its workflow separates exploring, building, and reviewing so evidence can guide revisions.
On tests comparing candidate and reference behavior, training Qwen3.8-27B on these task trajectories raised the average of each task’s test pass rate from 34.87% to 42.84% with the same Claude Code setup. Passing selected cases does not prove full equivalence.
The research
When an agent recreates software from documentation and an executable reference, it must discover behavior and make design choices without seeing the original code.
How can researchers give it reliable feedback across long tasks, when each environment can check only selected examples of the reference program’s behavior?
ProgramGym combines validated behavior tests for practice with separate analysis, implementation, and review sessions that carry evidence into revisions.
ProgramBench asks agents to reproduce a reference program’s behavior on selected inputs; its Tests Passed score averages each task’s share of tests passed.
With the single-session Claude Code setup held constant, Qwen3.8-27B trained on Program-Env trajectories scored 42.84%, versus 34.87% for its base checkpoint.
With model weights fixed, replacing that setup with Program-Agent raised the score from 34.87% to 55.71% for base Qwen3.8-27B, and from 42.84% to 56.25% for its trained version. These comparisons connect gains to both the training trajectories and the multi-session workflow.
The authors note that above-50% average test passing still coincided with near-zero rates of tasks almost or fully solved in these evaluations. Because each suite checks finitely many inputs, passing shows agreement on tested behavior, not complete equivalence.
What the evidence shows
- The task leaves choices open
ProgramGym is a framework for training agents to recreate programs from observable behavior. The task supplies usage documentation and an executable reference, but no source code or network access; an agent must discover requirements and make design decisions.
The paper identifies a lack of scalable environments that can reliably check this behavior as a central training problem.
- Practice needs dependable evidence
Program-Env converts repositories into practice environments with a reference executable, sandbox, documentation, and black-box tests. Build diagnostics help repair executables, while coverage feedback and validation help refine tests.
The final resource includes 1,648 environments across C, C++, Go, and Rust and 665,512 validated cases; mean repository-level coverage was 72.99%, measuring the share of valid measurable core source units reached by tests.
- Training lifts test passing
ProgramBench tests whether candidate programs reproduce reference behavior on selected inputs; Tests Passed is the average across tasks of each task’s test pass rate. With the single-session Claude Code setup held constant, training Qwen3.8-27B on Program-Env trajectories raised that score from 34.87% to 42.84%. The result associates trajectory training with higher average test passing in this setting.
- The workflow helps fixed models
Program-Agent tests the workflow contribution with model weights held fixed: compared with single-session Claude Code, its five-round analysis, implementation, and review workflow raised Tests Passed from 34.87% to 55.71% for base Qwen3.8-27B, and from 42.84% to 56.25% for its Program-Env-trained version.
Because the model is held fixed in each pair, these contrasts provide separate evidence for the scaffold.
- More rounds still leave a gap
On ProgramBench, the trained Qwen model with Program-Agent also scored 43.49% at one implementation round, 54.13% at three, and 56.25% at five.
These averages show that additional rounds coincided with more test passing in this condition, but the authors report near-zero rates of tasks almost or fully solved and conclude whole-program reproduction remains unsolved in their evaluation.
Deep
- Original paper authors
- Chengyi Yang, Renxu Bao, Jue Hou, Zongpei Teng, Ziheng Wang
- Submitted to arXiv
- License
- CC BY 4.0