Hidden checks expose coding agents’ incomplete infrastructure changes
InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell? · arXiv:2610.36589
In plain words
The paper introduces InfraDev, a set of tests for coding agents asked to recreate real changes to AI infrastructure. It adds hidden checks for requirements that a project’s own tests may miss, and gives agents task descriptions that state what the change must do without telling them how to build it.
When evaluators added the hidden checks, they rejected 32% of submissions that had passed all the official tests. That share applies only to submissions that passed those official tests.
The research
When a pull request’s own tests pass, do they show that an AI infrastructure change truly works, including any promised speed or memory savings?
The paper found that official tests often omit key numerical or behavioral requirements, so a passing patch may still fail to deliver the change it was meant to make.
InfraDev pairs extra hidden checks with task descriptions to test whether coding agents complete real changes to AI infrastructure.
InfraDev pairs 36 real changes from three repositories with hidden checks for requirements missed by official tests. Its task descriptions state the required behavior and interfaces without giving the implementation.
Evaluators graded the same submissions with and without the extra checks: probes rejected 32% of submissions that passed all official tests. Across the 36 tasks, GPT-6 Sol led six agents, resolving 38.9% under the authors’ task-resolution measure.
InfraDev covers 36 manually selected changes from three repositories, and the agent results are limited to six tested systems. Each run used an eight-GPU node for up to 2.5 hours; expanding the test set still requires domain experts to select suitable changes.
What the evidence shows
- A test for real changes
InfraDev is a test set for coding agents asked to reproduce real changes in AI infrastructure for training, inference, and reinforcement learning. It draws 36 tasks from three production repositories.
A task counts as resolved when the agent’s patch passes at least as many test cases as the developers’ completed change, giving the authors a way to compare task completion across agents.
- Passing tests can mislead
The authors found that the pull requests’ official tests often missed important requirements. For 13 of 15 performance changes, those tests did not check whether the change actually saved time, memory, or communication. So a patch could pass while failing to deliver the behavior the change promised.
- Extra checks change grading
InfraDev adds hidden probes, or extra tests for requirements the official tests miss. Its construction process first creates incorrect patches that pass the official tests, then writes probes to catch the missed requirements.
When evaluators added these probes to grading, they rejected 32% of submissions that had passed every official test. The comparison shows why official-test success alone can overstate completion.
- Results point to next steps
Across all 36 tasks, GPT-6 Sol had the highest resolved rate among six tested agents, at 38.9%. On the 19 Megatron-LM tasks, neither GPT-6 Sol nor DeepSeek V4 Pro resolved a task when given the original pull request descriptions and graded with hidden probes.
In a separate comparison on those Megatron-LM tasks, GPT-5.6 Sol’s resolved rate rose from 42.1% to 52.6% when it received a reusable development skill based on observed failure patterns.
Deep
- Original paper authors
- Yangshen Deng, Jiaqi Ruan, Yeqi Huang, Nick Lau, Guangming Sheng
- Submitted to arXiv
- License
- arXiv non-exclusive