Opus 5.5 with Claude Code scores above human references on average
Singularity-Bench: Can Agents Build Agents? · arXiv:2611.16591
In plain words
Singularity-Bench is an open-ended test of whether AI agents can build systems used to develop future agents. It gives agents six tasks, from training models to managing computing clusters, and compares their work with systems built by people.
The leading agent scored above the human references on average across the six tasks. But on one task, training a coding agent, its score was close to the human-built recipe's, and tests did not distinguish them.
The benchmark measures a limited set of tasks, not the full process of AI systems building their successors.
The research
Assessing whether AI agents can build the systems behind future agents depends on tests that allow meaningful design choices and catch shortcuts that inflate scores.
Singularity-Bench offers a way to compare agent-built systems with human references across the development stack; its results suggest the leading agent can match or exceed those references on average, with variation by task.
Singularity-Bench tests whether frontier AI agents can build systems across AI development, comparing their work with human-built references on six open-ended tasks.
On six open-ended tasks, evaluators compared agent-built systems with production systems built by people on the same hardware. The leading agent, Opus 5.5 with Claude Code, scored above the human references on average, supporting the conclusion that it could match or exceed them across much of this test.
That average has an important qualification: on post-training, its mean resolve rate was close to the Miles human-built recipe's, and paired tests did not distinguish the two. The results show capability on these tasks, not completion of the full process of AI systems building successors.
The six-task index is an average, and the benchmark's tasks are smaller than those needed to measure the full recursive self-improvement process. The attack finding covers only the attacks tested and listed in the paper.
What the evidence shows
- Testing end-to-end building
Singularity-Bench is an open-ended test of whether frontier AI agents can build systems used to develop future agents. It addresses a measurement problem: some existing tests restrict what agents can change, simplify the setting, or may fail to catch shortcuts.
The benchmark gives agents room to design core components and checks both their systems and the integrity of evaluation.
- An above-reference average
Each agent's Singularity-Index averages its scores relative to the human-built reference for each of six tasks; the reference value is 1. Opus 5.5 with Claude Code scored 1.252, above that average reference.
The index gives each task equal weight, so this is evidence of above-reference performance on average, not a claim that every task score exceeded its reference.
- Performance varied by task
The aggregate lead masks a close post-training comparison: Opus 5.5's mean resolve rate was 0.533, versus 0.539 for the human-built Miles recipe, and paired tests did not distinguish them. So the above-reference average does not mean it outperformed human references on every task.
- Evaluation needs safeguards
Open-ended systems can exploit evaluation procedures to gain unfair scores. In the paper's adversarial tests, both plain Harbor and Harbor with an AI judge were vulnerable.
Breakwater added safeguards to check the environment, restrict access, stop lingering processes, preserve external logs, and limit execution before verification; it caught every attack listed in the study.
- A bounded measure of progress
Together, the results show how a test can compare agent-built systems with human-built references while checking for studied forms of score manipulation.
They do not measure agents completing the full process of recursive self-improvement: the authors describe the tasks as a smaller-scale proxy, and the reported defense result is limited to the attacks tested.
Deep
- Original paper authors
- Yang Zhou, Krish Agarwal, Zhuoming Chen, Zerui Wang, Zhenyu Zhang
- Submitted to arXiv
- License
- CC BY 4.0