Studies test final self-improving systems more than the loop behind them
In plain words
This paper studies how research papers test AI systems that try to improve themselves. The authors read published studies and recorded whether each tested the finished system, checked the improvement process, or compared different improvement procedures.
In the 90 studies that called their own system or loop recursive self-improvement, tests of the finished system were more common than repeated checks of the whole process. The paper measures what studies reported; it does not show whether those systems succeeded or failed.
The research
A higher final score can support a claim that a system improved, but it cannot show whether gains build across rounds, recur when the loop is rerun, or come from a better improvement procedure. How often do published studies report designs that can answer those different process questions?
The authors coded published RSI studies to compare evidence about final system performance with tests of whether improvement loops repeat or improve.
The authors searched arXiv records using recursive self-improvement and related phrases, then coded 114 eligible studies, including 90 that called their own system or loop RSI. Their codebook linked each study’s claims to the evaluation designs it reported.
Among the 90, 54 reported evaluation on tasks unused for selecting changes, while 20 reported at least two audited reruns of the whole loop. This contrast shows that evidence about final systems was more often reported than evidence about repeatability.
These rates describe reported designs in a phrase-based arXiv frame frozen on 17 October 2026, not all self-improvement research. The study measures what papers report; it does not show that current loops fail to improve.
Analysis
Deep
- Original paper authors
- Zihan Chen, Di Zhu, Lei Nico Zheng
- Submitted to arXiv
- License
- arXiv non-exclusive