Studies test final self-improving systems more than the loop behind them

A higher final score can support a claim that a system improved, but it cannot show whether gains build across rounds, recur when the loop is rerun, or come from a better improvement procedure. How often do published studies report designs that can answer those different process questions?

The authors coded published RSI studies to compare evidence about final system performance with tests of whether improvement loops repeat or improve.

The authors searched arXiv records using recursive self-improvement and related phrases, then coded 114 eligible studies, including 90 that called their own system or loop RSI. Their codebook linked each study’s claims to the evaluation designs it reported.

Among the 90, 54 reported evaluation on tasks unused for selecting changes, while 20 reported at least two audited reruns of the whole loop. This contrast shows that evidence about final systems was more often reported than evidence about repeatability.

These rates describe reported designs in a phrase-based arXiv frame frozen on 17 October 2026, not all self-improvement research. The study measures what papers report; it does not show that current loops fail to improve.