A test of whether agents carry experience forward
Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents · arXiv:2610.33956
In plain words
The Continual Learning Exam tests whether language-model agents can use what they learn on one work assignment to do later assignments. Its 120 tasks each have five connected steps, and an agent must pass each step to reach the next.
Across 17 tested setups, allowing feedback and one revision after a failed submission raised median completion, measured as the share of all 600 levels passed, by 5.33 percentage points.
The trial comparison combines both changes, so it does not isolate the effect of experience carried from one task to another.
The research
Many tests score tasks one at a time, so they cannot answer whether an agent’s experience helps with later work. It remains unclear how to measure that carryover across assignments where later work can depend on earlier information, files, and procedures.
The Continual Learning Exam links professional assignments into gated task ladders to measure whether agents carry experience forward.
Can agents use experience from earlier assignments to perform better on later ones? The Continual Learning Exam links five distinct assignments in each professional task and requires an agent to pass each level before advancing.
A separate evaluator checks submissions against human-defined criteria, while overall completion counts passed levels out of 600.
Across 17 configurations, median completion rose by 5.33 percentage points when agents could receive feedback and revise after a failed submission. The comparison supports a benefit from that combined opportunity, but the best tested configuration still passed fewer than half of all levels.
The trial comparison changes feedback and the chance to revise together, so it cannot show how much experience retained between tasks contributed. The results describe the tested configurations under this task order and grading procedure.
What the evidence shows
- The missing measure
The Continual Learning Exam (CLE) is a benchmark for testing whether language-model agents can retain and reuse experience across professional assignments. Many existing tests score tasks separately, while memory tests ask agents to retrieve information from histories.
Neither directly measures whether experience improves performance on later task execution. CLE addresses that gap with linked tasks and a measure of progress across a sustained stream.
- Assignments build forward
Each of CLE’s 120 tasks is a five-level ladder of distinct assignments in a shared professional context. Later levels may depend on earlier files, information, procedures, strategies, user preferences, or domain knowledge.
Agents advance only after passing the current level. This makes later work depend on earlier progress, while a failure can leave the remaining levels in that task unreached.
- How performance is scored
A separate evaluation agent checks each submission against human-defined rubric criteria and reference materials, which remain hidden from the agent being tested. Every rubric item must pass.
Trial 1 allows one submission per level without diagnostic feedback; Trial 2 gives feedback after a failed first submission and allows one revision. Overall completion is the share of all 600 levels passed, with unreached levels counted as failures.
- Feedback helps, within limits
Across the 17 configurations, median overall completion rose by 5.33 percentage points from Trial 1 to Trial 2. The comparison measures the combined change from diagnostic feedback and an extra revision opportunity.
The best tested configuration in Trial 2 passed 47.67% of levels, showing that the added opportunity did not make full-stream completion routine among the tested systems.
- Progress fades across ladders
The median cumulative pass rate across configurations fell from 62.50% at the first level to 8.33% at the fifth. These rates count the share of 120 tasks in which the first one or five levels were all passed, so the fifth-level rate is full-task completion.
The steep gap shows that early progress often did not carry through the whole ladder. Since the trials also do not isolate retained knowledge, they cannot establish its contribution to performance.
Deep
- Original paper authors
- Haoyu Zhao, Sitian Tu, Yuxuan Huang, Huichi Zhou, Zhangyue Yin
- Submitted to arXiv
- License
- CC BY 4.0