Models can learn from textbooks while proving unfamiliar math
Incentivizing Self-Learning in Large Language Models via Reinforcement Learning · arXiv:2610.39671
In plain words
Can a language model learn a new idea from a textbook while solving a problem? This paper introduces a method where the model searches and reads textbook sections as it works on a mathematical proof; the exercise’s solution is hidden.
After reinforcement-learning training, the method reached 79.5% proof accuracy across three math test groups, compared with 73.1% for a method that retrieved sections before answering. That average does not describe every problem or model.
The research
A model that can use a long reference book while solving a problem may handle unfamiliar material differently from one that receives retrieved passages all at once.
This result changes the comparison: interactive study can outperform fixed retrieval on these mathematical proofs, though the advantage depends on the training setup and does not carry across every tested task.
A textbook-based self-learning method trains language models to study reference sections while reasoning, then apply them in mathematical proofs.
Across three mathematical test splits, the trained method averaged 79.5% proof accuracy, compared with 73.1% for a trained baseline that retrieved sections before answering. Proof accuracy counts sampled proofs accepted against a reference solution, with eight reviews required to agree.
The model’s accuracy also rose from 51.5% before reinforcement-learning training to 79.5% afterward. These results support interactive textbook study in this setup, though the before-and-after comparison does not isolate which training signal drove the gain.
The proof results average four sampled solutions per problem and rely on a judge that accepts a proof only when all eight reviews agree. Transfer was selective: every RL-trained policy scored below the initial model on CL-bench’s Procedural Task Execution category.
What the evidence shows
- Learning while proving
The textbook-based self-learning method is a training approach for large language models solving mathematical proofs. It lets a model choose what to study from a paired textbook while reasoning, because relevant ideas may be scattered across sections and the exercise solution is withheld.
The central question is whether training can help a model turn that on-the-task study into a valid proof.
- Interactive study beats fixed retrieval
The comparison suggests that choosing textbook sections as reasoning unfolds can outperform retrieving sections in advance when both approaches are trained. This result applies to the three tested math splits and this fixed-retrieval baseline.
- Training changed the result
The change with training is also substantial: Self-Learning’s overall proof accuracy rose from 51.5% before reinforcement-learning training to 79.5% afterward, averaged over the same three splits. This supports the view that the initially weaker interactive procedure benefited from training.
The before-and-after comparison does not identify which part of the training recipe caused the improvement.
- Feedback sustains study
Under correctness-only reinforcement learning, trajectories with a textbook call fell from 36.9% to 6.7% of all trajectories, compared with the initial policy.
Adding round-level feedback improved proof accuracy on held-out problems from training books, suggesting that feedback on study steps can complement final-proof correctness in that split.
- Book access and transfer have limits
With the trained model, proof accuracy was 79.5% when its paired textbook was available and 71.7% without the book or tool, suggesting access contributed beyond unaided problem solving in this test.
The method also improved on two external tests, but transfer was uneven: every reinforcement-learning-trained policy scored below the initial model on CL-bench’s procedural-execution category.
Deep
- Original paper authors
- Lingkai Kong, Yuzhe Gu, Haiteng Zhao, Zijian Wu, Youqing Fang
- Submitted to arXiv
- License
- CC BY 4.0