Training retrievers on their own search histories
Retrieving from Your Own Mistakes: On-Policy Training of Embedding Models for Multi-Hop Retrieval-Augmented Generation · arXiv:2610.40409
In plain words
A retriever can collect several passages to answer a question. It is usually trained using correct passages as its history, but later it must work with passages it found itself. A wrong passage can stay in the history and make the next search harder.
The paper introduces a way to train the retriever on some of its own histories, including mistaken ones, while still teaching it which correct passage is missing.
On one test set, this raised the share of questions with all evidence in the top five passages from 44.6% to 50.5%, compared with training only on correct histories.
The research
When a chain retriever finds a wrong passage, that passage stays in its query and can hinder later searches. Does training on the histories a retriever actually reaches help it recover, compared with training only on correct histories? The paper tests this question for multi-hop passage retrieval.
On-policy state training teaches chain retrievers from their own evolving search histories, improving recovery from mistaken passages and complete evidence retrieval.
On-policy state training changes which histories a chain retriever learns from: alongside histories containing correct passages, it trains on states reached by its current retriever, including mistaken histories. It labels each state from the gold evidence and refreshes visited states each epoch.
In a same-seed MuSiQue comparison, the method raised complete coverage at five passages from 44.6% to 50.5% of test questions. This means more questions had every gold passage in the top five, supporting the value of training on the states the retriever actually reaches.
The main comparison is a same-seed ablation on MuSiQue, and most experiments use one shared seed. Longer-chain transfer was tested without further training on two constructed datasets, so these results do not establish the same gains in every retrieval setting.
What the evidence shows
- Training and test histories differ
Chain embedding retrievers find evidence for multi-hop questions one passage at a time, adding each find to the query for the next step. The training mismatch is that they usually learn from histories made only of correct passages, then face their own retrievals at test-time.
In a MuSiQue probe, wrong passages were already present in 70% of feasible states at step three and 89% at step four for four-hop questions; the teacher-forced retriever next found a missing gold passage less often in contaminated than clean states.
- Teach from its own searches
The paper introduces on-policy state training, a method that trains a chain embedding retriever on both gold histories and states reached by its current retrieval chains. Each visited state is labeled using the gold evidence: the target is the earliest missing gold passage, and other missing gold passages are masked.
The query encoder is updated while the passage index remains fixed, and visited states are recollected each epoch so training follows the changing retriever.
- More complete evidence retrieved
On MuSiQue, the same-seed ablation reached 50.5% complete coverage at five passages, compared with 44.6% for teacher forcing on gold histories. Complete coverage counts a question only when all gold passages appear in the top five.
This comparison used the same encoder, training recipe and state budget, so it tests a change in training-state distribution within that experiment.
- Recovery after mistakes improves
A probe on the same fixed states showed where the gain appeared. For clean histories, next-passage success was similar for teacher forcing and refreshed on-policy training, at 67.0% and 69.0%.
After two or more wrong passages, it was 14.5% and 29.2%. Controls also found that contamination explained part of the coverage gain, while refreshing visited states with the current encoder added more in the MuSiQue experiment.
- Evidence extends to longer chains
Without further training, the MuSiQue-trained retriever also had higher complete coverage at ten passages than teacher forcing on HotpotQA-Long and 2Wiki-Long, two constructed longer-chain datasets. This is evidence of transfer in those evaluations, not a guarantee for other collections.
Most experiments used one shared seed, and two graph-based systems in broader comparisons were evaluated on reduced corpora.
Deep
- Original paper authors
- Chenjie Qiu, Zixuan Zeng, Shaobin Shi, Bingfeng Chen, Dongsheng Luo
- Submitted to arXiv
- License
- CC BY 4.0