Retrieval techniques lose gains when combined
Retrieval Augmented Generation Compositionality · arXiv:2610.29722
In plain words
Retrieval-augmented generation, or RAG, means finding documents and using them to help write an answer. This paper introduces lasagna, an open-source framework that lets researchers swap a pipeline component and test it alongside the rest of the system.
The authors found that three techniques with strong separate retrieval results produced a smaller gain when combined in one tested pipeline. They also found that finding useful documents and writing answers grounded in them did not reliably improve together across the configurations they studied.
The research
When a retrieval system combines several techniques, do the gains measured for each one survive in the full pipeline? And does a pipeline that finds relevant documents also produce answers grounded in them? The authors test these questions across pipeline configurations and corpus sizes.
The authors built lasagna, a stage-isolated framework for testing RAG components in full pipelines, then compared choices across retrieval, answer grounding and corpus sizes.
The authors ask whether improvements from individual retrieval techniques survive when combined in a full pipeline, and whether the same choices help retrieve documents and ground answers. They built lasagna, a framework that tests components inside a complete RAG system.
In one comparison, three techniques predicted a 46.6% retrieval gain over the default standard pipeline but achieved 17.2% together on the same test split. Across 113 configurations, retrieval quality and answer faithfulness were nearly uncorrelated.
These results show that isolated gains do not reliably predict combined performance, and retrieval alone does not indicate whether answers are grounded.
These findings come from one constructed English test bed, with at most 93 retrieval queries and 200 answer-generation queries per experiment.
Faithfulness scores depend partly on the judge’s calibration, and the results do not establish how the tested profiles perform on multilingual, private or enterprise data.
What the evidence shows
- Test components in context
The paper introduces lasagna, an open-source framework for evaluating retrieval-augmented generation (RAG) components inside complete pipelines. RAG systems retrieve documents and use them to help generate answers.
The framework gives pipeline stages fixed interfaces, so researchers can replace a component and evaluate it within the surrounding system. The authors use this setup to ask whether isolated gains hold when techniques are combined, across different goals and corpus sizes.
- Separate gains may shrink
The authors compared three techniques that separately improved retrieval with a balanced pipeline combining them. Retrieval quality was measured by NDCG@10, a score of how well relevant documents appear near the top of retrieved results.
On the same held-out split across five domains, the techniques’ separate gains predicted a 46.6% relative improvement over the default standard pipeline, while the combination achieved 17.2%. For this combination, isolated gains overstated the combined result.
- Measure both outcomes
Finding useful documents and producing answers grounded in those documents are distinct outcomes. The authors compared retrieval quality and answer faithfulness across 113 configurations evaluated on both tracks.
Faithfulness measures whether generated answer claims are supported by retrieved context; GPT-5.1 assigned the RAGAS scores. The two measures were nearly uncorrelated across these configurations, so a result on one did not reliably indicate the result on the other.
- Corpus size changes rankings
The authors also tested standard and balanced profiles as they increased a constructed corpus from 5,000 to 200,000 documents. Standard led at the two smallest sizes; balanced led from 25,000 documents onward.
At 200,000 documents, balanced’s retrieval score was 13.7% higher than standard’s across three seeded runs. This shows that the small-corpus ranking did not hold across the tested range, though the profiles differ in multiple components.
- No profile wins every measure
A comparison of four complete profiles on three question-answering tests found no profile led on every measure. At a fixed 30-second response-time reference, standard had the highest joint score for retrieval, faithfulness and response time; two more involved profiles had higher measured faithfulness but longer response times.
Together with the distinct retrieval and faithfulness results, this shows why the study does not identify one universally best pipeline.
Deep
- Original paper authors
- Ido Levy, Segev Shlomov, Leshem Choshen
- Submitted to arXiv
- License
- CC BY 4.0