Search retrieval can request useful features as it generates results
In plain words
Search and recommendation systems can find items by having a model generate codes that identify them. They also have clues such as user history or item freshness, but adding every clue in advance can be costly or unhelpful.
Agentic Generative Retrieval lets the model request a clue while generating an item code, choosing whether and when it may help. On held-out Industry search requests, it found more known relevant items in its top 32 results than a standard generative retrieval method.
The research
Search systems can use many clues about a request and its possible results, but adding them all in advance can cost more and introduce irrelevant information. Can a retrieval model decide which clue helps a particular request, and when that clue should influence the item it is generating?
Agentic Generative Retrieval lets a model call feature tools during item-ID generation, choosing whether, which and when to use them from the request and current ID prefix.
In offline retrieval tests, researchers compared AGR with TIGER, a standard generative retrieval method. On held-out Industry requests, AGR's Recall@32 was 14.04% higher relative to TIGER. Recall@32 averages the fraction of known relevant items found among the top 32 results across requests.
The comparison supports AGR's retrieval performance in this setting. The methods shared a text-pretrained backbone, SID codebooks, training data, learning rate and hardware configuration, but the paper does not establish a shared starting checkpoint.
The offline comparison is for Industry test requests, and the paper does not establish that AGR and TIGER started from the same checkpoint. The production result measures one relevance measure in one search system; neither result establishes effects across all systems or user outcomes.
Analysis
Deep
- Original paper authors
- Ming Yang, Yaqi Yin, Zhibo Wang, Xiaoze Jiang, Jingwei Zhuo
- Submitted to arXiv
- License
- arXiv non-exclusive