One model handles three kinds of shopping search
PailitaoMMSearch2.0 Technical Report: Scaling Generative Retrieval to Unified End-to-End Multimodal Search · arXiv:2610.36958
In plain words
A shopping search system may need to find something similar to a picture, change some of its features, or find a product that goes with it. This paper introduces one model that reads the picture and a written request, works out which kind of search is needed, and generates product identifiers.
In offline tests across those three kinds of search, training that used beam search to generate product candidates improved the share of top-10 results receiving the evaluator’s highest score by 5.97 percentage points over supervised training. The result applies to this test, not every shopping search.
The research
An image and a short instruction can describe different shopping goals: finding a similar item, changing an item’s attributes, or finding something that matches it.
Can one retrieval system infer which relation a shopper means and search across all three without an explicit intent label or separate route?
PailitaoMMSearch2.0 trains one retrieval model across three shopping relations and generates product IDs directly from an image and instruction.
For the offline comparison, researchers trained Sample–Beam reinforcement learning and supervised training from the same initialization, with the same intent mixture and reward evaluation.
A human-calibrated evaluator scored each query’s top 10 products, and researchers macro-averaged Excellent Rate across similar, editing, and matching retrieval.
Sample–Beam raised that rate by 5.97 percentage points over supervised training. Because the comparison matched those conditions, it supports a gain from the reinforcement-learning training procedure in this evaluation.
The offline evidence covers three shopping relations in one e-commerce domain, and its automatic evaluator may share errors with other evaluators used in the system. The online test measures the full deployed configuration, so it cannot show which component caused the business changes.
What the evidence shows
- One query, several relations
PailitaoMMSearch2.0 is a multimodal retrieval system for shopping queries that combine an image with a free-form instruction. A request can seek a similar product, an edited version, or a matching item.
The system generates product identifiers from the query and infers which relation is intended, without an explicit intent label or a separate serving route for each relation.
- How the system is trained
The system trains one model across the similar, editing, and matching relations. Continued pre-training adapts it to e-commerce products and product identifiers; supervised training then combines the relations and reasoning budgets; unified reinforcement learning optimizes them together.
At serving time, the model receives the image and instruction and generates identifiers for candidate products.
- What the offline score means
The offline test measures the quality of the system’s top 10 results for each query. A human-calibrated evaluator scores results from 0 to 3; Excellent Rate is the proportion receiving a 3.
Researchers calculate the rate separately for each relation and macro-average across the three, so the reported overall result gives each relation equal weight.
- Two training comparisons
With initialization, intent mixture, and reward evaluation matched, Sample–Beam reinforcement learning raised macro Excellent Rate by 5.97 percentage points over supervised training. Replacing standard reinforcement-learning rollouts with Sample–Beam rollouts added 2.30 points.
This comparison supports a further gain associated with the changed rollout procedure, though its incremental effect varied across relations.
- Evidence from production
In an online A/B test in Taobao Pailitao’s Multimodal Search application, the deployed system increased gross merchandise value by 8.94% over the production baseline.
The result comes from the complete deployed configuration, so it shows a system-level change in that application rather than identifying which training component produced it.
Deep
- Original paper authors
- Xiaohan Ye, Zihan Gong, Jingqian Zhao, Lianyu Du, Yunmeng Shu
- Submitted to arXiv
- License
- CC BY 4.0