Step-aware serving speeds up generative recommendation
RecoGEM: A Serving System for Production-Scale Generative Recommendation · arXiv:2611.12296
In plain words
Generative recommendation creates an item's identifier in several dependent steps instead of scoring a ready-made list. RecoGEM is a serving system built for that pattern: one part manages the steps and candidate items, while another runs the computations.
On two production models, it processed requests faster than a comparison system at batch size 32. The size of the gain differed between the models, so the result is specific to the tested systems and setting.
The research
Serving generated recommendations means coordinating several dependent item-generation steps while meeting latency demands. A uniform decoding loop may not fit that work well.
RecoGEM's measured advantage shows that serving design can affect request throughput on the two evaluated production models, though the size of the gain depends on which model is served.
RecoGEM is a production serving system that coordinates recommendation-specific steps and tunes computation to their shapes, improving throughput in two tested models.
At request batch size 32, the system was compared with GeneralGen, which uses one shared shape profile and general GPU kernels. RecoGEM processed requests 2.88 times as fast on OneModel and 1.84 times as fast on OneRec-v2.
The different gains show that the advantage depends on the model. A separate OneModel comparison, which retained specialized kernels, also found lower average decode latency with step-specific profiles than with one shared profile.
The performance comparisons cover two production models and specific serving configurations at a request batch size of 32. Distributed serving across GPUs is future work, so these results do not show how RecoGEM performs when serving state is distributed.
What the evidence shows
- Generation changes serving
RecoGEM is a production serving system for generative recommendation, where a model generates item identifiers through several dependent steps rather than scoring supplied items. Each step can have different candidate shapes, and selecting which candidates continue is part of the latency-critical work.
The system is designed around those demands, which differ from the uniform decoding loops used by conventional language-model serving.
- A system built for steps
RecoGEM separates step coordination from computation: a dedicated runtime tracks candidates and manages updates, while an execution engine runs tensor operations. The engine groups independent short attention rows, plans for the shapes of individual steps, and selects candidates in two stages.
This design keeps the recommendation-specific work visible to the system instead of treating the whole request as one uniform decoding loop.
- Gains vary by model
The batch-32 comparisons show RecoGEM's throughput advantage over GeneralGen varied by model: 2.88 times on OneModel and 1.84 times on OneRec-v2. This supports an advantage in the two tested production models, not a fixed gain across models.
- Step profiles matter too
A separate OneModel test kept RecoGEM's specialized kernels but replaced step-specific profiles with one profile shared across all generation steps. Average decode latency across steps was 30.4 ms with RecoGEM and 39.5 ms with the shared profile.
This comparison indicates that step-specific planning contributes to the measured latency difference in this setting, while leaving the other parts of the full-system comparison distinct.
Deep
- Original paper authors
- Xiangrui Yang, Yunfeng Zhao, Jiawei Yang, Fangyu Zhang, Enrui Zhao
- Submitted to arXiv
- License
- arXiv non-exclusive