A reward model helps rank ads from incomplete logs
Beyond IPS: Reliable Counterfactual Evaluation in Multi-Stage Ad Systems without Logged Propensities · arXiv:2610.40112
In plain words
The paper introduces a way to compare new ad ranking rules using past ad impressions, even when the logs do not record the action probabilities that some evaluation methods need.
It trains one reward predictor to focus more on impressions likely under the candidate rankings, then uses that predictor to estimate their results on logged situations.
In a simulated ad system, this weighted predictor more often picked the best ranking than a predictor trained without those weights. On real traffic, the top candidates were so close that the offline method could not reliably tell which was better.
The research
When an ad ranking is changed, can its value be estimated from past impressions if the logs omit action probabilities and downstream delivery steps shape what people see?
A reward model may be inaccurate on the impressions a candidate ranking would produce, making its offline ranking of candidates unreliable.
The method trains one reward model toward the combined impressions of candidate ad rankings, then replays them to estimate and compare their value.
The authors compare six estimators against oracle values in simulation, where Weighted DM more often selects the best candidate than Unweighted DM across all overlap regimes.
This supports weighting training toward candidate impressions when logged impressions alone can leave the reward model inaccurate where a new ranking acts.
On production traffic, Weighted DM also beats Unweighted DM on the reported averages. Yet the two leading candidates’ order changes across subsamples, so the gain does not establish which candidate is best.
The production results average over 50 data subsamples, and the leading candidates were too close for reliable offline separation. The theoretical routing rule is asymptotic, assumes replay and a shared reward distribution, and was not evaluated under live traffic.
What the evidence shows
- The missing action probabilities
The paper proposes mixture weighted reward modeling, a method for ranking candidate ad policies from logged impressions without logged action probabilities. In multi stage ad delivery, auctions, filtering, targeting, and business rules shape which ads appear, and the logs may not capture those steps.
A reward model can replay candidate actions, but its predictions may be unreliable for impressions that differ from those in the logs.
- Weight toward candidate impressions
The method replays each candidate ranking on logged contexts and uses a domain classifier to estimate how its impressions differ from logged ones. It combines those estimates into weights for training one reward model toward the mixture of candidate impression distributions.
The model then predicts rewards for each candidate’s replayed actions. This addresses distribution shift where logged impressions provide support; it cannot learn directly from candidate impressions absent from the logs.
- Simulation favors weighting
In simulation, the authors compared six estimators against oracle values across low, medium, and high overlap regimes. Weighted DM selected the oracle best more often than Unweighted DM in every regime, with Top-1 accuracy of 0.81 versus 0.67 in the low overlap group.
The DR comparison qualifies the result: at high overlap, Unweighted DR was slightly more accurate than Weighted DR, and their average regret was tied.
- Production leaves a close call
On production traffic, each of five candidate rankings had its own observed click through rate from an online A/B test. Across 50 data subsamples, Weighted DM improved on Unweighted DM across the three reported measures.
But its estimated winner led the runner up by only 0.15% on average, and their order reversed in 24 subsamples. The authors therefore say offline evaluation could not reliably separate these two candidates.
- Coverage bounds the conclusion
The theoretical analysis explains why better predictions on supported impressions do not remove all uncertainty. It bounds replay value error using reward prediction error on overlapping impressions, the candidate’s cold-start mass where logged support is low, and finite sample error.
An asymptotic rule uses the cold-start contribution to decide whether to trust offline selection or route candidates to online testing. Its assumptions and lack of live evaluation limit that recommendation.
Deep
- Original paper authors
- Mohsen Malmir, Mohamed A. Radwan, Houssam Nassif, Murat Ali Bayir
- Submitted to arXiv
- License
- CC BY 4.0