HiLO coordinates service slowdowns to protect product quality
HiLO: Towards Product-Level Graceful Degradation with a Quality-Aware, Load-Aware Control Framework · arXiv:2610.34844
In plain words
When a product’s services slow down independently, one service may protect itself while other parts keep sending it work. The product can lose quality and remain unstable even after the traffic spike ends.
HiLO is a system that coordinates predefined ways for services to reduce work together. In nine reported Bing traffic surges in January 2026, the worst search relevance drop was 0.425 points, below Bing’s stated 0.5-point target. Those results describe Bing events, not every system.
The research
When services in one product depend on one another, can they reduce work together during overload without leaving product quality to emerge from independent local decisions?
A service may stay available while its callers keep sending demand, so the product can still suffer lasting quality loss and instability.
HiLO combines service load signals to select tested product states that coordinate how dependent services reduce work under overload.
HiLO combines participating services’ load signals and selects a predefined product state, assigning coordinated service levels across the product. The states and service dependencies are tested before use, so overload response follows a validated combination rather than independent local decisions.
Across nine user-traffic surges in three Bing regions in January 2026, the worst OneDCG relevance drop was 0.425 points at 2.38 times baseline peak, below the stated 0.5-point target. This supports bounded quality loss in those reported production events.
The evidence comes from Bing production and capacity tests. The capacity comparison used short, gradual synthetic traffic ramps, and the paper does not report evaluations across other companies.
What the evidence shows
- The product-level problem
HiLO (High Load Optimizer) is an automated system that coordinates how services in a product reduce work during overload. The question it addresses is how to protect product quality when services depend on one another.
Independent protections may keep each service available, yet callers can continue sending demand to a bottleneck and trigger wider latency, errors, and prolonged instability.
- Availability did not protect quality
A Bing incident shows why service availability alone can be an incomplete measure. After a deployment increased calls to a shared model-serving service, a short traffic spike pushed a nearly full datacenter over capacity.
The service remained available, but callers continued sending work; relevance fell by 1.11 points and product stability dropped from about 90% to 63%, taking over four hours to recover. This incident illustrates the failure mode, rather than serving as a controlled test of HiLO.
- Coordinating service changes
HiLO combines service load signals and selects a predefined product state, which assigns coordinated service levels across participating services. Dependencies between callers and callees are resolved while states are designed and tested, so runtime selection does not need to discover how services depend on one another.
The approach trades fine-grained control for a small set of product modes intended to make degradation predictable.
- Reported production evidence
In January 2026, HiLO managed nine user-traffic surges that crossed the overload threshold in three Bing regions. Among those events, the worst reported OneDCG relevance drop was 0.425 points at 2.38 times baseline peak, below the stated 0.5-point target.
This production result supports a bounded claim about the reported Bing surges; it is not an average across events or evidence from other companies.
- What the other tests add
Two short capacity tests provide a separate comparison. With gradual synthetic traffic ramps to about twice baseline peak, OneDCG showed no change with HiLO, compared with small drops without it; the tests differed in duration and peak load, and the paper says production events are more representative.
HiLO is also used in Copilot Search with frontend feature changes, but that case study reports no quantified comparison.
Deep
- Original paper authors
- Knut Magne Risvik, Zhuowei Li, Nirupam Purushothama, Hunter Su, Greg Dunham
- Submitted to arXiv
- License
- arXiv non-exclusive