Client retries can keep an overloaded language model stuck
Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine · arXiv:2611.08684
In plain words
When a client waits too long for a language model, it may send the same call again. The engine might still be working on the first call, or cancellation might stop it, so the retry can add more work or repeat work already done.
Zhang and Wang traced retries inside vLLM and compared ways to handle them. After a heavy burst of calls, cancellation alone did not bring service back while clients kept retrying.
Limiting retries or letting a retry join work already underway restored service in their tests, but only joining work delivered every burst call by the study’s deadline.
The research
A timeout does not necessarily stop the engine’s work, and a retry can add more work while the service is already overloaded.
The result changes the expectation that cancelling timed-out calls alone will clear the queue: in these tests, recovery depended on controlling retries as well as handling the old execution.
Zhang and Wang trace retries inside vLLM and compare overload controls, finding that cancellation alone does not restore service while retries continue.
The researchers replayed matched workload traces across retry and cancellation settings, then linked client attempts to vLLM scheduler executions.
After the overload pulse, retries without cancellation prevented recovery, and propagating cancellation still failed to restore service on all three models.
The traces show aborted executions had consumed much of the computed prompt work, while scheduled retries began near their deadlines. This timing supports the authors’ account of retries refilling the queue, though it does not prove a single cause.
The conclusions come from synthetic, single-turn calls sent by an open-loop client to vLLM 0.30.0 on one GPU type. The study used one retry policy and fixed the base arrival rate and overload pulse length.
What the evidence shows
- What a timeout leaves behind
Zhang and Wang’s empirical study follows client attempts into executions in vLLM, an engine for serving language models. A timed-out attempt may leave work running without a waiting client, be cancelled, or remain live for its retry to join.
The question is whether stopping the old execution is enough to restore service when retries continue creating work.
- Cancellation did not restore service
Across the same diagnostic workload within each model, retries off allowed all three models to recover during the hold, while adding cancellation did not restore service when retries continued.
Cancellation improved delivery compared with retries without cancellation, but the comparison shows that stopping old executions alone did not stop the tested overload.
- Where the work went
The traces connect two parts of the overload: executions later aborted had already accounted for most computed prompt tokens, and scheduled retries began work just before their own deadlines.
The authors interpret this timing as consistent with retries refilling the queue as earlier work is aborted, though it does not prove that this is the only cause.
- Recovery is not full delivery
A retry budget can shed new attempts; coalescing can attach a retry to its still-live execution. With cancellation off, both controls restored service across the three models, but their delivery differed: the budget delivered about as many pulse calls as no control, while coalescing delivered every pulse call by 840 seconds, with some late.
The contrast shows that meeting a recovery rule does not mean every call from the overload was delivered.
- A tradeoff in a separate test
In a separate Qwen2.5-7B-Instruct comparison, the retry budget delivered more primary calls on time, while deferred cancellation with coalescing delivered every pulse call by the 840-second horizon.
Because the arms differed in both retry handling and cancellation timing, this describes a tradeoff between timely primary delivery and complete pulse-call delivery; it does not isolate either change.
Deep
- Original paper authors
- Xindan Zhang, Fei Wang
- Submitted to arXiv
- License
- Not specified by source