Randomness sets a floor on predicting language model response length
On the Fundamental Limits of Prefill-Based Decoding Length Prediction · arXiv:2610.40541
In plain words
A language model can give answers of different lengths to the same prompt. This paper develops a theory for how much that randomness limits predictions made before the model starts answering.
The authors compare the theory with repeated answers to prompts and with several predictor designs. In one tested model and dataset setting, different designs had similar average length errors, consistent with randomness limiting how much those predictors could improve.
The research
Published length-prediction errors can change substantially when the model, dataset or sampling setup changes. How can we tell whether one predictor is actually better when the responses themselves vary in length?
The paper addresses this by estimating the error that generation variability alone can impose on predictions made before generation.
The paper models response length as a random stopping process and derives an error floor for predictors using prompt-processing information alone.
The authors ask how much error a predictor can remove when it must estimate response length before a model generates anything. They model generation as a process that stops at a configured marker, with responses made of segments and a random segment count.
Under those assumptions, length is approximately normal, so the lowest possible average absolute error is about 0.798 times its standard deviation for that prompt.
Across 300 ShareGPT prompts, each answered 200 times by Qwen3-32B with its recommended sampling settings, the median distance of responses from each prompt’s average length, divided by their length spread, was 0.783, near the model’s 0.798 prediction.
In a separate fixed-setting test, two predictor designs also had similar errors. The authors interpret this evidence as showing that response variability constrained pre-generation prediction in these evaluations.
The empirical checks focus on one main model and dataset setting, and the authors did not control possible numerical differences between separate workers that produced the prompt features and response-length labels. The conclusions therefore describe the reported settings, not every model or sampling setup.
What the evidence shows
- Separate skill from variability
The paper presents a theory and empirical evaluation of how well a system can predict a language model’s response length using information available while processing the prompt, before generation starts.
The question is whether large or changing prediction errors reflect the predictor itself or the fact that the same prompt can produce responses of different lengths.
- Derive a prediction floor
To analyze that distinction, the authors model response length as the total length of a random number of segments, such as sentences or reasoning steps. Under assumptions about those segments, they derive an approximately normal length distribution for a given prompt.
The theory implies a minimum average absolute prediction error of about 0.798 times that prompt’s response-length standard deviation.
- Check repeated responses
The authors checked the distributional assumption by generating 200 responses for each of 300 ShareGPT prompts with Qwen3-32B. Across prompts, the median mean absolute deviation divided by the prompt’s standard deviation was 0.783, close to the normal model’s prediction of about 0.798. This supports using the model to estimate the relevant error scale in that evaluation.
- Compare designs and limits
In the main ShareGPT test, the simple MLP-Concat predictor’s mean absolute error was 169.3 tokens, close to MoE-Soft’s 170.7 tokens. Theoretical limits also put the maximum fraction of prompts predictable within 100 tokens at 57.9%, averaged across prompts; the best measured predictor reached 48.9%.
The authors read these results as evidence that response variability constrained performance more than added predictor complexity in this setting.
Deep
- Original paper authors
- Haowen Shi, Xuzhe Yan, Fahao Chen, Peng Li
- Submitted to arXiv
- License
- CC BY-SA 4.0