Changing model settings can reduce prediction coverage

If calibration and deployment use different model settings, does a strong coverage result from matched testing still describe how often prediction sets include the right answer in use?

That matters because quantization or temperature can change the scores used to build those sets, even when the task and trained model stay the same.

The paper formalizes inference-time model shift and tests how changing quantization or temperature between calibration and use affects conformal coverage.

The authors tested whether matched calibration describes coverage after deployment settings change. On MMLU-Pro, a multiple-choice question-answering test, they measured the fraction of evaluation examples whose true answer appeared in the prediction set.

With deployment fixed to nf4 at temperature 0.5, average coverage across eight models was 0.902 when settings matched, but 0.639 across mismatched calibration settings.

Because the task and trained model stayed fixed, the comparison links the drop to a changed inference score-generating procedure in this setting. Further checks averaging over deployment quantization modes or temperatures also showed mismatch effects, though their size varied. This supports testing configuration sensitivity alongside matched coverage.

The audit covers eight small language models and four finite-label tasks on a finite grid of quantization modes and temperatures. It does not predict which particular configuration change will cause undercoverage, overcoverage, or only a small effect.