In plain words
Preference training teaches a language model which of two responses should rank higher. But a model can get better at that comparison while its own writing becomes repetitive or collapses onto one token.
ECHO studies this mismatch by measuring a shared shift inside the model and testing whether it can drive a token to dominate the output.
In the reported driving-preference runs, 14 of 17 tested model lineages met the researchers’ collapse criterion within six times the base training budget; some were tested only to three times that budget.
The research
How can preference training make a model better at ranking responses while its generated text breaks down? The authors ask what internal change could explain that mismatch, and why a ranking measure might stay healthy even as the model’s output becomes repetitive or collapses onto one token.
ECHO traces a shared shift in final hidden states to explain how preference ranking can improve while generated text collapses.
The authors measure a common-mode offset, a shared shift in a model’s final hidden states, and examine how it changes chosen and rejected response scores.
In four collapsing runs, adding the offset lowered both scores far more than it changed their difference, showing why a relative ranking measure could stay comparatively healthy.
Across per-token ranking runs on driving-preference lists, 14 of 17 tested base-model lineages met the collapse criterion within six times the base budget.
Component and transplantation tests linked the offset to the vocabulary well, supporting the authors’ account of how generation can fail while ranking remains strong.
The results come from particular preference-training runs and evaluations. Penalty strengths were calibrated on driving data, the six-fold-budget result covers only the OLMo family, and seed repeats were limited to flagship settings.
Analysis
Deep
- Original paper authors
- Yun Li, Ehsan Javanmardi, Simon Thompson, Jiachen Gong, Qunli Zhang
- Submitted to arXiv
- License
- CC BY 4.0