Long-context midtraining raises selected coding scores

How can a coding model encounter whole repositories and tool interactions before post-training, when initial training often uses shorter contexts and later supervised training is largely synthetic?

The paper tests whether a separate, long-context stage can help prepare base models for agentic coding without relying on base-model proxy measures to predict later performance.

The recipe trains base models on long software-engineering data and some agent traces before post-training, then tests coding results after the same SFT.

The authors train base models for hundreds of billions of tokens at long context, mostly on natural software-engineering data with a smaller share of agent traces. They compare midtrained models with their starting checkpoints after the same full SFT, measuring success on coding tasks.

For Lightning-LC, 500 billion midtraining tokens raise pass@1, the mean success rate across sampled rollouts for each task, on SWE-bench Verified from 60.7% to 64.5%; for Super, 280 billion tokens raise it on DeepSWE from 11.7% to 15.7%.

These results show selected post-SFT gains, while Super’s Multilingual and Pro scores do not show consistent improvement.

The benchmark comparisons are single runs, and the authors note that differences of one or two points may reflect run-to-run variation.

The recipe was tested on Nemotron models, mainly for agentic coding, with an agent-focused SFT blend; the short, single-seed reinforcement-learning check does not show that midtraining increases what reinforcement learning can add.