Byte models miss results that need calculation

When a byte-level model has a limited number of patch starts, should it place them where the next byte is hard to predict, or where the model needs more context to calculate a value? The paper tests whether uncertainty about the next byte is a reliable guide to where a fresh global state helps.

The paper proposes boundary dependence, a label-free signal that places byte-model patch starts where a fresh global state can help prediction.

On GSM8K, an arithmetic word-problem test, the author compared BLT-1B patch layouts at about a 10% budget. The score was exact prediction of 796 numbers written after equals signs, with each prediction given its true preceding text.

Entropy placement covered only 16% of these results and got 19.0% exactly right; the equals-sign layout reached 51.8% at a similar patch rate.

Adding boundary dependence to entropy reached 67.1%, above the hand-written rule. This supports the answer that next-byte uncertainty alone misses places where a fresh global state can help predict a computed value, while a signal based on prediction loss can find them without result labels.

These comparisons measure prediction with the true preceding text, not whole-problem solving: adapted BLT-1B got about 2% of GSM8K final answers right end to end under every tested layout. The dependence table was also fitted on the arithmetic domain and did not transfer across formats.