A graph result powers faster structured language decoding
In plain words
The paper combines a math result about paths through graphs with stcon, a system for making language models follow formats such as JSON. The system checks which next pieces of text keep an answer valid, and does those checks on the graphics chip rather than making the main processor do extra work.
On one tested laptop, it decoded six JSON tasks faster than a comparison system. That result covers the tested models, tasks, and setup; it does not show that every model answer is correct.
The research
When an app requires JSON or typed tool arguments, the decoder must check each next token against the format. On a laptop, those CPU checks can make the GPU wait; the question is whether the format can be enforced without paying that repeated delay.
The paper turns a few-path graph result into an MLX decoder that recomputes format masks on the GPU and guarantees valid constrained outputs.
The authors pair a deterministic format-checking machine with a tree of token prefixes. Since each token follows one path through that tree, stcon can recompute allowed-token masks on the GPU instead of storing a large state-by-token table.
In the same MLX loop and with the same model and kernels, stcon decoded six JSON extractions with Qwen3.5-2B 1.28 times faster than llguidance fast-forward. This supports a speed advantage for the tested decoding setup.
The timing and other measurements come from one Apple M2 Pro, one model family, six JSON schemas, four recursive schemas, and greedy decoding. The recursive-format support accepts only certain grammars and caps nesting at 64 levels.