Parallel compilation speeds finite-action language model control

When a language model must choose among a limited set of actions, can it keep later choices dependent on earlier ones without paying for a separate model pass for every generated token?

The paper asks whether the controller’s decisions can be computed in parallel while preserving the same history-dependent rule.

The method evaluates reachable action prefixes in parallel, preserving a finite-action controller’s choices while replacing token-by-token decoding with two model passes.

The authors compile a constrained controller that selects document sections into two model passes: one resolves the first choice, and one evaluates reachable later choices together while preserving each choice’s history.

This keeps the controller’s sequential decision rule without generating its constrained text token by token.

On QASPER, a dataset of scientific papers used for question answering through hierarchical section retrieval, compiled control matched the serial controller’s complete traversal on 84–89% of questions per split across four splits with a 7B model.

It reduced mean per-question latency by 35–41%. The measured differences occurred at near-tie decisions under finite precision, while the equivalence proof assumes exact arithmetic.

The exact-equivalence result assumes exact arithmetic and matching masks, candidate order, and tie-breaking.

In the tested finite-precision runs, near-tie decisions sometimes differed; the evaluated retrieval controller used horizons of up to three actions, and the parallel workload grows sharply at longer horizons.