In plain words
Phone agents ask a model what to do after each screen. Their requests repeat some instructions but change the recent actions or screen details, so saved model state can miss the useful point to resume.
Tandem is software that manages this reuse, speeds up checking proposed answer tokens, and lets the phone’s neural processor rest between tasks.
On a 61-request text replay trace on a Snapdragon 8 Elite phone, Tandem’s full setup reduced average model-request time per step by 30% versus llama.cpp’s fastest neural-processor setup. That is an average for this trace, not a guarantee for every request or phone.
The research
When a phone agent asks for an action after each screen, how can it reuse what the model has already processed when recent actions and screen details change? The same runtime also needs to handle short outputs efficiently and avoid keeping the phone’s neural processor fully active between tasks.
Tandem places reusable model state at changing prompt boundaries and reshapes speculative execution for hybrid agents on mobile NPUs.
Tandem saves recurrent state where consecutive prompts diverge, then changes how speculative tokens are prepared and checked. The authors compared its full runtime with llama. cpp’s fastest NPU configuration using the same model and device.
Mean text-step latency fell 30% on a 61-request replay trace, while screenshot-step latency fell 16% on a 21-screen app set. The comparison supports a latency improvement for those workloads with Tandem’s combined serving changes.
It does not isolate each change’s contribution: the full runtime includes several optimizations, and the reported step time excludes app execution.
The end-to-end latency results are averages from specified workloads on one Snapdragon 8 Elite, and step latency excludes app execution. The authors also evaluated checkpoint placement on LFM2 and Granite and measured latency gains on a second Snapdragon generation.
These results do not establish that the complete speculative path, which depends on particular model features and accelerator kernels, transfers unchanged to other designs.
Analysis
Deep
- Original paper authors
- Zhuomin Chen, Jingchao Ni, Xu Zheng, Yong Chen, Mo Sha
- Submitted to arXiv
- License
- arXiv non-exclusive