A browser runtime adapts language model decoding to each GPU
GPU-Resident Speculative Decoding for In-Browser LLM Inference with WebGPU · arXiv:2610.39761
In plain words
The paper introduces a WebGPU runtime, a system that runs language models inside browser pages. It tries to reduce the waiting between a browser and its graphics chip by having the chip check several possible next words in a row and adjust how many it checks based on its measured speed.
Across the tested prompts and GPUs, its speculative answers matched its own ordinary greedy answers exactly. This result does not mean they match other runtimes or the original full-precision models.
The research
Can a browser run language models faster when dispatch and CPU-to-GPU waiting add costs, and when those costs vary across client GPUs? A further question is whether checking several proposed tokens at once can preserve the runtime’s ordinary greedy output while reducing that waiting.
The paper builds a WebGPU runtime that adapts speculative decoding to device costs, reducing browser overhead while matching its own greedy output.
The authors ask whether browser decoding can be sped up without changing a runtime’s greedy output, despite device-dependent costs. Their WebGPU runtime keeps token checking on the GPU and adjusts how many proposed tokens to check.
On the workstation, it decodes 4.8 to 8.6 times faster than WebLLM across seven models, measured as total decoding time divided by generated tokens on prompts WebLLM completes. Its speculative output also matches its own greedy output token for token across the tested prompts and GPUs.
The experiments target greedy decoding and cover Chrome on the tested NVIDIA and AMD GPUs, not Safari, Firefox, Apple GPUs or mobile GPUs.
Because the study includes neither a native-runtime comparison nor a version of the proposed kernels with floating-point activations, it cannot separate the effects of 8-bit activations from those of small-batch design.
What the evidence shows
- The browser decoding problem
The paper introduces a WebGPU runtime, a system for running language models inside browser pages.
Its central question is how to speed up generation when browser dispatch and CPU-to-GPU communication add costs, and when checking several proposed tokens at once can make those costs worse on a client GPU.
- Keeping decisions on the GPU
The runtime moves token drafting and checking into a GPU-resident loop, reducing the need for a CPU decision after each token. Its fixed-order integer calculations let it check multiple proposed tokens while preserving exact agreement with single-token greedy steps in the same quantized runtime. A controller adjusts the number checked using device costs and recent acceptance.
- Why fixed speculation can lose
A fixed eight-node tree tests whether checking many proposed tokens pays off on different hardware. On the laptop’s integrated GPU, it is slower than decoding without speculation in all three tested model configurations; for Llama-3.2-1B it takes 2.7 times as long.
The controller avoids most of these regressions, though it is still slower than plain decoding for two larger models on that GPU.
- Workstation speed and agreement
On the workstation, the proposed runtime’s decoding latency per output token is 4.8 to 8.6 times lower than WebLLM’s across seven models, using prompts WebLLM completes. The measure divides total decoding time by generated tokens.
Across the tested prompts and GPUs, speculative output also matches the same runtime’s plain greedy output token for token.
- Average math accuracy
GSM8K tests math word problems; here, an answer counts as correct when its last number matches the reference. Across seven models and 200 problems per model on the workstation, the proposed runtime and WebLLM each solve 52.3% on average.
Individual model results differ by up to six percentage points in either direction, so the average does not establish parity for every model.
Deep
- Original paper authors
- Wei Song, Yuxin Cao, Ziqi Ding, Xiao Cheng
- Submitted to arXiv
- License
- CC BY 4.0