A GPU tokenizer preserves IDs while speeding cached requests
In plain words
When a language model sees a long conversation again, its server can reuse stored work for the repeated part. But it still has to turn the whole request into token IDs before finding that stored work, so tokenization can become a bottleneck.
GraphTok moves supported tokenizers onto a GPU while preserving their reference token IDs. In one vLLM test, when nearly all of each prompt was cached, its median tokenizer call was 2.5–5.8 times shorter than TokTier’s across the tested models and lengths. The paper tests five tokenizer configurations, not every tokenizer.
The research
When a model server reuses stored work for a long repeated prompt, tokenization still has to produce the IDs needed to find that work. Can this remaining step be made faster while preserving the exact IDs the model receives?
GraphTok runs supported tokenizers on a GPU while preserving reference token IDs, aiming to reduce encoding time when prefix caching exposes that work.
The question is whether a serving system can reuse stored work for repeated text without leaving tokenization as a major part of the request. The system still needs token IDs before it can look up cached model state, and tokenizing only the new suffix is not generally safe.
GraphTok moves supported encoding steps onto a GPU while preserving reference token IDs. In vLLM, with 99.85–99.99% of prompt tokens cached, its median tokenizer call was 2.5–5.8 times shorter than TokTier’s across the tested models and lengths.
This supports faster tokenization under those conditions, alongside matching IDs in the evaluated correctness campaign.
The results cover five concrete tokenizer configurations, and every experiment used one H100 NVL. Dynamo’s GPU workers were simulated, so that test measures frontend capacity rather than end-to-end model serving.
Analysis
Deep
- License
- arXiv non-exclusive