A tokenizer reuses chat history to cut first-token delays

When a coding-agent call adds only a short update to a very long conversation, why should the CPU turn the old text into the model’s numbered pieces again? Can serving avoid that repeated work while preserving the pieces the model receives?

Delta Tokenizer is a CPU system that turns text into model-readable pieces, reuses verified pieces from earlier turns, and splits eligible long requests into safe chunks for serving.

The researchers compared Delta Tokenizer, a CPU system that reuses verified numbered text pieces, with the Hugging Face reference tokenizer, the standard software that converts text into those pieces, in SGLang using Qwen3-8B.

They tested rendered chat requests with 100,000-token contexts and 400-character appends. Across 60 warm turns, median delay from scheduled arrival to the first streamed token was 59.61 ms with Delta Tokenizer and 144.80 ms with the reference.

Because these requests reused session history, the result supports the conclusion that avoiding a full CPU re-encode can reduce delay when tokenization holds up serving. Short contexts showed little gain.

The gains depend on the workload: short contexts showed little benefit because model work set the delay, and when every long document was new, GPU processing set the tail. Some tokenizer configurations and processing paths were not tested equally.