Separate GPU paths speed prompt and token processing

How can a transformer process a long prompt and then generate one token at a time efficiently when the two phases spend time reading and reusing data in different ways?

The answer matters to the design of inference kernels, but speed comparisons alone cannot show what low-bit generation costs in model quality.

FuTransformer gives prompt processing and one-token generation separate Hopper GPU paths, using fused FP8 operations for prompts and packed INT4 data for generation.

The paper asks how to speed up transformer inference when processing a prompt and generating one token at a time are limited by different data movement. FuTransformer is a Hopper GPU implementation that gives each phase a separate path, combining prefill operations and packing decode weights and cache values in layouts suited to the computation.

On one H100, the tested batch-one prefill path ranged from 0.92x to 1.48x the speed of compiled PyTorch across configurations, with different activation-scaling rules. For decode, the INT4 block beat the faster of two PyTorch blocks at each tested point, though precision and cache formats differed. On Llama-3.1-8B, INT4 also raised WikiText-2 perplexity, showing a measurable quality cost.

These results come from one H100 and bounded layer and model tests. They do not establish complete serving performance, performance on other GPU generations, or quality across other models and tasks.