Measure transfer and capacity costs before moving GPU serving work or data

A model can exceed GPU memory, a long request can occupy the text-processing frontend, and a cached prompt can take time to fetch. Moving data or work may relieve each pressure, but the new location has its own cost.

Across the three studies, the useful question is which measured constraint a placement change relieves. The answer depends on the tested workload: host-memory access can slow generation, GPU text preparation can raise frontend capacity, and remote cache loading can cost more time than recomputing.

Imagine serving a model whose data exceeds the GPU's memory. Keeping some data in host memory lets the model run, but the GPU must reach across an interconnect to read it.

Moving data into GPU memory speeds access to that data while using space another piece of the model may soon need. Ember, a memory manager for workloads that exceed GPU memory, studies this choice during model generation. Its problem is where to place data as the GPU repeatedly uses it.

A different request can strain the system before model computation begins. An agent may resend a long conversation while adding little new text.

A cache can spare much of the model's work on the repeated beginning, yet the serving frontend still has to turn the incoming text into token IDs, the ordered identifiers the model reads. GraphTok studies moving that preparation from the CPU to a GPU.

It spends GPU capacity to relieve a frontend constraint rather than to make room for stored model data.

There is a third option for repeated prompts: store the results of earlier prompt processing and fetch them for reuse. That stored state is called a KV cache.

A remote copy saves GPU computation only if fetching it serves the relevant goal at an acceptable cost. The remote-cache study asks how fast the state must arrive and what work the engine still performs for a hit.

Together, the cases raise one deployment question: what cost does each move remove, and what cost does it introduce?