Systems & Infrastructure
Research on Systems & Infrastructure: explanations, evidence, limitations and connections between papers.
7 results
Measure transfer and capacity costs before moving GPU serving work or data
A model can exceed GPU memory, a long request can occupy the text-processing frontend, and a cached prompt can take time to fetch. Moving data or work may relieve each pressure, but the new location has its own cost. Across the three studies, the useful question is which measured constraint a placement change relieves. The answer depends on the tested workload: host-memory access can slow generation, GPU text preparation can raise frontend capacity, and remote cache loading can cost more time than recomputing.
Why a shared prompt cache raised tail latency
A remote prompt cache raised tail latency in one serving setup; a sharded store lowered it in a single hit-heavy replay.
Different retry limits help protect important service requests
DiffTry shifts overload rejections toward low-priority requests, with network-model predictions close to a live deployment.
A coding agent can tune training recipes while checks guard correctness
ATTUNE pairs coding-agent recipe search with independent checks and improves measured training speed on two of three public recipes.
Match inference changes to the operation under pressure
A system handling a long document may spend work finding text to attend to, or run out of room to retain useful document entries. A decoder can also spend time repeatedly moving its state, while a small speech device struggles to fetch model weights. The useful first step is to locate that pressured operation. The studies change different operations: attention retrieval, stored cache selection, recurrent-state writes, and weight traffic. Their evidence gives reasons to consider each change under its own tested conditions, not a common speed or memory saving.
Ember keeps oversized GPU workloads moving with adaptive placement
Ember adapts GPU data placement for workloads exceeding memory, with faster mean runtimes than default UVM across six tested applications.
A GPU tokenizer preserves IDs while speeding cached requests
GraphTok preserves reference token IDs while shortening median tokenizer calls in tested vLLM conditions with nearly all prompt tokens cached.