Systems & Infrastructure
Research on Systems & Infrastructure: explanations, evidence, limitations and connections between papers.
6 results
Measure transfer and capacity costs before moving GPU serving work or data
A model can exceed GPU memory, a long request can occupy the text-processing frontend, and a cached prompt can take time to fetch. Moving data or work may relieve each pressure, but the new location has its own cost. Across the three studies, the useful question is which measured constraint a placement change relieves. The answer depends on the tested workload: host-memory access can slow generation, GPU text preparation can raise frontend capacity, and remote cache loading can cost more time than recomputing.
Why a shared prompt cache raised tail latency
A remote prompt cache raised tail latency in one serving setup; a sharded store lowered it in a single hit-heavy replay.
Different retry limits help protect important service requests
DiffTry shifts overload rejections toward low-priority requests, with network-model predictions close to a live deployment.
A coding agent can tune training recipes while checks guard correctness
ATTUNE pairs coding-agent recipe search with independent checks and improves measured training speed on two of three public recipes.
Ember keeps oversized GPU workloads moving with adaptive placement
Ember adapts GPU data placement for workloads exceeding memory, with faster mean runtimes than default UVM across six tested applications.
A GPU tokenizer preserves IDs while speeding cached requests
GraphTok preserves reference token IDs while shortening median tokenizer calls in tested vLLM conditions with nearly all prompt tokens cached.