Systems & Infrastructure
Research on Systems & Infrastructure: explanations, evidence, limitations and connections between papers.
7 results
Match each serving move to the constraint it relieves
A serving system can slow down because model data exceeds GPU memory, long requests occupy the text-processing frontend, or fetching a cached prompt state takes too long. Moving work or data relieves one pressure while placing a cost somewhere else. Across these studies, the useful first step is to name the constrained resource and measure the cost at the destination. Their evidence concerns generated-token rates, offered frontend request load, and a calculated cache bandwidth threshold. Each answers a different part of the placement decision.
Why a shared prompt cache raised tail latency
A remote prompt cache raised tail latency in one serving setup; a sharded store lowered it in a single hit-heavy replay.
Different retry limits help protect important service requests
DiffTry shifts overload rejections toward low-priority requests, with network-model predictions close to a live deployment.
A coding agent can tune training recipes while checks guard correctness
ATTUNE pairs coding-agent recipe search with independent checks and improves measured training speed on two of three public recipes.
Match inference changes to the operation under pressure
A system handling a long document may spend work finding text to attend to, or run out of room to retain useful document entries. A decoder can also spend time repeatedly moving its state, while a small speech device struggles to fetch model weights. The useful first step is to locate that pressured operation. The studies change different operations: attention retrieval, stored cache selection, recurrent-state writes, and weight traffic. Their evidence gives reasons to consider each change under its own tested conditions, not a common speed or memory saving.
Ember keeps oversized GPU workloads moving with adaptive placement
Ember adapts GPU data placement for workloads exceeding memory, with faster mean runtimes than default UVM across six tested applications.
A GPU tokenizer preserves IDs while speeding cached requests
GraphTok preserves reference token IDs while shortening median tokenizer calls in tested vLLM conditions with nearly all prompt tokens cached.