Systems & Infrastructure
Research on Systems & Infrastructure: explanations, evidence, limitations and connections between papers.
12 results
Measure the bottleneck before moving GPU work
A model can exceed GPU memory, a video stream can outrun its processor, and repeated model calls can spend time preparing text or fetching saved state. Each problem invites a different use of memory and computation, with its own movement or preparation cost. Across four studies, the useful first step is to measure where a particular workload spends time. Placing data, preparing work early, moving text processing to the GPU, and reusing remote state change different costs; saving computation does not by itself reduce stored memory or guarantee a faster response.
A broad power-flow test finds physics solvers can match learned models
A shared power-flow test finds parallel CPU and GPU physics solvers match learned models on the tested task.
StreamAhead adjusts video processing before questions arrive
StreamAhead sustains 30 video frames per second with question-to-first-token waits below 0.5 seconds at the 95th percentile.
A training system adapts memory to rare expert-load spikes
A system adjusts activation memory during expert-routing spikes and maintains steady computation use in one 512-GPU training run.
Reusable AI session context can strain GPU memory
Production trace replays show how idle caps and recoverable cache copies reduce AI context rebuilding under memory limits.
Serving performance depends on the resource under pressure
A serving system can slow down because model data exceeds GPU memory, long requests occupy the text-processing frontend, or fetching a cached prompt state takes too long. Moving work or data relieves one pressure while placing a cost somewhere else. Across these studies, the useful first step is to name the constrained resource and measure the cost at the destination. Their evidence concerns generated-token rates, offered frontend request load, and a calculated cache bandwidth threshold. Each answers a different part of the placement decision.
Why a shared prompt cache raised tail latency
A remote prompt cache raised tail latency in one serving setup; a sharded store lowered it in a single hit-heavy replay.
Different retry limits help protect important service requests
DiffTry shifts overload rejections toward low-priority requests, with network-model predictions close to a live deployment.
A coding agent can tune training recipes while checks guard correctness
ATTUNE pairs coding-agent recipe search with independent checks and improves measured training speed on two of three public recipes.
Inference bottlenecks shift across attention, cache, state, and weight traffic
A system handling a long document may spend work finding text to attend to, or run out of room to retain useful document entries. A decoder can also spend time repeatedly moving its state, while a small speech device struggles to fetch model weights. The useful first step is to locate that pressured operation. The studies change different operations: attention retrieval, stored cache selection, recurrent-state writes, and weight traffic. Their evidence gives reasons to consider each change under its own tested conditions, not a common speed or memory saving.
Ember keeps oversized GPU workloads moving with adaptive placement
Ember adapts GPU data placement for workloads exceeding memory, with faster mean runtimes than default UVM across six tested applications.
A GPU tokenizer preserves IDs while speeding cached requests
GraphTok preserves reference token IDs while shortening median tokenizer calls in tested vLLM conditions with nearly all prompt tokens cached.