Systems & Infrastructure
Research on Systems & Infrastructure: explanations, evidence, limitations and connections between papers.
13 results
An AI harness produces a faster consensus protocol
AI-greement supports consensus design; its A-greement protocol lowered mean latency in tested failure-free workloads.
Measure the bottleneck before shifting serving work
A serving shortcut can move work before a question, onto another processor, or into a cache. Each move changes a different delay or capacity limit. A stored result may avoid computation yet still take time to retrieve. The useful first step is to name the service target and measure the cost the change shifts under the intended workload. These studies illustrate that choice through streaming video, tokenization for repeated model calls, and remote retrieval of cached model state.
A broad power-flow test finds physics solvers can match learned models
A shared power-flow test finds parallel CPU and GPU physics solvers match learned models on the tested task.
StreamAhead adjusts video processing before questions arrive
StreamAhead sustains 30 video frames per second with question-to-first-token waits below 0.5 seconds at the 95th percentile.
A training system adapts memory to rare expert-load spikes
A system adjusts activation memory during expert-routing spikes and maintains steady computation use in one 512-GPU training run.
Reusable AI session context can strain GPU memory
Production trace replays show how idle caps and recoverable cache copies reduce AI context rebuilding under memory limits.
Measure the destination cost before moving serving work or data
A serving system can run short of GPU memory, CPU time for preparing requests, or bandwidth for retrieving cached model state. Moving data or work addresses each pressure differently and introduces a cost at its destination. The useful first step is to name the constrained resource and choose an outcome that tests it. The studies report generated-token rates, frontend request capacity, and a calculated cache-transfer threshold. Those measures answer different questions; none alone establishes a general serving gain.
Why a shared prompt cache raised tail latency
A remote prompt cache raised tail latency in one serving setup; a sharded store lowered it in a single hit-heavy replay.
Different retry limits help protect important service requests
DiffTry shifts overload rejections toward low-priority requests, with network-model predictions close to a live deployment.
A coding agent can tune training recipes while checks guard correctness
ATTUNE pairs coding-agent recipe search with independent checks and improves measured training speed on two of three public recipes.
Match inference changes to the costly operation
A system can struggle while reading a long document, retaining it for later questions, generating the next token, or recognizing speech on a small chip. Each case puts pressure on a different operation. The studies suggest identifying that operation before choosing a change. Fewer retrieved keys, a smaller retained cache, fewer state writes, and less weight traffic are distinct outcomes. Evidence for one does not establish another.
Ember keeps oversized GPU workloads moving with adaptive placement
Ember adapts GPU data placement for workloads exceeding memory, with faster mean runtimes than default UVM across six tested applications.
A GPU tokenizer preserves IDs while speeding cached requests
GraphTok preserves reference token IDs while shortening median tokenizer calls in tested vLLM conditions with nearly all prompt tokens cached.