In plain words
AutoCompress is a method for shrinking language models to fit a particular device and workload. It estimates how candidate model shapes would use memory and time in the target serving setup, searches among those shapes, then trains the chosen smaller model to match its larger parent.
In three tested settings, its models scored higher on average across eight tests than a deployment-adapted pruning method. The size of the difference varied by setting, and the average does not mean every test improved.
The research
A smaller parameter count can still miss a device’s memory or response-time limit, so pruning by size alone may not fit the actual workload.
AutoCompress makes hardware and serving conditions part of that choice; in the tested settings, its models met deployment budgets and scored higher on average than the adapted baseline.
AutoCompress searches model structures against target serving costs, then recovers the selected model to fit deployment limits while retaining quality.
AutoCompress searches target-specific model structures using estimated serving costs, then recovers its choice by distillation.
Across three deployments, its compressed models met budgets and averaged higher than Minitron++ on eight tests covering reasoning, coding, knowledge, and instruction following.
Both methods shared the parent and recovery training, supporting an advantage for AutoCompress’s search in these settings. The averages do not mean every test improved.
The head-to-head comparisons cover three workloads and two parent models; the larger Nemotron case has no Minitron++ comparison. Latency estimates can also lose accuracy when architecture changes affect kernels, cache behavior, or serving overhead.
What the evidence shows
- Pruning for real deployments
AutoCompress is a hardware-guided pruning framework that searches model-wide and per-layer structures against serving costs.
The reason for doing so is practical: parameter count alone can miss latency, throughput, and memory requirements, and existing workflows can require manual translation of those requirements into pruning targets.
- A clear RTX 5070 gap
The eight tests cover reasoning, coding, knowledge, and instruction following, so Avg 8 summarizes performance across these areas. Its average does not show that a method wins on every test.
- A smaller coding gain
On the RTX 5070 game-character workload, AutoCompress’s average was 26.67%, compared with 20.10% for Minitron++. This was the largest gap among the three deployment comparisons, showing the clearest measured advantage in that setting.
- A narrow datacenter edge
For long-context coding on an RTX 5080, AutoCompress had the higher eight-test average within a 13.96 GiB memory budget and an 11.5-second latency target. This supports an advantage in that deployment setting, while the average does not imply a win on every test.
- What the comparison isolates
The methods shared the importance-sorted parent, deployment budgets, workloads, and recovery training data and schedule. Their search procedures and candidate spaces differed: AutoCompress searched global and per-layer structure, while Minitron++ used uniform dimensions and retained all layers. Minitron++ also used more architecture-selection tokens in each of the three comparisons.
- Scale with a boundary
The 74-billion-parameter model fit one B200 at 128K context using 178 GB, but its nine-metric mean score was below its parent’s. With no Minitron++ comparison, this case shows feasibility for one target, not a comparative advantage.
Deep
- Original paper authors
- Sheldon Liang, Ali Taghibakhshi, Saurav Muralidharan, Sharath Turuvekere Sreenivas, Ashwath Aithal
- Submitted to arXiv
- License
- CC BY 4.0