Prompt-aware routing balances model quality and cost
Ensemble Routing for Prompt-Adaptive Cost-Aware LLM Inference · arXiv:2611.10193
In plain words
Different language models vary in both their answer quality and their cost. This paper introduces PISA, which estimates which model is likely to answer a prompt correctly, and PISA-E, which can choose a group of models and how many to use.
Across four model-routing tests, PISA led the average cost, quality comparison for choosing one model. PISA-E had the highest average task scores among the tested ensemble methods both on familiar task groups and on groups kept aside for testing, when answer quality was prioritized.
The research
Choosing a model for every prompt can trade answer quality for cost, and a fixed group of models cannot adjust its size as prompts change.
These results show that prompt-specific choices can improve average cost–quality trade-offs in four routing tests, changing what an aggregate comparison of routing methods suggests.
PISA estimates which model fits each prompt; its ensemble extension selects a prompt-specific group and size to balance expected answer quality against inference cost.
The researchers tested PISA and PISA-E on four routing tests, averaging task scores within groups and then across groups. For held-out evaluation, each router was trained on other task groups.
PISA led the average cost–quality score for single-model routing, while PISA-E led average ensemble task performance when quality was prioritized.
At selected lower-cost settings, PISA-E also paired slightly higher average task scores with lower average query costs than the strongest competing ensemble method in each setting. The findings support prompt-specific routing choices across these tests, with individual-test exceptions.
The evaluation uses four routing tests, their supplied model responses and scores, and held-out task groups for the out-of-distribution results.
PISA-E estimates ensemble success by assuming model correctness events are independent for a prompt; this estimate can differ from whether its response-combination step selects a correct answer.
What the evidence shows
- Route models by prompt
PISA is a single-model router that estimates each candidate language model’s chance of answering a prompt correctly. PISA-E extends it to choose which models to query together and how many to use.
Their purpose is to let routing respond to differences between prompts while balancing answer quality against inference cost.
- PISA leads on average
Hypervolume summarizes a router’s cost–performance frontier against fixed reference points. PISA’s macro-average lead over the strongest single-model router appears in both task groups represented in training and held-out groups, but it ranks first in only six of eight test-setting comparisons.
- Ensembles score higher on average
When response quality was prioritized, PISA-E’s task performance, averaged within each task group and then macro-averaged across groups, led the strongest competing ensemble method in both training-represented and held-out task groups.
Some competitors scored higher on individual tests, so this aggregate advantage is not a win on every test.
- Cost changes the comparison
These selected operating points pair slightly higher average task scores with lower average query costs. The in-distribution and held-out comparisons use different cost preferences and competing methods, so they show two separate cost–quality trade-offs.
- Prompt-specific choices matter
Versions forced to use a fixed group size kept PISA-E’s other named components. PISA-E had higher average cost–performance scores in both settings, though fixed size did better on RouterBench in each.
A separate comparison found an average advantage for prompt-specific over global model ranking, but global ranking did better on two held-out tests. These controls support aggregate gains while leaving test-specific exceptions.
Deep
- Original paper authors
- Saul Urso, Emanuele Carlini, Patrizio Dazzi, Matteo Mordacchini
- Submitted to arXiv
- License
- CC BY 4.0