TokenRouter helps models hand off requests without restarting work
In plain words
A response can move between language models as it is being written, but serving systems built for one model can make those handoffs slow and awkward.
TokenRouter is a serving system that lets developers describe when a request switches models, while its runtime handles the handoff and lets the request resume with its saved state.
In tests at concurrency four, TokenRouter produced output tokens faster than the stronger available baseline in each of 15 routing-algorithm and workload combinations. The size of the gain varied across those combinations.
The research
When a response switches between language models, a serving system designed around one model may delay the request or redo work when it returns. Can a serving system let each model run independently while preserving a request across those handoffs?
TokenRouter separates routing instructions from asynchronous model serving, so requests can switch models and resume without restarting their saved serving work.
The system comparison tested whether TokenRouter’s serving design could handle model switches more efficiently than existing implementations.
At concurrency four, across 15 matched algorithm-workload combinations, it generated output tokens faster than the stronger available baseline for each combination.
A separate R2R check compared the scheduler model’s predicted best batching threshold with hardware measurements. The match at every evaluated concurrency supports using the model to choose thresholds in those tested configurations.
The threshold model assumes routing probabilities stay constant across decoding steps and routing decisions are independent across requests.
Its reported match with measured optimal thresholds is limited to the evaluated R2R configurations and concurrency levels; the authors leave cases that violate the model’s assumptions for future work.
Analysis
Deep
- Authors
- Tianyu Fu, Tengxuan Liu, Ruoxi Wang, Yixin Dong, Yi Ge
- License
- CC BY 4.0