Home automation can use different models for different calls
In plain words
A home-automation system asks a model to do different jobs, such as sort a request, choose a device action, plan a task, or write code. This paper measures model performance separately at each job and assigns a model to each one.
In one live installation, hosting only the planning and code-writing jobs matched the all-hosted setup on human judgments of the counted cases, while spending less on hosted models. The result is specific to that system and test.
The research
A home-automation agent may use one model to sort a request and another to plan or write code. If every call uses a model capable of the hardest job, simpler calls may also incur hosted-model spend. Can a system assign models by job and still work as well when run end to end?
The authors measure models at each existing Wactorz call site, then test fixed assignments to see which calls can run locally.
The authors test whether a home-automation system can use local models for some jobs while preserving end-to-end results. They measure models separately at five existing call sites, then compare fixed routing choices on the same cases and in a live system.
In paired tests, the strongest all-round local model was indistinguishable from hosted models at four sites, while code writing favored the hosted models. On the offline test, hosting only planning and code writing scored 93.9% accuracy, versus 95.4% for all-hosted Sonnet, with 82% less hosted spend. The live comparison also matched human judgments on the counted cases.
The results come from Wactorz, two installations, and one live deployment judged by a single unblinded person. The live comparison excludes three system-limited cases, and the benchmark’s case mix differs from real call frequency.
Analysis
Deep
- License
- arXiv non-exclusive