A fast updater keeps slow stream models in the loop
RESYNC: Residual Latent-State Synchronization for Deadline-Bound Streaming Inference · arXiv:2611.16523
In plain words
A large model may make better predictions than a small one, but it can take so long that the next data point arrives first. RESYNC lets the large model work in the background while a small updater adjusts its latest saved representation using the newer observations. The large model’s prediction head then answers each arrival.
In tests with 1 millisecond between arrivals, RESYNC returned a prediction for at least 99.4% of arrivals in every tested model and stream pair. A synchronous large model returned none within that deadline. Prediction quality was tested separately.
The research
In a fast data stream, a late prediction may be useless even when the model behind it is strong. RESYNC changes the question from whether to use a slow or small model to whether recent work from the slow model can still help answer each arrival on time. The results suggest it can under the tested stream and timing conditions.
RESYNC updates a slow model’s latest exact representation from newer observations, then reuses its prediction head to answer each stream arrival.
In fixed-age tests, researchers compared RESYNC with the better-scoring compact or distilled compact model on the same held-out targets.
In the twelve pairs where a fresh teacher had a material lead, RESYNC’s paired interval was above zero in all pairs at ages 4–12 and in eleven at older tested ages through 32.
A separate replay found that RESYNC met the 1 ms per-arrival deadline for at least 99.4% of arrivals in every pair, while synchronous teachers met none.
The quality comparison supports recovering much of the teacher’s advantage at tested ages; the replay establishes timely service, not the quality of its delivered predictions.
Quality was measured at fixed anchor ages, separately from live replay. The evaluation used one real stream, two synthetic streams, one training seed and one GPU node; it did not test changing stream dynamics.
What the evidence shows
- Use the slow model’s recent work
RESYNC is a streaming inference method for situations where a high-capacity model takes longer to answer than the interval between observations. It lets that model compute in the background, then uses a lightweight updater to adjust its latest exact internal representation from the observations received since. The unchanged prediction head turns the estimate into an answer for the current observation.
- Old representations lose their edge
The stale-teacher comparison shows that an exact representation quickly loses much of the fresh teacher’s advantage, giving RESYNC a quality problem to solve.
Its advantage over the best fast model across most targeted pairs supports updating from exact anchors, while the Synthetic-1 Transformer exception below shows this pattern is not universal.
- One pair changes the pattern
This exception shows that RESYNC’s fixed-age quality advantage is not universal: on Synthetic-1 Transformer, the distilled compact model catches and then passes it at older gaps. The broader advantage therefore depends on the stream and model, rather than following from the method alone.
- Timely service is a separate result
A separate live replay asked whether each system could answer before the next arrival. With 1 millisecond between arrivals, RESYNC returned its own prediction for at least 99.4% of arrivals in every pair, while no synchronous teacher prediction met the one-interval deadline.
This demonstrates timely service in that replay; it does not establish the quality of the predictions delivered there, which the study measured separately.
- Direct updates help at longer gaps
Because both updaters had the same capacity, objective, training budget and label-free checkpoint selection, their difference isolates the update strategy within this control.
Direct estimates from exact anchors generally held up better over longer gaps than chaining one-step estimates, supporting the direct approach on the tested streams and gaps.
Deep
- Original paper authors
- Eljas Linna, Hao Dong, Juho Kanniainen
- Submitted to arXiv
- License
- CC BY 4.0