A compact screen speeds forecast-ensemble searches
In plain words
Forecast ensembles combine several models, but choosing their weights after data arrive can mean recalculating calibrated prediction intervals for each candidate in every group.
CALQA is a fixed-size summary that screens weight choices without rereading the data rows; it can also be merged and adjusted when data are deleted.
The researchers screened candidates with CALQA, then exactly recalculated the 100 leading choices. Across five public forecast panels, their mean held-out block score was 99.95% of exact search’s score, and the search was faster in all 34 tested families.
The research
When forecast weights are chosen after data arrive, exact calibration can require scoring the same group’s rows again for every candidate. Can a compact summary narrow a large search while keeping the chosen configuration’s calibrated intervals close to those found by exact recomputation?
CALQA screens post-hoc ensemble weights from a mergeable summary, then exactly rechecks leading candidates before choosing a forecast configuration.
The researchers built CALQA, a mergeable summary that screens candidate ensemble weights without rereading the calibration rows. It estimates each candidate’s interval shift from stored block-score histograms, then exact recomputation can check the leading choices.
Across the public-panel search, rechecking the top 100 produced a mean held-out block-Winkler score equal to 99.95% of exact search’s score. Because the screen narrowed which candidates needed row-level recalculation, search ran faster across all 34 tested families.
The close search scores do not establish target coverage for the selected forecasts when deployed with their exact shifts: mean held-out block coverage was 0.84–0.90 per family, below the 0.9 target.
The paper recommends recalibrating a selected configuration on held-out blocks when nominal coverage is needed.
Analysis
Deep
- Original paper authors
- Xiufeng Liu, Ruyu Liu, Yusen Liu, Huan Huo, Per Sieverts Nielsen
- Submitted to arXiv
- License
- arXiv non-exclusive