A method turns unused risk guarantees into more informative decisions
Recertifying decisions via e-closure: uniform improvements of conformal risk control and randomized e-BH · arXiv:2610.35821
In plain words
Some statistical rules promise to keep average mistakes below a chosen limit, but may leave part of that allowance unused. This paper introduces recertification, a method that measures the unused allowance and uses it to permit more informative decisions while keeping the promise.
The authors prove results for risk control and multiple testing, and test image predictions on CIFAR-100. In that image task, repeated recertification made predictions more specific and smaller on average than the comparison rule, while average risk remained below its target. The finding is limited to the tested image pool and setup.
The research
When a statistical procedure keeps expected loss below a limit, how can it make more informative decisions if its certificate leaves some of the allowed risk unused?
Making decisions less cautious by itself may break the guarantee, so the unresolved issue is whether that slack can be identified and safely put to use.
Recertification measures unused allowance in a procedure’s validity certificate and spends it to admit less conservative decisions under the same guarantee.
Bai and Ramdas introduce recertification, a method that measures unused allowance in a statistical procedure’s validity certificate and uses it to admit less conservative decisions under the same guarantee.
For conformal risk control, an exact augmented-sample characterization shows how to identify that allowance.
On CIFAR-100, 500 rounds improved average specificity and reduced prediction size versus conformal risk control, while mean risk stayed below its target. The theory also gives recertified multiple-testing procedures with certificate-relative guarantees.
The image result is an average over paired splits from a fixed pool of 10,000 CIFAR-100 test images, not a guarantee about each image or other data.
The theoretical improvements and optimality claims also have specific conditions and certificate-relative scope; whether RC-eBH is admissible among all false discovery rate procedures remains open.
What the evidence shows
- The unused-budget problem
Bai and Ramdas introduce recertification, a method for improving statistical procedures that control expected loss under a specified budget. A procedure may leave some of that allowance unused, yet directly making its decisions less cautious can break the guarantee.
Recertification asks whether the unused amount can be measured and spent while keeping the same guarantee.
- Measure before enlarging
The method enlarges a procedure’s validity certificate by an allowable amount, then uses the enlarged certificate to define a menu of actions. That menu retains the original actions and may admit less conservative ones.
For conformal risk control, an exact characterization on each augmented sample identifies the unused budget: the target bound minus the sum of losses when each point in turn is treated as the test case.
- Why the risk bound holds
This characterization explains why the CRC improvement can retain validity. Each round distributes the measured allowance across the held-out positions, so their total loss remains within the original budget.
For valid symmetric controllers, the resulting action is no more conservative on any calibration sample. With an interval of actions and convex losses, repeated rounds converge geometrically to a pointwise-admissible controller.
- A bounded image result
On CIFAR-100 hierarchical image classification, the authors compare 500 rounds with envelope CRC, which coincides with CRC for the confidence-adaptive scores tested. At target risk 0.15, mean correct specificity increases from 70.550% to 71.014%, and mean prediction size drops from 10.77 to 8.92 labels; mean risk remains below target at 0.14955. These averages come from paired splits of a fixed pool of cached predictions.
- A second application
The paper also recertifies randomized e-BH for multiple testing. RU-eBH uses the e-BH certificate; RC-eBH uses a larger closed e-BH certificate.
Each contains the corresponding baseline rejection sets under the shared randomization and is optimal only among procedures certified by its own collection. A Gaussian simulation shows RC-eBH can have higher power in one specified setting, while the broader results vary by setting.
Deep
- Original paper authors
- Tian Bai, Aaditya Ramdas
- Submitted to arXiv
- License
- CC BY 4.0