A run-count method for testing training equivalence

When a few training runs show no clear difference, researchers still cannot tell whether two methods are practically equivalent. How many independent runs would give enough evidence to make that claim, rather than mistake an inconclusive comparison for proof?

The authors model run-to-run variation and derive a formula for planning equivalence tests between stochastic training methods.

The paper asks how researchers can establish that two stochastic training methods are close enough to count as practically equivalent when a few runs show no clear difference. The authors propose testing a margin chosen in advance, treating each independently trained run as the unit of evidence.

They derive a run-count formula from anticipated variation and compare its predictions with simulated certification rates. Within the tested design-relevant range, the predictions stayed close to those rates.

In a two-run CIFAR-10 example, the test did not certify equivalence; the formula estimated about 12 runs per method for 80% power if the methods were exactly equal.

The CIFAR-10 example had only two runs per method, and its margin was chosen retrospectively. Its estimate of about 12 runs per method assumes the methods are exactly equal and uses a highly uncertain variance estimate, so it is an order-of-magnitude guide, not a certified sample size.