A GPU kernel checker adapts its error limits to each input
Beyond atol and rtol: Sound Test Oracles for Low-Precision GPU Kernels · arXiv:2610.38983
In plain words
A correct GPU calculation can produce slightly different answers for different inputs because numbers are rounded during computation. One fixed error limit may reject a correct answer or overlook a faulty one.
numoracle is a test checker that works out an allowed range for each output from an annotated reference calculation. In tests of 13 Triton operators, it detected more deliberately introduced faults than four fixed checks on the main inputs, though some shifted-input calls could not be certified.
The research
When correct numerical error changes with the input, how can a test avoid rejecting valid GPU results while still exposing faulty kernels? The answer matters because a fixed tolerance has to cover the hardest cases, which can make it too permissive elsewhere.
numoracle turns a precision-annotated tensor reference into per-input output checks, so valid rounding variation can be accepted while faulty outputs are flagged.
numoracle tests GPU kernels by running a precision-annotated reference that tracks error bounds for each input. It checks each output against those bounds and, for low-precision stores, against values reachable by rounding.
On the main inputs for 13 Triton operators, it detected 116 of 124 effective mutants on an RTX PRO 6000, versus 59–83 for four fixed checks. A detection meant failing a mutant while passing its correct kernel on the same input.
The faults were generated edits, not a sample of real-world faults. Under shifted inputs, final-implementation reruns of 634 affected calls were inconclusive, so they were not certified passes.
What the evidence shows
- Why fixed limits fall short
numoracle is an executable test oracle for low-precision GPU kernels. It addresses a testing problem: legitimate numerical error changes with the input, number format and computation’s rounding plan.
A fixed tolerance may therefore be too strict for correct results or too permissive to catch a fault. The checker derives an acceptance region for each output from a reference annotated with the required precision plan.
- Correct error varies by input
The researchers measured this variation using CPU emulations of a correct RMSNorm backward kernel using the 16-bit bfloat floating-point format (bf16), comparing against a PyTorch reference. The smallest equal absolute and relative tolerance that passed every run was 0.00625 for Gaussian inputs and 0.125 for near-degenerate rows.
The 20-fold difference applies to these input families and this setup; with a float64 reference, a different family was hardest.
- More generated faults detected
On the main inputs, numoracle detected 116 of 124 effective mutants on an RTX PRO 6000, compared with 59–83 for four fixed checks. These were generated changes to 13 Triton operators from Liger-Kernel, Unsloth and vLLM.
A mutant counted as detected only when the check failed it and passed the correct kernel on the same input.
- Tuning still had a trade-off
A separate Liger comparison tested 400 tuned tolerance configurations. At most 31 of 53 main mutants could be detected while keeping false alarms at or below 1% in every evaluated Liger input group; numoracle detected 50.
This result covers the tested Liger mutants and input groups, not every possible fixed tolerance rule.
- Shifted calls limit the claim
Shifted inputs tested four families not used to tune the checks. Original campaigns recorded no false alarms among 14,400 correct runs and 2,616 behavior-preserving-variant runs per GPU.
But final-implementation reruns of 634 affected calls per GPU were all inconclusive, not certified passes. The results support the checker in the tested settings while leaving both its assumptions and some shifted calls unresolved.
Deep
- Original paper authors
- Xinyi Shi
- Submitted to arXiv
- License
- arXiv non-exclusive