Tabular AI models did not reliably calibrate clinical risks
Tabular foundation models are not a shortcut to calibrated clinical risk prediction: a pre-registered simulation against the true risk · arXiv:2610.39065
In plain words
A clinical risk model should give probabilities that match what happens, not just rank patients from lower to higher risk. Checking that match is difficult in real data because each patient’s true risk is unknown.
This study gave simulated patients known risks, then compared three pretrained models that read data tables with established prediction methods. On average, the models were usually within the study’s comparison margins of its main reference.
But many individual simulated samples still failed the study’s calibration criteria, even with 3,000 patients.
The research
A clinical risk model can rank patients correctly while giving probabilities that are too high or too low. Since real data show each patient’s outcome, not their true risk, how can researchers tell whether a model’s probabilities are calibrated?
The study compares three pretrained table models with established methods using simulated patients whose true clinical risks are known.
To know whether clinical risk probabilities are right, researchers need a way to compare them with true risks, which real patient data do not reveal. This study simulated patients with known risks and compared three pretrained table models with established methods, using FLIC as the main reference.
Across 48 core scenarios, model averages were usually within pre-registered margins of FLIC for true calibration error. Yet many simulated samples did not meet the study’s weak-calibration criteria, even at 3,000 patients.
The average comparison supports similarity in those scenarios, not reliable calibration for every sample.
The simulations assess performance in the population that generated the development data. The real-data analyses were exploratory, and the study does not establish calibration in new populations or for other model versions or settings.
What the evidence shows
- The probability problem
Clinical risk predictions need accurate probabilities, not just a ranking of who is more likely to experience an outcome. Calibration asks whether predicted risks match actual risks, but real data reveal outcomes rather than each patient’s true risk.
That makes it difficult to establish whether a model’s probabilities are close to the underlying risks.
- A direct comparison
The study evaluated three pretrained models for predicting risks from tables: TabPFN-3.5, TabICLv2 and Mitra-v2. Simulations assigned a known true risk to each patient, allowing the researchers to compare predictions with risks directly.
The main reference was FLIC, an established logistic regression method; both approaches used the same simulated development data and predicted the same evaluation data.
- What the averages show
Across 48 core simulation scenarios, the researchers found that mean true calibration error for the models was usually within pre-registered margins of FLIC. “Equivalent” means the paired difference fell within those margins, not that results were identical.
The comparison describes averages over simulated replicates within scenarios, so it cannot establish that a particular development sample will yield calibrated predictions.
- Why averages are not enough
A separate check counted development samples meeting the study’s criteria for weak calibration. At a development size of 3,000, many samples still fell short, especially at event rates of 3%.
This diagnostic qualifies the average comparisons: similar mean error does not mean that calibration is dependable in every sample.
- Where the result stops
The researchers also checked whether estimates from a fixed scenario grid carried over to randomly drawn designs. A statistical model’s predictions agreed with the grid estimates in only 12 to 15 of 48 scenarios, below the study’s transfer threshold.
The authors therefore bound their conclusions by the simulated designs and exploratory real-data analysis.
Deep
- Original paper authors
- Ebrahim Khaled Ebrahim
- Submitted to arXiv
- License
- arXiv non-exclusive