Tabular transformers can replace half their blocks with low classification loss

Deep tabular foundation models can solve new tasks without additional training, but their transformer stacks consume memory and inference computation while simpler tree models remain competitive.

The concrete question is whether much of that depth can be removed without losing the model’s predictive behavior.

A label-free affine translator replaces half of a tabular transformer’s blocks while keeping its original head and nearly preserving classification performance.

The authors fit the map using unlabeled calibration activations and choose the window whose output probabilities most closely match the full model.

Comparing each compressed model with its own full version, half-depth compression reduced mean AUROC by 0.001 to 0.005 and median loss by at most 0.002 across available runs and three seeds.

An identity map that simply dropped the blocks collapsed performance toward chance for most models, while a supervised early-exit classifier lost about 0.02 AUROC and the translator matched or beat it in 81% of 2,664 configurations.

These controls support the interpretation that the translator approximates useful skipped computation.

The strongest one-block setting was not reliable for regression: most regressors lost substantially more R² than at half depth, and one-block-at-a-time tests do not establish that all blocks can be removed together. For very wide dual-axis datasets, translator overhead can outweigh removed parameters.

The native-shallow comparison used only a 1.4-million-parameter nanoTabPFN proxy, so production-scale parity remains untested.