Joint models improve intervals for group treatment effects
Unit-level aggregation breaks coverage in causal foundation models · arXiv:2610.22770
In plain words
Causal models estimate how treatment changes outcomes for people, then average those effects over a group. Their prediction errors can move together because the same uncertain causal model affects everyone. Treating those errors as independent can make the group interval too narrow.
This paper introduces two models for group effects. Added to an existing causal estimator called CausalPFN, they more often placed the true full-sample average inside their intervals across six standard tests with known effects than when the same weights combined predictions unit by unit. Their intervals still fell short of 95% overall.
The research
Should a causal model's interval for a subgroup or full-sample average shrink as if each person's estimation error were independent?
The authors ask because effects for different people depend on the same uncertain cause-and-effect model, so their errors can move together and a group interval can miss the true average too often.
The authors introduce two models that represent uncertainty shared across units when estimating the average treatment effect for a queried group.
One model predicts the queried group average directly; the other shares one learned latent draw across its members and averages their effects.
Across six standard tests with known effects, both models as add-ons to a fixed causal estimator, CausalPFN, more often placed the true full-sample average inside nominal 95% intervals than when the same weights combined unit predictions independently.
This supports representing shared uncertainty, though the direct model also changes its point estimate and neither reached 95% overall.
The tests use simulated response surfaces and score intervals against known conditional effects, not differences between noisy potential outcomes. The proposed models are small and trained on a synthetic family; the authors do not claim they match CausalPFN's accuracy or generality.
What the evidence shows
- Why group intervals can fail
The authors introduce two learned models that estimate uncertainty for an average treatment effect across a chosen set of units, such as a subgroup or the full sample. One represents uncertainty shared across the group; the other predicts the group average directly.
The target is the average of the units' conditional treatment effects, so averaging individual estimates alone does not guarantee a reliable group interval.
- Errors can move together
Every unit's effect depends on the same uncertain structural causal model, the process that generated the data. Errors in predictions for different units can therefore move together.
When a model combines unit estimates as though those errors were independent, it leaves out cross-unit covariance. The paper shows that this can make reported uncertainty too small as the queried group grows; its coverage result also assumes mean-zero Gaussian errors and fixed reported variance.
- Two ways to model dependence
The latent neural process uses one shared learned latent draw for the queried group, predicts each member's effect from it, then averages within that draw. The direct-decode conditional neural process takes the group as input and predicts a distribution for its average directly.
Both methods are designed to preserve uncertainty shared across units when estimating a group effect.
- Existing intervals miss the target
The researchers first test released CausalPFN, an existing causal estimator, on six standard causal test families with simulated response surfaces. Because the true effects are known, they measure what fraction of group intervals contain the true average.
Its calibrated nominal 95% intervals fall below their intended coverage as groups grow on every test family. On ACIC 2016, only 8 of 40 full-sample dataset intervals contain the true average; the public interface uses nearest-neighbor context for unit windows.
- Joint estimates improve coverage
In a fixed-weight synthetic comparison, joint aggregation had median coverage of 89% at the largest group size across 60 cells, while the highest unit-level result was 47%. For the direct model, the point estimate also changes, so this is not a variance-only comparison.
On six standard tests, add-on models on fixed CausalPFN more often placed the true full-sample average inside their intervals than unit-level aggregation from the same weights, though their overall rates remained below 95%.
Deep
- Original paper authors
- Gabe Schulman, Lucius E. J. Bynum, Yoav Wald
- Submitted to arXiv
- License
- CC BY 4.0