One model links driving-scene understanding with 3D reconstruction

Can a driving-scene model use the same camera video to identify objects and occupied space, then reconstruct the scene's appearance, without combining every view and time point in one costly operation? The paper tests a shared model for these connected needs.

Foundation4D shares multi-camera video features across driving-scene perception and reconstruction, then aligns window outputs into longer scene tracks.

The researchers compare task combinations from the same initialization, with the same four cameras and 80,000 training updates. On the detection test, adding geometry and occupancy raised top-class mAP from 40.94% to 42.14%.

On a separate occupancy test, geometry plus occupancy scored higher than the configuration that also included detection.

These results show that shared training can help one task without improving every task in the tested combinations. A separate test after further training also found higher occupancy overlap scores than the strongest listed baseline.

The task-combination results come from a four-camera study with specific training and validation sets; the researchers do not establish that the pattern holds across all settings.

They also introduce Gaussian reconstruction after perception pretraining, so the paper does not test training both together from the outset.