Shared scene views can align separately trained models
Proving and Improving the Platonic Representation Hypothesis with JEPAs · arXiv:2610.39302
In plain words
This theory paper introduces the Platonic Representation Theorem, which asks when models trained separately on different kinds of observations can learn representations that line up.
It says that, under ideal conditions, Joint-Embedding Predictive Architectures can align when they treat observations of the same world states as related examples.
In Flickr tests, text training based on different captions of one image retrieved the correct caption more often than training based on masked parts of one caption or predicting the next word.
That result supports the idea in this tested setting, but does not prove the theorem for real training conditions.
The research
Whether independently trained image and text systems align can depend on how training examples are paired.
The theorem gives a precise condition for shared representations, while the experiments suggest that descriptions of the same scene can matter to alignment alongside the amount of training; the claim remains limited to the objectives and conditions tested.
The Platonic Representation Theorem shows that separately trained JEPAs can align when their positive pairs connect the same world states.
To test whether the theorem’s distinction matters in practice, researchers trained image and text encoders separately, pairing text views either across captions of one image or within a caption.
On 1,000 Flickr test images, the correct caption ranked first after one validation-fitted rotation for 4.4% of good-view cases, versus 2.7% with masked spans and 2.1% with next-word training.
The higher retrieval rate supports the claim that how examples are related during training can affect cross-modal alignment in this setting. A caption-pair control for next-word training reached 2.3%, suggesting that paired captions alone did not reproduce the good-view result.
The theorem assumes a finite world with injective observations, shared positive-pair distributions, population optima and solutions unique up to rotation.
The experiments use LeJEPA’s full-distribution constraint, whose uniqueness is assumed, and mostly use image crops, so they do not meet those hypotheses exactly.
What the evidence shows
- A condition for alignment
The Platonic Representation Theorem is a theoretical result about when separately trained Joint-Embedding Predictive Architectures can learn aligned representations. It says that, under idealized conditions, the models agree on how to represent each world state up to a rotation if their training pairs connect the same world states.
This turns a proposed similarity between modalities into a precise condition for alignment, rather than a claim that all separately trained models will converge.
- Training pairs shape views
The theorem makes the training pair important: two observations of related world states can emphasize shared scene content, while two corruptions of one observation can preserve details such as wording or viewpoint. The authors extend this reasoning to next-token prediction because it relates a document prefix to its continuation.
That language-model interpretation is explicitly heuristic, since next-token prediction is not the spectral JEPA covered by the theorem.
- Flickr separates the objectives
The Flickr comparison tested whether the choice of text views mattered when the image encoder, captions and text architecture were held comparable. After fitting one rotation on validation pairs, the true caption ranked first for 4.4% of test images with good views, versus 2.7% with masked spans and 2.1% with next-token prediction.
A next-token control trained on paired captions reached 2.3%. The result favors the good-view construction in this experiment.
- The pattern holds on web text
On WIT-1024, which scores overlap between image and text representations’ ten nearest neighbors, the good-view JEPA scored. 184 versus. 111 for next-token training on the same texts. This supports the view-based approach in this evaluation, for the models and protocol tested.
- Efficiency and prediction have bounds
A sample-count comparison found that the good-view JEPA reached nearly the next-token model’s WIT alignment using 42 times fewer training samples. The runs used the same encoder size and corpus, but different objectives, sample counts, learning rates and batch sizes, so the comparison does not isolate the effect of sample count.
A separate fitted relation predicted Flickr retrieval scores on 85 additional runs with error similar to rerun variation, within its tested range.
Deep
- Original paper authors
- Randall Balestriero, Yann LeCun
- Submitted to arXiv
- License
- CC BY 4.0