Action prediction gives relational channels meaning in video models
Inside-JEPA: From Prediction to Relational Grounding in Latent World Models · arXiv:2610.31194
In plain words
A model that predicts what a scene will look like later can use any predictable internal code; prediction alone does not make that code describe how objects relate. Inside-JEPA adds signals for pairs of objects and trains the model to infer the action between frames from how those signals change.
In short action videos, its signals matched relationships calculated from object boxes better than signals trained only for prediction. That test shows alignment with box-based labels, not proof of physical relationships beyond them.
The research
When a model predicts what a scene will look like later, what makes its internal signals stand for relationships between objects rather than just any predictable pattern? That matters because a world model's internal state is useful for relational goals only if those relations can be read from it.
Inside-JEPA trains pairwise channels to help infer the action between frames, giving them a reason to track box-defined object relations.
On Something-Else, a test of short human-action videos with object boxes, researchers compared Inside-JEPA with a model trained only to predict its future latent state. For each relation, they measured whether a matched channel ranked box-defined positive examples above negative ones, then averaged those scores across relations.
The inverse head on relational channels scored 0.658, versus 0.593 for prediction-only training, across three training seeds.
The result supports the idea that action supervision helps channels align with relations: the whole-latent inverse head scored 0.670, while entity-only and shuffled-action controls scored 0.608 and 0.539.
A label-fraction control also found little gain with labels on 1% or 5% of pairs, but a +0.047 gain at 25%.
The evaluations define relations from object boxes, so they show alignment with box-based labels, not proof that channels represent physical relations beyond those boxes.
The study compares against one prediction-only baseline, and its planning test is one-step action selection within Something-Else clips using relation-labelled readouts.
What the evidence shows
- What should channels mean?
Inside-JEPA is a JEPA world-model method that adds a vector of relational channels for every ordered pair of entities in the state. In the Something-Else video test, the evaluated relations come from two-dimensional object boxes; for example, “inside” is an image-plane proxy for containment.
The question is whether these channels track such relations, rather than just carrying information that is easy to predict.
- Give channels an action signal
Prediction alone gives no reason for a channel to represent a relation: a static or arbitrary code can also be predictable. Inside-JEPA trains an inverse-dynamics head to recover the action between frames from changes in the relational channels.
That creates a training signal tied to action, without using relation labels to train the channels or prescribing which channel stands for which relation.
- Compare the training signals
On the main Something-Else setup, the researchers matched each box-defined relation to a different channel and averaged how well the matched channels ranked positive examples above negative ones. This mean score, averaged over three training seeds, was 0.658 for the inverse head reading relational channels and 0.593 for prediction-only training.
A whole-latent inverse head scored 0.670, while entity-only and shuffled-action controls scored 0.608 and 0.539.
- Check labels and robot actions
A control varying action labels found almost no grounding gain with labels on 1% or 5% of training pairs, but a +0.047 gain with labels on 25% (3,490 pairs), compared with +0.066 with all pairs labelled. In robot episodes with continuous end-effector actions and whole tasks held out, the gain over prediction-only training was +0.081. This extends the result to another action type, under the provided robot boxes.
- Test what channels support
Forecasting changed relations scored better with the true action than with a permuted action. In a separate one-step selection test, a probe on relational channels put the correct action among the top five for 42.5% of goals, compared with 24.4% for a probe on the prediction-only model's whole latent.
Both results suggest uses for the channels in the tested video clips; they do not establish broad action success or general planning.
Deep
- Original paper authors
- Hossein Resani, Javen Qinfeng Shi
- Submitted to arXiv
- License
- CC BY 4.0