A new image tokenizer scales while resisting shortcut features
Generative Representation Learning for Scalable Latent Tokenization · arXiv:2610.20533
In plain words
An image tokenizer turns an image into representations a generative model can use.
This paper introduces Relative Prediction Representations (RePR), a self-supervised way to train one: a prediction model estimates how much a partly hidden image view helps predict a patch, and a second model learns patch representations to match that information.
On ImageNet images with marks that identify their source image, LeJEPA and DINOv3 reportedly kept only the mark. RePR avoided that collapse in this test, suggesting it can retain information when a shortcut is available.
The research
If an image representation keeps only the clues needed for one learning task, can it still preserve the other details a generative model needs?
This question matters because generation must account for variables that a discriminative objective may ignore, and a tokenizer built around that narrower objective could leave useful image information out.
Relative Prediction Representations (RePR) trains image tokenizers to match how much one partly hidden view helps predict another, aiming to keep information generation needs.
The authors test ImageNet generation with Latent Forcing, which denoises the learned representation before producing pixels.
For guided ImageNet generation, with JiT-L and near-matched total training compute, RePR-78M scored 2.18 versus 2.36 for direct pixel JiT-L on the paper’s reported FID score for generated-image results, across 50,000 generations per condition.
Because the budget includes tokenizer and generator training, this supports a generation gain at similar total cost in these configurations.
The result covers the listed ImageNet configurations. The approximately sevenfold comparison with DINOv2 is an extrapolation from an encoder-compute trend, not a directly matched training run; the width-1024 ImageNet encoder also began to overfit after 200 epochs.
What the evidence shows
- The tokenizer’s information gap
An image tokenizer supplies representations that a generative model uses to model images. A representation objective can reward a small set of cues, such as image identity, and discard other image variables.
That feature suppression is a poor fit for generation when the omitted information still needs to be represented.
- A prediction-based method
Relative Prediction Representations (RePR) is the paper’s self-supervised method for training generative-model tokenizers. It uses a prediction model to estimate how much one partly hidden image view helps predict a patch, then trains a separate representation model to make patch vectors reflect that amount. The target comes from a generative prediction task, not negative examples.
- A test of shortcut learning
The authors create watermarked ImageNet images in which every patch carries a mark that identifies its source image. They train LeJEPA, DINOv3, and RePR on both standard and marked images.
On the watermarked images, LeJEPA and DINOv3 reportedly collapse to the watermark alone, while RePR remains robust in this constructed setting.
- Generation at similar compute
For generation, the authors use Latent Forcing, which denoises learned representations before pixels are produced, and compare RePR-78M with JiT-L against direct pixel JiT-L on guided ImageNet generation. At near-matched total training compute, RePR reports a better image-generation score over 50,000 generations per condition. The comparison includes training compute for both tokenizer and generator.
- Scaling across data settings
The authors also vary RePR encoder compute on ImageNet and GPIC, a text-to-image dataset. Their fitted trends project matching DINOv2-level ImageNet performance and a WebSSL-DINO-1B-level GPIC score at lower tokenizer-training compute.
These are projections rather than direct matched runs, and the supplied passages do not define the GPIC score’s calculation.
Deep
- Original paper authors
- Alan Baade, Kyle Sargent, Vincent Sitzmann, Eric Ryan Chan, Justin Johnson
- Submitted to arXiv
- License
- CC BY 4.0