Patch-level visual learning scales with a simpler objective
In plain words
The paper introduces JEM, a way to teach a vision model from unlabeled images. A student model learns to match a teacher’s descriptions of the same image, even when the two see different crops or masked patches. Extra training terms help keep each patch informative and preserve differences between patches.
The authors tested features with small task-specific probes while keeping the vision model fixed. In comparisons at two model sizes, JEM did better than a retrained DINOv2 on six of nine task scores at each size, but not on every task.
The research
Many scalable methods for learning visual features combine several objectives that may not work naturally together, along with extra mechanisms to keep training stable. Can a more principled objective learn strong features across image tasks without relying on that complexity?
JEM trains image features by matching student and teacher patch descriptions across views, with two regularizers that preserve information and spatial structure.
To learn strong visual features at scale, do researchers need to combine several training objectives and extra stabilizing mechanisms? The paper tests a simpler approach: align corresponding image patches across altered views and use two regularizers to preserve information and patch structure.
With backbones fixed, task-specific probes found JEM ahead of retrained DINOv2 on six of nine task scores at each of two model sizes, with consistent gains on dense prediction. This supports the view that patch-level learning can perform strongly across the tested tasks.
The results come from lightweight probes trained on fixed image features, so they do not establish end-to-end performance after retraining the backbone for each task.
The method also relies on an assumption that the chosen image views share the information downstream tasks need; whether that holds depends on the views and tasks.
Analysis
Deep
- License
- arXiv non-exclusive