Two-way attention helps compact audio-visual data stay useful
In plain words
A large audio-visual dataset can be difficult to shrink while keeping useful links between what a video shows and what its audio contains. X-AVDD is a method for making a small labeled set of paired examples, using each type of information to help shape the other.
In one test, a ConvNet trained on its synthetic examples got more VGGSound test examples right than one trained on AVDD examples, at 10 examples per class. That result describes this dataset and setup, not every model or task.
The research
Can a small audio-visual dataset keep enough information to train a useful classifier while preserving the link between sounds and images? The paper identifies memory demands and weak cross-modal alignment as obstacles in earlier methods, including AVDD.
It tests whether shaping synthetic pairs with information from both modalities can address that problem.
X-AVDD creates compact paired audio-visual training data by matching examples in both directions through cross-attention.
The method is tested by training classifiers on its synthetic examples and checking how often they correctly label test examples. On VGGSound, at 10 examples per class, a ConvNet with ensemble fusion reaches 24.0% test accuracy with X-AVDD data, compared with 8.2% with AVDD data.
The feature-center analysis also finds closer audio and visual class centers for X-AVDD synthetic data on AVE. Together, these results support the idea that two-way matching can preserve useful paired information in the tested settings.
These findings concern audio-visual classification, the task used to evaluate the distilled data. The paper does not test whether the method works for temporal localization or audio-visual retrieval.
Analysis
Deep
- License
- arXiv non-exclusive