One parser brings grounded records to images, audio, and video
Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities · arXiv:2610.39301
In plain words
Youtu-Parsing-Omni is one system for turning documents, pictures, sound, and video into structured records. Each record ties what it says to where something appears or when it happens. The system shares most of its perception machinery and uses one output format across ten tasks.
In the authors’ test, it scored higher on average than another open-weight parser, with its advantage concentrated on audio and video. It did not lead on average for images and graphics.
The research
Can a single parser handle documents, images, sound, and video while keeping each extracted item tied to its place or time? The paper asks this because current systems typically split modalities across models and formats, even though structured records need a common grounded form.
Youtu-Parsing-Omni combines a shared perception encoder and output schema so one parser can produce grounded records across documents, images, audio, and video.
On OmniParsingBench, a test of structured parsing, the authors’ toolkit scores grounded fields against references and questions answered from each parse. Its Overall score also accounts for whether outputs remain parseable after schema repair.
Youtu-Parsing-Omni scored 75.08 on the five-split average, versus 72.26 for Logics-Parsing-Omni. Its average lead was concentrated on audio and video, while it trailed on natural images and graphics, a pattern the authors say is consistent with shared temporal alignment during perception.
The OmniParsingBench result uses the authors’ toolkit and covers 3,738 retained samples; the camera-video subset was set aside because of annotation issues, and 136 samples with unavailable media were excluded.
The encoder and post-training comparisons also test specific training settings, so they do not establish the same effects in every parsing setup.
What the evidence shows
- The shared parsing question
Youtu-Parsing-Omni is an omni-modal parsing system that converts documents, images, audio, and video into structured records grounded in places or times. The paper asks whether one parser can cover these inputs with a shared output format.
Its motivation is that different input types are typically handled by different models and formats, even though the resulting records have a common need to anchor content in space or time.
- Where modalities meet
The system’s Youtu-Omni-Encoder uses one mostly shared Transformer for visual and audio inputs, with modality-specific parts to convert inputs and prepare outputs. Visual and audio tokens share a time coordinate, and selected encoder layers let them interact within short windows.
This allows some frame-and-sound alignment to happen during perception, before the decoder produces a record. Shared weights do not eliminate the need to train on a newly added modality.
- How the systems compare
OmniParsingBench tests structured parsing across natural images, graphics, audio, and two kinds of video. It scores grounded fields against references and asks an evaluator to answer questions from each parse; its overall score also accounts for outputs that remain unreadable after schema repair.
Using their own toolkit to score all compared systems, the authors report a five-split average of 75.08 for Youtu-Parsing-Omni and 72.26 for Logics-Parsing-Omni.
- The lead depends on input
The average lead is not uniform. Relative to Logics-Parsing-Omni, Youtu-Parsing-Omni leads by 6.5 points on average across audio and the two video splits, but trails by 2.8 points on average across natural images and graphics.
The authors say the temporal pattern is consistent with their shared timeline design, which lets audio and video interact during perception. The split results qualify that interpretation: the advantage is concentrated where timing matters.
- Two design checks
A separate encoder study compared text-language-model initialization with training from scratch, using the same decoder, general multimodal data without parsing examples, and training schedules. At 300 billion Stage 1 tokens, the initialized encoder scored 767.0 on OCRBench, an image text-reading test, versus 636.4 from scratch.
A sequential post-training comparison on page parsing attributed 1.75 of the 1.99 points gained after supervised fine-tuning to six OSAD rounds, versus 0.24 to the preceding reinforcement-learning stage.
Deep
- Original paper authors
- Yunhang Shen, Zhangquan Chen, Xiaotian Li, Zhongpeng Cai, Qing Zhou
- Submitted to arXiv
- License
- arXiv non-exclusive