O-MARC: Omni Memory-Augmented Compression Distillation for Efficient Video Understanding

Peiran Wu, Yunze Liu, Chi-Hao Wu, Chen Chen, Junxiao Shen — University of Bristol, Memories.ai Research & University of Central Florida
arXiv:2605.26584v1 [cs.CV], 26 May 2026

Overview

Omnimodal large language models enable unified audio-video understanding, but long joint token sequences make inference costly, and existing benchmarks do not fully isolate audio-visual association in noisy user-generated videos. This work introduces UGC-AVQA, a public UGC benchmark with 1,000 videos and 4,816 QA pairs, where an audio-removal test ensures that benchmark questions require both acoustic and visual evidence. To reduce inference cost, it proposes OMAC, a training-free plug-in compression method that preserves salient visual memory and temporally grounded audio anchors. To further make compact models robust to compressed inputs, it introduces O-MARC, a compression-distillation framework for learning with memory-compressed multimodal contexts. On Qwen2.5-Omni-3B, O-MARC improves the average score across four benchmarks to 45.8, outperforming full-token inference at 44.1 and OmniZip at 41.0. OMAC also keeps inference efficient, reducing latency by 34.6% (1.53× speedup) and memory by 34.7% compared with full-token inference.

Problem and Contributions

Because audio and video are both token-intensive, their joint representation easily produces long multimodal sequences with substantial memory overhead and slow inference, especially for long videos. Token compression is a natural remedy, but audiovisual compression is non-trivial: video tokens carry spatial and temporal redundancy while audio tokens may encode brief but decisive cues, and the importance of each modality is often defined relative to the other — a sound is meaningful only when grounded in the visible scene, and a visual event may be ambiguous without its audio. Compressing the two modalities in isolation can therefore remove cross-modal evidence needed for accurate reasoning. Evaluation is a second challenge: existing audiovisual benchmarks cover broad omnimodal understanding, long-video reasoning, and temporal alignment, but do not explicitly isolate joint audio-visual association in noisy UGC video. The work makes three contributions:

• UGC-AVQA, a public UGC benchmark whose questions require both modalities, enforced by an audio-removal difficulty filter that keeps a sample only when a strong model fails on the audio-ablated (visual-only) input.
• OMAC (Omni Memory-Augmented Compression), a training-free, plug-in method that keeps compact visual memory and audio anchors and uses the visual-memory distribution to allocate the audio budget, without modifying the backbone model.
• O-MARC, an omni memory-augmented compression-distillation framework that teaches omnimodal models to reason under compressed multimodal contexts.

The UGC-AVQA benchmark

1,000 videos are collected from publicly accessible short-video platforms; only original video URLs, source metadata, detailed captions, and QA annotations are released (no raw video files), preserving provenance and platform access conditions. Each video receives an omni detail caption covering visual entities, scene transitions, actions, acoustic events, and temporal relations. Human annotators then write multiple-choice questions in four categories: Audio-Visual Event Progression, Audio-Visual Scene or Temporal Transition, Cross-Scene Audio-Visual Alignment, and Fine-Grained Audio-Visual Contrast.

To prevent questions from being solved by visual priors alone, the audio track is removed from each candidate and Gemini-3.1-Flash is evaluated on the visual-only input; a sample is included only if the model fails under this audio-ablated setting, and 30% of retained candidates are reviewed and corrected by trained annotators. The resulting benchmark split contains 206 videos and 1,648 QA pairs. The remaining 794 videos form the training split, where Gemini-assisted annotation creates 3,168 QA pairs across four aligned categories (Event Progression, Scene Shift, Audio-Scene Match, Key Multimodal Difference), with 30% human-reviewed. All QA pairs are multiple-choice; overall and category-level accuracy are reported.

OMAC: training-free compression

OMAC compresses long audiovisual inputs by preserving compact memory carriers rather than treating the context as a flat token stream. Given audio tokens, video tokens (indexed by frame t and position p), and a query embedding, it outputs a compressed multimodal sequence with audio memory and visual memory embedded in the original temporal order. Compression strength is controlled by ρ_a for audio and ρ_v for video, with other selection and merging hyperparameters fixed across experiments.

Frame scoring and visual memory. Each frame is summarized into a coarse descriptor (mean over its tokens) and scored by query relevance (cosine with the query). High-scoring frames are selected as key frames, with a selection rule that adds temporal coverage so retained frame memories are both query-relevant and spread across the video. Inside each selected frame, a token contrast score (one minus cosine to the frame centroid) favors tokens that depart from the average frame content — better carriers of local evidence than repetitive background; the top-scoring tokens are kept as explicit visual memory. A small frame memory token is also formed by weighted pooling over the selected tokens and reinserted into the original token order.

Audio memory and budget. OMAC keeps audio anchors, merges nearby audio tokens into anchors, and drops lower-score tokens, then allocates the audio budget across time using the visual-memory distribution, so frames that receive more visual memory also keep more audio.

O-MARC: compression-aware training

O-MARC makes compact models robust to compressed inputs through a compression-aware variant of GRPO (C-GRPO). For each sample, a full-token rollout and a compressed rollout are both sampled from the same current policy, and their reward gap shapes the GRPO advantage. Concretely, the original advantage is replaced by a compression-aware shaped advantage that adds a term proportional to the (rectified) reward gap between the full-token and compressed rollouts, so samples that are already beneficial under GRPO and are strongly degraded by compression receive larger positive updates. The full-token branch therefore does not supervise the compressed branch through token-level probability matching; instead it distills its preference through advantage shaping, explicitly biasing policy optimization toward behaviors that remain robust under memory compression.

Experimental Setup

For all video experiments, frames are sampled at 1 FPS with at most 32 frames per video, a maximum single-frame size of 50,174, and an audio sample rate of 16,000. Unless otherwise specified, compression experiments keep the retained ratio at 30% and evaluate the compressed multimodal context on downstream QA benchmarks; all results are run twice and averaged. For O-MARC distillation, Qwen2.5-Omni-3B is the base model, trained on 4× NVIDIA RTX PRO 6000 GPUs (96 GB each) with the same preprocessing as evaluation (num_train_epochs = 1, batch_size = 1, learning rate 1×10⁻⁵, num_generations = 4). The 3B base is chosen to match the goal of an efficient omnimodal model; larger Qwen2.5-Omni-7B and Qwen3-Omni-30B-A3B provide reference points but are less consistent with the efficiency-oriented setting. Evaluation spans four benchmarks — DailyOmni, UGC-AVQA, OmniVideo(Bench), and WorldSense — reported as an average score.

Results

Benchmark gap

On UGC-AVQA, proprietary models remain strongest (Gemini-3.1-pro 69.1, Gemini-3.1-flash-lite 67.1 average accuracy). Among open-source models, the much larger Qwen3-Omni-30B-A3B reaches 65.0, while Qwen2.5-Omni-3B reaches only 49.0. Cross-Scene Audio-Visual Alignment is the hardest category for several models (e.g., Qwen3-Omni-30B-A3B 60.9), motivating compression that preserves temporally aligned acoustic and visual evidence rather than only reducing visual redundancy.

Training-free compression (OMAC)

At a 30% retained ratio, OMAC consistently beats direct token pruning (OmniZip). On Qwen3-Omni-30B-A3B the average score rises from 48.5 to 50.9 (UGC-AVQA 53.9 → 56.9; OmniVideo 31.7 → 35.5); on Qwen2.5-Omni-7B the average rises from 43.5 to 45.0 (WorldSense 39.6 → 42.4). Preserving visual memory and audio anchors retains more reasoning evidence than pruning at the same ratio.

Compression-aware training (O-MARC)

The larger improvement comes from training. With only 3k training samples, O-MARC on Qwen2.5-Omni-3B raises the average score from 42.8 (OMAC) to 45.8, surpassing the full-token 3B baseline (44.1); UGC-AVQA accuracy rises from 49.2 to 57.9 — the compact model learns to use compressed multimodal memory rather than merely tolerate it. Scaling the same framework to Qwen2.5-Omni-7B improves the average from 45.0 to 47.2 (3k samples) and to 51.1 (100k samples). Against the contemporary OmniSIFT (open-sourced in May, 107k training samples), O-MARC with comparable 100k-plus data reaches a higher average under the same 7B backbone (51.1 vs. 48.7) and substantially higher UGC-AVQA accuracy (64.6 vs. 56.4); OmniSIFT is slightly stronger on WorldSense, while O-MARC is stronger on DailyOmni, UGC-AVQA, and OmniVideo.

Ablations and efficiency

Compression ratio. On UGC-AVQA, OMAC reaches 49.5 at 55% compression (above full-token 49.0 and OmniZip 48.6) and 49.2 at 70% (vs. OmniZip 47.1), showing memory tokens matter most under tight budgets. Video size. On WorldSense, OMAC improves over OmniZip at both 64 frames (41.7 vs. 40.2) and 32 frames (41.2 vs. 39.7), indicating the gain comes from allocating retained memory effectively rather than from more frames. Efficiency. On long video, OMAC reduces latency from 3.81 s to 2.49 s and memory from 24.2 GB to 15.8 GB versus full-token inference, matching OmniZip's cost (2.47 s / 15.7 GB) while giving stronger accuracy. Guidance phase. On DailyOmni, audio-guided visual compression scores 47.7 and visual-guided audio compression 48.2, while the full OMAC design — building visual and audio memory separately, then using retained visual memory to adjust the audio budget — reaches 49.9.

Conclusion & Limitations

Conclusion. The work presents a unified study of efficient omnimodal reasoning across benchmark construction, memory compression, and compression-aware training. UGC-AVQA targets audiovisual association in public UGC videos and uses audio-removal filtering to ensure evaluation requires both acoustic and visual evidence. OMAC introduces a training-free memory-compression mechanism that preserves salient visual memory and temporally grounded audio anchors at nearly the same inference cost as direct pruning; building on this, O-MARC trains compact models to reason under a reduced token budget. Experiments show OMAC consistently improves over OmniZip across model scales, while O-MARC achieves the strongest gains in the efficient 3B setting and outperforms the contemporary OmniSIFT when trained with comparable 100k-plus data. The results indicate that audiovisual compression can be viewed not only as an inference-acceleration technique but as a trainable condition for robust, efficient omnimodal reasoning.

Limitations. UGC-AVQA is built from publicly accessible short-video platforms, so its coverage reflects the style, language, and content distribution of currently available UGC videos. Because video links are released rather than raw files, a small portion of the data may become unavailable over time as platform content changes.

Summary of arXiv:2605.26584v1. All figures and reported numbers are drawn directly from the source paper.