OmniRetriever: Any-to-Any Audio–Video–Text Retrieval via Fusion-as-Teacher Distillation

Yunze Liu · Chi-Hao Wu · Enmin Zhou · Junxiao Shen — Memories.ai Research
arXiv:2605.26641v1 [cs.CV], 26 May 2026

Overview

Recent audio–video–text (AVT) encoders can produce a joint (T, V, A) embedding whenever all three modalities are present, but standard pairwise InfoNCE objectives leave this joint signal unused during training. OmniRetriever closes that gap with fusion-as-teacher distillation, which treats a stop-gradient copy of the fused embedding as a teacher signal for the single-modal embeddings, paired with a Tuple-InfoNCE term that supervises the fused embedding directly. The objective is instantiated as OmniRetriever-7B. Across six zero-shot benchmarks it surpasses closed-source Gemini Embedding 2 by 13.3–18.0 R@1 on Clotho and SoundDescs, and reaches the contemporary zero-shot specialist band of open video–text encoders on MSR-VTT and MSVD. The authors also release OmniRetriever-Bench, a 12-direction AVT benchmark of 3,782 triples; on it OmniRetriever-7B attains AVG-all 34.84 (+1.72 over Gemini Embedding 2, +8.03 over the best prior open-source AVT method). Weights, datasets, and code will be released.

Problem and Contributions

Unified AVT encoders (e.g., WAVE, Omni-Embed-Nemotron) route all three modalities through one backbone and produce z_TVA on a single forward pass — exactly what a dual-modal or full-tuple query (e.g., T+V→A) needs at inference. Yet their standard recipe optimizes only three pairwise InfoNCE losses on the single-modal sub-encoders, so those sub-encoders are trained in isolation with no signal about their cross-modal neighbours. The gap is sharpest on audio-anchored retrieval: on Clotho A→T, Gemini Embedding 2 reaches R@1 = 1.34 and the best open omni-modal system 3.5, while CLAP-family audio–text specialists reach 25–26.

• Fusion-as-teacher distillation. The joint embedding z_TVA of a unified AVT encoder supervises its own single-modal sub-encoders. L_D alone gives the dominant single-loss gain; L_T further improves the A↔V routes.
• OmniRetriever-7B, an open 7 B AVT retriever: +13–18 R@1 over Gemini on Clotho and SoundDescs, within ~2 R@1 of the audio–text specialist band on Clotho T→A, and matching the open video–text specialist band on MSR-VTT and MSVD.
• OmniRetriever-Bench, a 12-direction pool of 3,782 held-out triples across all six single- and six dual-modal directions — the first public benchmark to evaluate dual-modal AVT queries.

Method

A unified embedder f_θ produces single-modal embeddings z_T, z_V, z_A and a joint embedding z_TVA from the same forward backbone. OmniRetriever combines three objectives:

Pairwise alignment (L_A)

A symmetric InfoNCE over the pairs {(T,V), (T,A), (V,A)} — the recipe used by existing open AVT embedders, retained by OmniRetriever. Because L_A never operates on the joint vector, z_TVA is neither supervised nor distilled under pairwise alignment alone. The paper's Pairwise baseline uses L_A only.

Fusion-as-teacher distillation (L_D) — primary

The joint forward of f_θ on (T, V, A) is the only step where all three modalities interact. A symmetric InfoNCE pulls each single-modal embedding toward a stop-gradient copy of z_TVA and away from other in-batch joint vectors. Because the teacher is the same backbone consumed jointly — not an external unimodal teacher — each student is trained against the joint geometry an A+T+V query reaches at inference, at the cost of one extra joint forward pass per step. Audio-anchored directions benefit most (audio-related mean +3.73 vs. video-only +2.48).

Tuple-InfoNCE refinement (L_T)

A regularizer that keeps z_TVA informative about all three modalities, preventing collapse onto a dominant pair (in practice T–V). It supervises z_TVA directly against the in-batch tuple grid plus one modality-cycled hard negative per anchor, built by shuffling exactly one modality with a derangement; the shuffled slot cycles across {T, V, A} with period 3 (k = 1 negative per anchor).

Final objective

L = λ_D·L_D + λ_T·L_T + λ_A·L_A with weights (1, 1, 1) chosen a priori (not tuned on OmniRetriever-Bench); the recipe is reported insensitive to the loss-weight ratio across a wide range.

Experimental Setup

OmniRetriever-7B is an adapter fine-tune of the open-weights WAVE-7B backbone (LoRA on the LLM trunk, an all-layer fusion head, and a BEATs adaptor). Training data is a 1.5 M-triple subset from four public video–text datasets — InternVid, InternVid-FLT, Panda-70M, PVD — plus a small in-house corpus collected with consent, restricted to clips with all three modalities and sample-identifier disjoint from every evaluation pool. Comparisons are against frozen WAVE-7B, open Omni-Embed-Nemotron, and closed Gemini Embedding 2 (fed the same raw audio+video bytes). Numbers are means over seeds {42, 43, 44}; aggregate seed std ≤ 0.18 R@1 is treated as the noise floor.

Results

Audio–text retrieval (zero-shot, R@1)

OmniRetriever-7B beats closed Gemini Embedding 2 by 13.3–18.0 R@1 in every direction. On Clotho T→A its 19.14 sits within ~2 R@1 of the zero-shot specialist band (Cacophony, MGA-CLAP, M2D-CLAP at 20–21) without task-specific fine-tuning; on SoundDescs it reaches 25.00 / 20.70, the strongest open-system numbers in the authors' literature search.

Video–text retrieval (zero-shot, R@1)

OmniRetriever-7B reaches 47.6 / 43.7 on MSR-VTT (matching PE-coreB) and 66.9 / 63.3 on MSVD, with 46.0 / 45.1 on DiDeMo and 58.0 / 55.0 on VATEX. It trails the strongest specialist reporting DiDeMo/VATEX (InternVideo2-1B) by ~9–12 R@1 on three of four directions, and closed Gemini wins all eight video directions by 5–15 R@1 — attributed to Gemini's larger closed video–text corpus. Taken with the audio results, the authors describe it as the strongest open unified omni-modal embedder in their literature search.

OmniRetriever-Bench (12 directions, R@1) and ablation

OmniRetriever-Bench holds 3,782 (T, V, A) triples scored against one shared gallery in every direction (Recall@1), captions human-corrected from a Gemini 3.0 Pro draft. OmniRetriever-7B reaches AVG-all 34.84: +9.52 over frozen WAVE-7B, +8.03 over Omni-Embed-Nemotron, +1.72 over Gemini Embedding 2 (33.12), with gains concentrated on the eight audio-anchored directions (e.g., A→T 11.92 vs. 1.48). The stair-step ablation isolates each loss:

Conclusion, Limitations & Ethics

Conclusion. OmniRetriever turns the otherwise-unsupervised z_TVA into a training signal: L_D distills its stop-gradient copy into the single-modal embeddings and L_T supervises it directly with modality-cycled hard negatives. Both losses reuse the same backbone forward and apply to any unified retriever that produces a joint embedding. OmniRetriever-7B reaches the audio–text specialist band on Clotho T→A, reduces the open omni-modality gap on A→T, and improves over Gemini on OmniRetriever-Bench by 1.72 AVG-all R@1. Weights, code, and the benchmark are released.

Limitations. On T↔V and T↔A+V directions it trails Gemini by 6–10 R@1 on the bench and 5–15 R@1 on external video benchmarks (attributed to data scale). Gemini is accessed only through a deployed multipart API, so its numbers reflect the deployed product, not a capacity ceiling. The recipe is verified on a 7 B WAVE backbone (plus a 3 B replication); other backbones and compression-aware training are left to future work.

Ethics. Embedding inversion is a known attack on retrieval embeddings; application-layer defences are recommended for sensitive deployments. Both artifacts are released for research, and the research-use license prohibits deployment in surveillance applications affecting natural persons. The assembled training subset is not redistributed; OmniRetriever-Bench releases source identifiers, clip intervals, and captions rather than the underlying media.

Summary of arXiv:2605.26641v1. All figures and reported numbers are drawn directly from the source paper.