Paper

X-lebench: A benchmark for extremely long egocentric video understanding

table of content

Understanding human behavior from first-person video has long been a challenge in AI research. Most benchmarks today focus on short clips ormoderately long videos, rarely exceeding an hour. But in reality, our lives unfold over many hours, with rich temporal dependencies and contextualcontinuity. X-LeBench introduces the first benchmark designed specifically for extremely long egocentric videos, spanning from 23 minutes up to16.4 hours, to push AI models toward reasoning over long time horizons.

Dataset and Life-Logging Simulation Pipeline

To overcome the difficulty of collecting continuous day-long recordings, the researchers created a life-logging simulation pipeline. This pipelinecombines synthetic daily plans with real-world footage from Ego4D, generating coherent multi-hour video life logs. The process unfolds in threestages:
- Persona Generation: Create diverse character profiles (location + MBTI type), daily agendas, and activity chunks.- Video Extraction: Select from 7,852 Ego4D clips, extracting time, scene, and activity information.- Matching & Simulation: Align daily plan chunks with real video clips to build continuous simulated life logs. The result is 432 video life logs divided into short (~2.3h), medium (~5.3h), and long (~8.6h) categories. They cover 135 daily scenarios such as cooking, shopping, commuting, and watching TV.

[Overview of life-logging simulation pipeline]

Benchmark Tasks
X-LeBench introduces a suite of tasks designed to evaluate long-form video understanding. These tasks challenge models not only to recognize whathappens in short clips but also to track, summarize, and reason across hours of activity:
- Temporal Localization: Identify when specific objects, people, or actions occur.- Summarization: Generate single-video, multi-video, and holistic summaries.- Action Counting: Count occurrences of specific actions.- Summary Ordering: Reconstruct the chronological order of shuffled summaries.

[Tasks in this benchmark and their corresponding examples. Q and A denote query and answer examples, respectively.]

Performance of Existing MLLM Approaches

Poor overall performance: Across all tasks, existing MLLMs struggled significantly on long videos.
Retrieve-Socratic: Best in temporal localization, outperforming Gemini-1.5 Flash by +8.26% recall.
Gemini-1.5 Flash: Strongest in summarization, but limited by token constraints, leading to information loss on long inputs.
Ordering: Accuracy above 85% on short videos, but below 25% on long ones, highlighting difficulty in temporal reasoning.

[Performance comparison on X-LeBench. (a): The performance comparison of each method across tasks for different data categories. (b): The performance comparison across different data categories for different tasks.]

Conclusion

X-LeBench sets a new milestone for egocentric video research, redefining “long video” as multi-hour, continuous,and context-rich recordings. Itreveals that today’s multimodal large language models remain far from mastering such inputs. By providing 432 synthetic yet realistic video life logsand a comprehensive suite of tasks, X-LeBenchlays the foundation for advancing AI toward robust long-term video understanding and real-world memorysystems.