ST-Think: How Multimodal Large Language Models Reason About 4D Worlds from Ego-Centric Videos

How well can AI reasonss about the dynamic, four-dimensional world we live in? Humans effortlessly recall routes, reverse mental timelines, anddescribe how actions unfold over time. For multimodal large language models (MLLMs), however, this kind of spatio-temporal reasoning remains a majorchallenge. The ST-Think project introduces new datasets and training methods designed to push MLLMs closer to human-like reasoning about time andspace.

Background and Challenge

Most benchmarks today only measure spatial reasoning, such as recognizing objects or static layouts. They fail to test whether AI can reason acrosstime—tracking paths, understanding direction changes, or inferring what happened before or after an event. Without this, applications likeself-driving, robotics, and AR/VR assistants remain limited in reliability.

How ST-Think Tackle the Challenge

• Ego-ST Bench: A new dataset with 789 egocentric videos and 5,000+ QA pairs, covering path descriptions, direction shifts, landmark changes, and action changes. Each has both forward and backward reasoning tasks.

**Demonstration of tasks for Ego-ST bench. It includes 8 QAs for 4 types of tasks, which are route description, direction description, landmarkdescription and action description.**

• ST-R1 Training: A two-stage method that combines supervised chain-of-thought (CoT) training with GRPO reinforcement learning. This allows models to learn step-by-step reasoning and refine it via reward optimization.

**\[Spatial Temporal Reasoning Model. Our model is trained in two stages: (1) Create Chain of Thought (CoT) data for supervised fine-tuning (SFT). (2)Enhancing the model using the rule-based reinforcement learning GRPO algorithm.\]**

• Reverse Reasoning: Unique focus on teaching models to 'think backwards,' mimicking human retrospective reasoning.

Results

• Benchmark Tests: On Ego-ST Bench, both open-source (Qwen, LLaVA, InternVL) and closed-source (o3-mini, Gemini) models were evaluated. Results showed reasonable performance on static tasks like landmark recognition, but poor performance on dynamic path and direction reasoning.

• Model Gaps: Most models scored highest on landmarks, lowest on direction reasoning—highlighting a key limitation in temporal understanding.

• ST-R1 Gains: With only limited high-quality CoT data and GRPO, ST-R1 significantly outperformed traditional supervised fine-tuning. In cross-dataset generalization, ST-R1 achieved over 30% improvement compared to baselines.

Future Directions

Looking ahead, ST-Think points toward new horizons for AI reasoning. Promising directions include expanding beyond video to incorporate multimodalsignals like audio and haptics, improving long-term temporal modeling to handle extended sequences, and developing more efficient algorithms that canscale to real-world deployment in robotics, AR/VR, and assistive technologies.

Conclusion

ST-Think advances the frontier of spatio-temporal reasoning for multimodal large language models. By introducing Ego-ST Bench and the ST-R1 trainingmethod, it challenges models not just to recognize what they see, but to reason about how events unfold and even reverse in time. This work brings AIcloser to understanding our 4D world, with wide-reaching implications for navigation, robotics, and human-AI interaction.