SpatialMem: Metric-Aligned Long-Horizon Video Memory for Language Grounding and QA

Xinyi Zheng, Yunze Liu, Chi-Hao Wu, Fan Zhang, Hao Zheng, Wenqi Zhou, Walterio W. Mayol-Cuevas, Junxiao Shen — University of Bristol & Memories.ai Research
arXiv:2601.14895v2 [cs.CV], 5 Mar 2026

Overview

SpatialMem is a memory-centric system for long-horizon, language-grounded retrieval and QA from egocentric video, where metric 3D serves as an interpretable indexing scaffold rather than an explicit mapping objective. Starting from casually captured egocentric RGB video, it builds a metric-aligned spatial scaffold for indoor scenes, detects structural 3D anchors (walls, doors, windows) as first-layer support, and populates a hierarchical memory with open-vocabulary object nodes that link evidence patches, visual embeddings, and two-layer textual descriptions to 3D coordinates for compact storage and fast retrieval. This enables interpretable, spatially grounded queries over relations (e.g., distance, direction, visibility) and supports downstream tasks such as language-guided retrieval/QA and offline navigation-style guidance over a prebuilt memory, without specialized sensors. Experiments on one public Replica scene and two real-world egocentric indoor scenes show that SpatialMem maintains stable layout reasoning, offline guidance, and hierarchical retrieval despite increasing clutter and occlusion. A compact ablation shows the two-layer description memory improves path-level grounding, while moderate scale perturbation causes only limited degradation.

Problem and Contributions

The target task is building a 3D queryable memory from casual, egocentric RGB video streams alone — no depth sensors (RGB-D), inertial measurement units (IMUs), or calibrated Visual-SLAM — to maximize accessibility and enable low-cost deployment on commodity hardware like mobile phones and egocentric cameras. Building such a unified memory from monocular 2D images poses four coupled challenges: (1) Monocular Reconstruction — recovering accurate camera pose and dense depth from monocular streams, robust to motion blur, occlusion, and dynamic lighting; (2) Metric-Upright Alignment — aligning recovered structure and objects to a common metric upright coordinate frame so spatial expressions have precise, actionable geometric meaning; (3) hierarchical structure — indoor scenes are hierarchical (walls define rooms, which contain objects), and this context must be captured and maintained for compositional reasoning and multi-hop retrieval; (4) Low-Latency Querying after memory construction, even with noisy inputs and scene dynamics.

The main contributions are:

• A unified, hierarchical 3D memory system built purely from egocentric RGB video, integrating geometry, semantics, and language into a single queryable structure.
• A two-layer object description mechanism that supports compositional reasoning by encoding both general attributes and contextual relations.
• 3D-grounded open-vocabulary querying via structural anchors (e.g., walls and doors) with explicit metric relations for precise spatial reasoning.
• A practical low-latency query architecture over a prebuilt memory using lightweight indexing for retrieval and offline guidance.

Method

Given a casually captured RGB sequence with intrinsics and estimated extrinsics, SpatialMem reconstructs a point set in an upright, metric frame and stores the memory as a rooted tree T = (V, E) with typed nodes and edges. It is evaluated as an offline memory-construction plus low-latency query system: heavy geometry is computed once, while downstream retrieval and navigation-style guidance operate over the constructed memory.

3D environment preparation

Camera poses and dense depth are estimated with a swappable back-end (e.g., VGGT / SLAM3R / COLMAP), fused into a point set, and aligned by fitting the floor plane (normal → +z), stabilizing the heading, and applying a height prior for scale — yielding an allocentric, metric frame with well-defined predicates. Unless otherwise stated, VGGT is the geometry back-end. On the aligned geometry, anchors are proposed — walls as large vertical planes, doors/windows as thin vertical openings from edge/depth gaps — then verified across views, deduplicated, and kept only where consistently supported (sufficient point support, multi-view coverage, and temporal persistence). Coarse 3D object boxes may optionally be seeded with an off-the-shelf SpatialLM cue and refined later by multi-view evidence. Stable anchors are promoted to Level-1 nodes with geometry, semantics, and a confidence score.

Unified hierarchical spatial memory

The memory is a rooted, metric hierarchy with four layers: Root (scene metadata and global frame; for indoor scenes, rooms); Level-1 Anchors (structural elements — walls, doors, windows — as oriented planes/boxes, initially detected by SpatiaLLM); Level-2 Objects (instances linked to 3D boxes and multi-view 2D crops/masks); and Level-3 Descriptions (attributes such as category, color, size, material, plus relations to anchors/objects). Each node stores geometry G(v), semantics S(v), and text D(v); edges represent parent–child hierarchy and typed relations (e.g., near / attached / left-of).

Metric grounding and relational semantics

After alignment, the world frame is upright with +z as up, so vertical relations are uniform. Plain definitions are used: an object is on a support when its bottom is roughly level with the support's top and their footprints overlap; over/above when clearly higher; below/under when clearly lower — implemented with fixed geometric heuristics (tolerance thresholds for height difference and footprint overlap). Lateral relations (left/right/front/behind) are recorded as frame tags, reconciled across views using anchors, and retained as view-local hints that can be fused with image understanding at query time. In short, vertical predicates are evaluated once in the global z-aligned frame, while lateral predicates are first egocentric and then consolidated into allocentric relations where reliable support exists.

Two-layer descriptions

Two description layers are used. Layer 1 (image-level) is tied to the current frame: it records what an object is and where it is relative to nearby 3D anchors and objects as seen now, and its content may change with occlusion, lighting, or viewpoint. Layer 2 (uniform scene-level) summarizes what remains true across frames, with conservative updates — observations are promoted only when attributes or anchor-relative relations stay sufficiently consistent across repeated evidence, avoiding drift. All relations in both layers refer to named 3D anchors or objects in the shared metric frame rather than free text.

Query and retrieval (low latency)

Queries are answered by walking the memory graph. A "locate" query finds nodes whose names and simple geometry match the request; a "relational" query follows a short chain of steps (e.g., wall → window → mug) while checking distances and directions. To keep this fast, nodes are organized by type and 3D region so a query only inspects nearby candidates, with lightweight distance, relative-orientation, and visibility checks. In the offline evaluation, indexes are built once from the recorded scene memory, so query response time stays short. Navigation-style guidance transforms a natural-language instruction into a sequence of anchor and object waypoints over the prebuilt memory to generate step-wise guidance.

Experimental Setup

SpatialMem is evaluated on three representative scenes: Scene 1 (Replica), a public indoor scene with a relatively simple layout and limited clutter; Scene 2 (Suite main room), a more complex room with multiple major furniture pieces; and Scene 3 (Laboratory/storage), a highly challenging real engineering lab with many heterogeneous items. The two real-world egocentric scenes (Scenes 2–3) were captured with a lightweight LUCI pin camera and will be released with query sets, annotations, and the evaluation protocol. All quantitative results use VGGT as the geometry back-end; evaluations run on an offline memory constructed once from the recorded RGB sequence. Navigation is evaluated as navigation-style guidance over a shared 2D floor-projected reference map of predefined, human-verified decision points, rather than online closed-loop control, and ground truth is established with human verification. The system is measured on three axes: relative position (Relation Accuracy, plus per-anchor accuracy), navigation-style guidance (Success Rate, Step Completion, SPL, Navigation Error), and object retrieval (Retrieval Success, hierarchical correctness Acc_parent / Acc_path, and description quality Color Acc, Loc@parent, Loc@path).

Results

Layout understanding

Across 1,500 relative-position experiments over the three scenes, SpatialMem's overall Relation Accuracy is approximately 0.84 / 0.78 / 0.74 for Scenes 1–3, closely tracking Gemini (~0.86 / 0.80 / 0.74). In Scene 1 the strongest anchor-specific performance is on wall relations (0.88), with door and window both 0.82; SpatialMem is slightly lower than Gemini on wall relations but comparable on door- and window-related relations. As complexity increases in Scenes 2 and 3 some degradation is expected, but the model stays stable across all three scenes, whereas models such as Qwen and MiniCPM drop more clearly in Scene 3 — and SpatialMem remains faster at test time.

Indoor guidance

Guidance is evaluated as memory-grounded offline navigation-style guidance rather than coordinate-level path planning. In Scene 1, SpatialMem achieves the highest Step Completion (0.89 vs. Gemini's 0.84), a competitive SPL (0.69 vs. 0.68), a comparable Success Rate (0.77 vs. 0.78), and a slightly lower Final Error (3.1 m vs. 3.2 m). Across scenes it maintains Step Completion of 0.89 / 0.86 / 0.83 (Scenes 1–3), while clutter and occlusion increase Navigation Error to 3.1 / 3.7 / 4.6 m; the system remains practically useful for offline decision-point guidance.

Object retrieval

In Scene 1, SpatialMem reaches Retrieval Success 0.83 with strong hierarchical accuracy (Acc_parent 0.86, Acc_path 0.76) and localization (Loc@parent 0.86, Loc@path 0.72), close to or slightly better than the strongest baseline, Gemini (SR_obj 0.81, Acc_parent 0.85, Acc_path 0.74), with slightly higher Color Accuracy (0.82 vs. 0.80). Retrieval Success is 0.83 / 0.78 / 0.72 across scenes, with localization remaining relatively strong (Loc@parent 0.86 / 0.83 / 0.77; Loc@path 0.72 / 0.66 / 0.69). The main limitation appears in fine-grained details such as shelf levels or small-object disambiguation when image cues are unclear.

Ablation

On Scene 2, removing the two-layer descriptions causes a consistent drop across all reported metrics (e.g., SR_nav 0.79 → 0.72, SR_obj 0.79 → 0.68, Acc_path 0.69 → 0.62, Loc@path 0.65 → 0.58, NE 3.7 → 4.3 m), indicating this layer improves long-horizon memory organization and path-level grounding. Under ±10% scale perturbation the changes remain limited, with only a small decrease in path-level metrics, suggesting reasonable robustness to moderate metric-scale bias.

Conclusion & Limitations

Conclusion. SpatialMem is a memory-centric system for offline indoor guidance and object retrieval that unifies geometry, semantics, and language in a single queryable representation. From egocentric RGB input it reconstructs an upright metric 3D structure, attaches open-vocabulary objects to structural anchors, and assigns concise two-layer descriptions encoding attributes and anchor-relative relations. Across layout understanding, instruction-based offline guidance, and object retrieval it remains competitive as scene complexity increases; explicit anchoring and preserved local relations support stable step-wise execution, competitive path efficiency, and reliable hierarchical retrieval, with only moderate degradation in more cluttered environments. Future work will broaden usability through more diverse scenes, incremental updates, and richer interaction.

Scope and limitations. The evaluation is a proof-of-concept on one public Replica scene and two self-captured indoor scenes of increasing complexity. All experiments use offline memory, and navigation is evaluated as decision-point guidance over a shared prebuilt reference rather than online closed-loop control. The three scenes represent difficulty levels within the current evaluation subset, not the full extent of the real-world data. Broader public-benchmark coverage and stronger 3D-aware baselines remain future work.

Summary of arXiv:2601.14895v2. All figures and reported numbers are drawn directly from the source paper.

‍

‍

‍