SpatialGPT: Zero-Shot Vision-and-Language Navigation via Spatial CoT over Structured Spatial Memory
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Vision-and-Language Navigation (VLN) is a challenging multimodal task in which an autonomous agent must navigate unknown environments by following natural language instructions. Recent zero-shot VLN approaches leverage Large Language Models (LLMs), such as GPT, to interpret instructions and visual inputs for navigation inference without environment-specific training. However, these methods rely solely on the inherent spatial reasoning abilities of LLMs, which often fail to align panoramic observations with language instructions in zero-shot settings. To address this limitation, we propose SpatialGPT, a novel GPT-based VLN agent that incorporates spatial domain knowledge and the Chain-of-Thought (CoT) paradigm to enhance spatial reasoning. SpatialGPT integrates a Directional Connected Landmark List and a Spatial Knowledge Graph to jointly model local and global visual context as structured spatial memory. Built on this memory, we introduce a Synchronize-Align-Backtrack reasoning chain that synchronizes with instruction progress, aligns panoramic views to determine the next action, retrieves alternative paths or infers new frontiers during backtracking. Extensive experiments on the Room-to-Room (R2R) benchmark demonstrate that SpatialGPT achieves state-of-the-art zero-shot performance across all evaluation metrics, showcasing its enhanced spatial reasoning capabilities and strong generalization as an LLM-based VLN agent. The source code is available at SpatialGPT (GitHub)1.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.000 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.000 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.002 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it