A Modular Vision-Language-Action Robotics Framework for Indoor Environments

2026-06-30

AI Summary

This work addresses the challenge of enabling autonomous robots to execute complex tasks in indoor environments based on natural language instructions. To this end, the authors propose a dual-stream modular architecture that processes visual and linguistic inputs in parallel under temporal constraints. The system leverages OwlViT to construct a semantic voxel map and integrates a vision-language model for instruction parsing and contextual grounding, thereby translating real-time perception into precise action decisions. By effectively fusing geometric, semantic, and linguistic information, the approach supports end-to-end language-driven navigation within limited exploration time. Experimental results demonstrate that the system reliably completes complex tasks within 500 seconds, significantly improving alignment between natural language instructions and robotic behaviors.

Abstract

This paper presents an integrated system for the CMU Vision-Language-Action (VLA) Challenge, designed to enable an autonomous agent to perform complex tasks based on natural language instructions. Our framework employs a modular architecture that orchestrates environment mapping, question processing, and navigation. The system operates in two parallel streams: a perception pipeline that constructs a semantic voxel map from real-time camera feeds using OwlViT embeddings, and a language pipeline that classifies user commands with a Vision-Language Model. The mapping is time-constrained; the system proceeds with a partial map if a 500-second exploration limit is reached. The classified query is then grounded in the geometric and semantic context of the map to generate a detailed prompt for the VLM. This yields an actionable output, demonstrating a capable solution for bridging the gap between human language and robotic action.

Problem

• Vision-Language-Action
• natural language instructions
• autonomous agent
• indoor environments
• robotic action

Innovation

• modular architecture
• semantic voxel mapping
• vision-language model
• time-constrained exploration
• language grounding