Multi-Agent Robotic Control with Onboard Vision-Language Models

2026-07-08

AI Summary

This work addresses the limitations of existing vision-language models in robotic control—namely, poor interpretability, weak generalization, and reliance on cloud-based computation—by proposing a fully onboard multi-agent architecture. The system deploys lightweight vision-language models (3–20B parameters) alongside vision-language-action models on an AMD Ryzen AI mini PC, enabling autonomous mobile manipulation without external support through fine-tuning and hardware-in-the-loop simulation. A novel “Megamind” coordinating agent is introduced to mitigate the challenge of context retention in long-horizon tasks faced by smaller models. The architecture’s feasibility in terms of cost, performance, and real-world transferability is validated across five industrial warehouse tasks, and the associated simulation environment is open-sourced.

Abstract

Vision Language Models (VLMs) and Vision Language Action (VLA) models have shown promise in robotic control. Yet, they face significant challenges regarding explainability, generalization, and compute requirements. This paper presents a Multi-Agent System (MAS) architecture that addresses these limitations by deploying specialized agents on onboard hardware - eliminating dependence on external compute. The system controls a multi-purpose autonomous mobile manipulator in a simulated industrial warehouse, fulfilling five task categories: safety inspection, warehouse maintenance, warehouse search, package quality verification, and responding to human requests. Compact VLMs (3-20B parameters) are used throughout, with fine-tuning applied to improve package inspection accuracy. A novel "Megamind" orchestration agent mitigates context retention issues inherent to long-horizon planning with smaller models. The system was validated in a hardware-in-the-loop simulation using an AMD Ryzen(TM) AI mini PC. Results demonstrate that a fully onboard MAS architecture is a viable, cost-efficient alternative to cloud-dependent deployments, with strong potential for real-world transfer. The simulation environment has been released as open source under the Apache 2.0 licence.

Problem

• Vision Language Models
• Multi-Agent System
• Onboard Computation
• Robotic Control
• Generalization

Innovation

• Multi-Agent System
• Onboard Vision-Language Models
• Megamind Orchestrator
• Hardware-in-the-Loop Simulation
• Autonomous Mobile Manipulator