ROSBag MCP Server: Analyzing Robot Data with LLMs for Agentic Embodied AI Applications

2025-11-05

AI Summary

Current embodied AI research lacks systematic tooling for synergistic analysis of multimodal robotic data—such as trajectories, LiDAR scans, and time-series sensor streams—with large language models (LLMs) or vision-language models (VLMs). Method: We introduce the first Model Context Protocol (MCP)-based ROS/ROS 2 robotics data analytics server, tightly integrating domain-specific knowledge to build a dedicated toolchain. It enables natural-language-driven querying, visualization, and processing—including `ros2 bag` management, topic filtering, and temporal cropping—alongside a lightweight UI for cross-model LLM/VLM benchmarking. Contribution/Results: We evaluate eight state-of-the-art models and find Kimi K2 and Claude Sonnet 4 achieve the highest tool-calling success rates. Our analysis identifies tool description quality, parameter scale, and toolset complexity as key determinants of performance. This work establishes a scalable, interpretable, and model-agnostic analytics paradigm for agentic embodied AI.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Agentic AI systems and Physical or Embodied AI systems have been two key research verticals at the forefront of Artificial Intelligence and Robotics, with Model Context Protocol (MCP) increasingly becoming a key component and enabler of agentic applications. However, the literature at the intersection of these verticals, i.e., Agentic Embodied AI, remains scarce. This paper introduces an MCP server for analyzing ROS and ROS 2 bags, allowing for analyzing, visualizing and processing robot data with natural language through LLMs and VLMs. We describe specific tooling built with robotics domain knowledge, with our initial release focused on mobile robotics and supporting natively the analysis of trajectories, laser scan data, transforms, or time series data. This is in addition to providing an interface to standard ROS 2 CLI tools ("ros2 bag list"or"ros2 bag info"), as well as the ability to filter bags with a subset of topics or trimmed in time. Coupled with the MCP server, we provide a lightweight UI that allows the benchmarking of the tooling with different LLMs, both proprietary (Anthropic, OpenAI) and open-source (through Groq). Our experimental results include the analysis of tool calling capabilities of eight different state-of-the-art LLM/VLM models, both proprietary and open-source, large and small. Our experiments indicate that there is a large divide in tool calling capabilities, with Kimi K2 and Claude Sonnet 4 demonstrating clearly superior performance. We also conclude that there are multiple factors affecting the success rates, from the tool description schema to the number of arguments, as well as the number of tools available to the models. The code is available with a permissive license at https://github.com/binabik-ai/mcp-rosbags.

Problem

• Enabling natural language analysis of robot data through LLMs and VLMs
• Addressing scarce research in Agentic Embodied AI applications
• Providing tooling for analyzing ROS bag data with robotics domain knowledge

Innovation

• MCP server analyzes ROS bags using LLMs
• Built robotics tools for trajectory and sensor data
• Lightweight UI benchmarks multiple LLM tool-calling capabilities