ROSAnnotator: A Web Application for ROSBag Data Analysis in Human-Robot Interaction

2025-01-13

AI Summary

HRI research has long suffered from fragmented multimodal ROS 2 Bag data (video, audio, text, custom messages) and qualitative field notes, resulting in low annotation efficiency. This paper introduces the first web-based platform deeply integrating multimodal large language models (Whisper for speech and Qwen-VL for vision-language understanding) to enable human-LLM collaborative annotation of ROS 2 Bag time-series data alongside researcher notes in a unified coding and analysis framework. The system employs a full-stack architecture featuring WebSocket-based real-time synchronization, an extensible annotation protocol, and open RESTful APIs—supporting custom message types and third-party tool integration. Compared to conventional workflows, it significantly improves annotation throughput and enables one-click generation of statistical summaries. The source code is publicly released and has been adopted by multiple HRI laboratories for analysis of real-world robot experiments.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Human-robot interaction (HRI) is an interdisciplinary field that utilises both quantitative and qualitative methods. While ROSBags, a file format within the Robot Operating System (ROS), offer an efficient means of collecting temporally synched multimodal data in empirical studies with real robots, there is a lack of tools specifically designed to integrate qualitative coding and analysis functions with ROSBags. To address this gap, we developed ROSAnnotator, a web-based application that incorporates a multimodal Large Language Model (LLM) to support both manual and automated annotation of ROSBag data. ROSAnnotator currently facilitates video, audio, and transcription annotations and provides an open interface for custom ROS messages and tools. By using ROSAnnotator, researchers can streamline the qualitative analysis process, create a more cohesive analysis pipeline, and quickly access statistical summaries of annotations, thereby enhancing the overall efficiency of HRI data analysis. https://github.com/CHRI-Lab/ROSAnnotator

Problem

• Human-Robot Interaction
• Data Integration
• Analysis Tools

Innovation

• ROSAnnotator
• Human-Robot Interaction (HRI)
• Data Analysis Efficiency