An Intelligent-Cloud Edge Multimodal Interaction System for Robots

2026-07-16

AI Summary

This work addresses the challenge of accurate human gesture perception and semantic scene understanding for resource-constrained robots operating in complex environments by proposing a cloud-edge collaborative multimodal interaction framework. At the edge, an enhanced YOLO-DC gesture detector—integrating CBAM attention mechanisms and DIoU loss—is deployed to improve recognition accuracy for small-scale and occluded gestures. In the cloud, large language models (LLMs) and vision-language models (VLMs) are leveraged for high-level semantic interpretation and task planning. Evaluated on both public and custom datasets, the system achieves precision rates of 98.9% and 95.0%, respectively, with mAP@0.5 scores of 90.7% and 92.7%. The approach attains a maximum task success rate of 95% and a mean user satisfaction score of 3.69 out of 5.

Abstract

Robust human-robot interaction in complex environments requires accurate gesture perception, semantic scene understanding, and reliable task planning under limited onboard computing resources. This paper presents a cloud-edge multimodal interaction framework that integrates an enhanced YOLO-based gesture detector with coordinated large language model (LLM) and vision-language model (VLM) agents. The proposed detector, incorporates the Convolutional Block Attention Module (CBAM) into the neck and replaces the baseline bounding-box regression objective with Distance-IoU (DIoU) loss. These modifications improve feature discrimination and localization for small or partially occluded gestures in complex backgrounds. The cloud layer performs gesture detection, scene understanding, multimodal fusion, and action planning, whereas the TonyPi robot locally handles data acquisition, communication, action execution, and feedback. Experiments on a public gesture dataset and a custom dataset show that YOLO-DC achieves precision values of 98.9% and 95.0%, with mAP@0.5 values of 90.7% and 92.7%, respectively. System-level evaluation yields success rates of 95%, 88%, and 82% for single-action, composite-action, and vision-dependent tasks. A 30 participant evaluation yields an overall mean satisfaction score of 3.69 out of 5. These results demonstrate the feasibility of combining refined gesture detection with multimodal agents for resource-constrained robotic interaction.

Problem

• human-robot interaction
• gesture perception
• scene understanding
• task planning
• resource-constrained robotics

Innovation

• cloud-edge collaboration
• multimodal interaction
• YOLO-DC
• attention mechanism
• robotic task planning