A Gesture-Based Visual Learning Model for Acoustophoretic Interactions using a Swarm of AcoustoBots

2026-04-21

AI Summary

This work addresses the limited real-time interactivity of existing AcoustoBot systems, which rely on scripted commands and lack intuitive user interfaces. To overcome this, the authors propose a vision-based gesture learning framework that, for the first time, integrates the OpenCLIP vision-language model into acoustic swarm robotics control. The system leverages an ESP32-CAM module and a PhaseSpace motion capture setup to collect hand gesture data, enabling contactless multimodal interaction through linear probing. Evaluated on a dual-robot platform, the approach achieves 98% accuracy in gesture classification and 87.8% accuracy in modality switching, with an end-to-end latency of 3.95 seconds. These results demonstrate a significant improvement in both the naturalness and responsiveness of human–robot interaction.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Humans and AI: Interaction Techniques and Devices
• Computer Vision: Biometrics, Face, Gesture & Pose

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

AcoustoBots are mobile acoustophoretic robots capable of delivering mid-air haptics, directional audio, and acoustic levitation, but existing implementations rely on scripted commands and lack an intuitive interface for real-time human control. This work presents a gesture-based visual learning framework for contactless human-swarm interaction with a multimodal AcoustoBot platform. The system combines ESP32-CAM gesture capture, PhaseSpace motion tracking, centralized processing, and an OpenCLIP-based visual learning model (VLM) with linear probing to classify three hand gestures and map them to haptics, audio, and levitation modalities. Validation accuracy improved from about 67% with a small dataset to nearly 98% with the largest dataset. In integrated experiments with two AcoustoBots, the system achieved an overall gesture-to-modality switching accuracy of 87.8% across 90 trials, with an average end-to-end latency of 3.95 seconds. These results demonstrate the feasibility of using a vision-language-model-based gesture interface for multimodal human-swarm interaction. While the current system is limited by centralized processing, a static gesture set, and controlled-environment evaluation, it establishes a foundation for more expressive, scalable, and accessible swarm robotic interfaces.

Problem

• AcoustoBots
• human-swarm interaction
• gesture-based interface
• real-time control
• intuitive interface

Innovation

• gesture-based interaction
• visual learning model
• acoustophoretic robots
• vision-language model
• multimodal swarm control