NeoLightning: A Modern Reimagination of Gesture-Based Sound Design

2025-05-15

AI Summary

This work addresses the limited accessibility and constrained interactivity of legacy gestural music systems (e.g., Buchla Lightning) due to technological obsolescence. Methodologically, it introduces a modern reimplementation that tightly integrates MediaPipe’s deep-learning-based pose estimation with the Max/MSP + Processing real-time multimedia framework, enabling low-latency, 3D immersive gesture-driven sound design; a custom multimodal synchronization engine achieves end-to-end latency under 50 ms. Contributions include the revival and substantial extension of the pioneering 1990s interactive paradigm—supporting high-fidelity 3D hand tracking and live improvisational music creation—while ensuring full open-source availability. The system has been deployed in digital arts education and experimental music performances, providing a reproducible technical pathway and practical exemplar for next-generation accessible, scalable gestural audio interaction.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Computer Vision: Biometrics, Face, Gesture & Pose

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

This paper introduces NeoLightning, a modern reinterpretation of the Buchla Lightning. NeoLightning preserves the innovative spirit of Don Buchla's"Buchla Lightning"(introduced in the 1990s) while making its gesture-based interaction accessible to contemporary users. While the original Buchla Lightning and many other historical instruments were groundbreaking in their time, they are now largely unsupported, limiting user interaction to indirect experiences. To address this, NeoLightning leverages MediaPipe for deep learning-based gesture recognition and employs Max/MSP and Processing for real-time multimedia processing. The redesigned system offers precise, low-latency gesture recognition and immersive 3D interaction. By merging the creative spirit of the original Lightning with modern advancements, NeoLightning redefines gesture-based musical interaction, expanding possibilities for expressive performance and interactive sound design.

Problem

• Modernizing gesture-based sound design for contemporary users
• Enhancing accessibility and support for historical musical instruments
• Improving gesture recognition and 3D interaction in music performance

Innovation

• Uses MediaPipe for deep learning gesture recognition
• Employs Max/MSP and Processing for real-time multimedia
• Offers precise, low-latency 3D interaction