Two Web Toolkits for Multimodal Piano Performance Dataset Acquisition and Fingering Annotation

2025-09-18

AI Summary

Multi-modal data acquisition and fingering annotation for piano performance have long suffered from cumbersome, fragmented workflows, hindering progress in related research. To address this, we propose an integrated web-based toolchain featuring a dual-GUI architecture: a front-end interface enabling high-precision, synchronized capture of audio, RGB video, MIDI, and performance metadata; and a back-end interface supporting frame-level interactive fingering annotation. Built on modern web graphics technologies, the system ensures real-time operation, scalability, and usability. Experimental evaluation demonstrates that our tool reduces multimodal dataset construction time per piece by ~70% and improves annotation efficiency by over threefold, substantially lowering the barrier to high-quality data production. This work establishes, for the first time, a full-stack, low-threshold, high-accuracy paradigm for multimodal piano performance data construction—providing a reusable foundational infrastructure for music information retrieval, human-computer interaction, and intelligent accompaniment research.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Piano performance is a multimodal activity that intrinsically combines physical actions with the acoustic rendition. Despite growing research interest in analyzing the multimodal nature of piano performance, the laborious process of acquiring large-scale multimodal data remains a significant bottleneck, hindering further progress in this field. To overcome this barrier, we present an integrated web toolkit comprising two graphical user interfaces (GUIs): (i) PiaRec, which supports the synchronized acquisition of audio, video, MIDI, and performance metadata. (ii) ASDF, which enables the efficient annotation of performer fingering from the visual data. Collectively, this system can streamline the acquisition of multimodal piano performance datasets.

Problem

• Acquiring large-scale multimodal piano performance data
• Annotating performer fingering from visual data efficiently
• Overcoming laborious process hindering piano performance analysis

Innovation

• Web toolkit for multimodal piano data acquisition
• GUI for synchronized audio, video, MIDI recording
• Interface for efficient fingering annotation from visuals