PESTO: Real-Time Pitch Estimation with Self-supervised Transposition-equivariant Objective

2025-08-02

AI Summary

Real-time monophonic pitch estimation remains challenging due to strict latency constraints and the scarcity of labeled training data. Method: This paper proposes PESTO, a self-supervised model built upon a Siamese architecture and translation-equivariant priors. It introduces a class-level transposition-equivariant self-supervision objective and incorporates Toeplitz-constrained fully connected layers to explicitly encode pitch-shift invariance—eliminating reliance on annotated data. To enable low-latency streaming inference, PESTO employs the Variable-Q Transform (VQT) and cached convolution. Contribution/Results: With only 130K parameters and end-to-end latency under 10 ms, PESTO significantly outperforms existing self-supervised methods on MIR-1K, MDB-stem-synth, and PTDB, matching supervised state-of-the-art performance. Its core contribution is the first integration of structured equivariance into a lightweight self-supervised pitch estimation framework, achieving both strong cross-domain generalization and real-time capability.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Intelligent Robots: State Estimation
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In this paper, we introduce PESTO, a self-supervised learning approach for single-pitch estimation using a Siamese architecture. Our model processes individual frames of a Variable-$Q$ Transform (VQT) and predicts pitch distributions. The neural network is designed to be equivariant to translations, notably thanks to a Toeplitz fully-connected layer. In addition, we construct pitch-shifted pairs by translating and cropping the VQT frames and train our model with a novel class-based transposition-equivariant objective, eliminating the need for annotated data. Thanks to this architecture and training objective, our model achieves remarkable performances while being very lightweight ($130$k parameters). Evaluations on music and speech datasets (MIR-1K, MDB-stem-synth, and PTDB) demonstrate that PESTO not only outperforms self-supervised baselines but also competes with supervised methods, exhibiting superior cross-dataset generalization. Finally, we enhance PESTO's practical utility by developing a streamable VQT implementation using cached convolutions. Combined with our model's low latency (less than 10 ms) and minimal parameter count, this makes PESTO particularly suitable for real-time applications.

Problem

• Self-supervised pitch estimation without annotated data
• Real-time lightweight model for music and speech
• Cross-dataset generalization surpassing supervised methods

Innovation

• Self-supervised learning with Siamese architecture
• Toeplitz fully-connected layer for translation equivariance
• Streamable VQT implementation with cached convolutions