Problem
Research questions and friction points this paper is trying to address.
Estimating pitch using self-supervised learning
Generalizing across tasks with lightweight models
Reducing performance gap in pitch estimation methods
PESTO: Pitch Estimation with Self-supervised Transposition-equivariant Objective
🤖 AI Summary
This work addresses fundamental frequency (F0) estimation from small-scale, unlabeled monophonic audio data. We propose a lightweight self-supervised equivariant learning framework. Our method introduces three key innovations: (1) a class-conditional pitch-shift equivariant objective to mitigate encoder collapse; (2) a learnable Toeplitz matrix structure that explicitly encodes pitch-shift invariance; and (3) a Siamese network (<30K parameters) integrating Constant-Q Transform (CQT) with class-conditional contrastive learning. Evaluated on cross-dataset tasks spanning vocal and instrumental music, our approach significantly outperforms existing self-supervised F0 estimators and approaches the performance of state-of-the-art supervised methods. Moreover, its compact architecture enables real-time inference under low-resource constraints, making it suitable for edge deployment.
📝 Abstract
In this paper, we address the problem of pitch estimation using Self Supervised Learning (SSL). The SSL paradigm we use is equivariance to pitch transposition, which enables our model to accurately perform pitch estimation on monophonic audio after being trained only on a small unlabeled dataset. We use a lightweight ($<$ 30k parameters) Siamese neural network that takes as inputs two different pitch-shifted versions of the same audio represented by its Constant-Q Transform. To prevent the model from collapsing in an encoder-only setting, we propose a novel class-based transposition-equivariant objective which captures pitch information. Furthermore, we design the architecture of our network to be transposition-preserving by introducing learnable Toeplitz matrices. We evaluate our model for the two tasks of singing voice and musical instrument pitch estimation and show that our model is able to generalize across tasks and datasets while being lightweight, hence remaining compatible with low-resource devices and suitable for real-time applications. In particular, our results surpass self-supervised baselines and narrow the performance gap between self-supervised and supervised methods for pitch estimation.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Self-supervised learning for pitch estimation
Lightweight Siamese neural network
Transposition-equivariant objective function
🔎 Similar Papers
No similar papers found.
