On the de-duplication of the Lakh MIDI dataset

2025-09-20

AI Summary

The Lakh MIDI Dataset (LMD) contains extensive symbolic music duplicates arising from multi-user arrangements, minor edits, and metadata modifications, severely compromising the reliability of model training and evaluation. Method: This work presents the first systematic study on symbolic music deduplication, proposing three progressively stricter deduplication strategies for LMD and constructing the Clean MIDI benchmark subset. Our approach integrates rule-based pre-filtering, conventional music retrieval models (e.g., LPQ, Chroma), and an enhanced contrastive-learning BERT model to achieve high-precision duplicate detection and clustering. Contribution/Results: Applied to 178,561 MIDI files, our method identifies and removes at least 38,134 duplicate instances, substantially mitigating data leakage risks and significantly improving generalization and evaluation fidelity across downstream music information retrieval tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Web Mining and Content Analysis: Web data integration and cleaning
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

A large-scale dataset is essential for training a well-generalized deep-learning model. Most such datasets are collected via scraping from various internet sources, inevitably introducing duplicated data. In the symbolic music domain, these duplicates often come from multiple user arrangements and metadata changes after simple editing. However, despite critical issues such as unreliable training evaluation from data leakage during random splitting, dataset duplication has not been extensively addressed in the MIR community. This study investigates the dataset duplication issues regarding Lakh MIDI Dataset (LMD), one of the largest publicly available sources in the symbolic music domain. To find and evaluate the best retrieval method for duplicated data, we employed the Clean MIDI subset of the LMD as a benchmark test set, in which different versions of the same songs are grouped together. We first evaluated rule-based approaches and previous symbolic music retrieval models for de-duplication and also investigated with a contrastive learning-based BERT model with various augmentations to find duplicate files. As a result, we propose three different versions of the filtered list of LMD, which filters out at least 38,134 samples in the most conservative settings among 178,561 files.

Problem

• Identifying duplicate files in Lakh MIDI Dataset caused by multiple arrangements
• Addressing data leakage issues from duplicates during model training evaluation
• Developing effective retrieval methods for detecting symbolic music duplicates

Innovation

• Contrastive learning BERT model with augmentations
• Rule-based approaches for duplicate identification
• Symbolic music retrieval models for de-duplication