Timed text extraction from Taiwanese Kua-á-hì TV series

2026-01-01

arXiv.org

AI Summary

This study addresses the longstanding reliance on manual annotation for extracting subtitles and sung lyrics from low-quality televised Taiwanese opera (Gezai opera) recordings, a bottleneck that has hindered related research. The authors propose a two-stage approach: first, automatic localization of lyric segments is achieved by integrating OCR-driven subtitle segmentation with speech-and-music activity detection (SMAD); second, an interactive real-time OCR correction system refines extraction accuracy and ensures precise temporal alignment. This work presents the first integration of OCR and SMAD for traditional Chinese opera audiovisual analysis and constructs the first Gezai opera lyric dataset with accurate time stamps, thereby enabling downstream music information retrieval tasks such as lyric recognition and melody-based search.

Technology Category

• Natural Language Processing: Information Extraction
• Planning, Routing, and Scheduling: Activity and Plan Recognition
• Search and Optimization: Local Search

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web crawling and indexing
• Web Mining and Content Analysis: Web data integration and cleaning

Abstract

Taiwanese opera (Kua-\'a-h\`i), a major form of local theatrical tradition, underwent extensive television adaptation notably by pioneers like I\^unn L\=e-hua. These videos, while potentially valuable for in-depth studies of Taiwanese opera, often have low quality and require substantial manual effort during data preparation. To streamline this process, we developed an interactive system for real-time OCR correction and a two-step approach integrating OCR-driven segmentation with Speech and Music Activity Detection (SMAD) to efficiently identify vocal segments from archival episodes with high precision. The resulting dataset, consisting of vocal segments and corresponding lyrics, can potentially supports various MIR tasks such as lyrics identification and tune retrieval. Code is available at https://github.com/z-huang/ocr-subtitle-editor .

Problem

• timed text extraction
• Taiwanese opera
• Kua-á-hì
• low-quality video
• manual data preparation

Innovation

• OCR correction
• Speech and Music Activity Detection
• two-step segmentation
• Taiwanese opera
• lyrics alignment