A Hybrid Framework for Song Lyric Annotation Based on Human-LLM Alignment

2026-06-28

AI Summary

This study addresses the challenges of lyrical emotion annotation, which is inherently subjective and often inconsistent with the overall emotional tone of a song. It presents the first systematic investigation into the agreement between human annotators and large language models (LLMs) in judging lyrical sentiment. To enhance annotation reliability, the authors propose a hybrid intelligence framework that integrates human judgment with LLM-based predictions and introduces an alignment prediction mechanism to anticipate and resolve discrepancies between annotators. Evaluated on a newly constructed sentence-level lyrical emotion dataset, the proposed approach significantly improves both inter-annotator agreement and annotation efficiency, demonstrating the effectiveness of hybrid human–LLM collaboration in handling highly subjective text annotation tasks.

Abstract

Emotion recognition of song lyrics is a challenging task since lyrics may not necessarily align with the overall emotion of a song. As a result, lyrics annotation remains largely underexplored. Drawing inspiration from research in large language model (LLM) assisted annotation, we examine the alignment between humans and LLMs for annotation of lyrics by creating a new sentence-level dataset of lyrics. Our observations highlight the subjectivity of the task and the inherent challenges. Following this, we present a hybrid annotation framework that optimizes human and LLM annotation by predicting potential misalignment in annotation.

Problem

• emotion recognition
• song lyrics
• annotation
• subjectivity
• human-LLM alignment

Innovation

• human-LLM alignment
• lyric emotion annotation
• hybrid annotation framework
• subjectivity in annotation
• sentence-level dataset