Are Lexicon-Based Tools Still the Gold Standard for Valence Analysis in Low-Resource Flemish?

2025-06-04

AI Summary

This study addresses the problem of whether traditional lexicon-based tools (LIWC, Pattern) remain the gold standard for sentiment valence analysis in low-resource Flemish, and evaluates the applicability of Dutch-finetuned large language models (LLMs) based on BERT and LLaMA architectures. Using a dataset of 25,000 naturally occurring Flemish narratives with human-annotated continuous valence scores (−50 to +50), we systematically compare model performance via regression and correlation analyses. Our results—first of their kind for Flemish—demonstrate that all LLMs exhibit significantly lower predictive accuracy than lexicon-based methods; LIWC and Pattern further show superior robustness and contextual adaptability. The study identifies critical gaps in cultural adaptation and language-specific modeling, underscoring the need for linguistically grounded, culturally informed benchmarks. It advances the development of domain-customized evaluation frameworks and model adaptation paradigms for low-resource languages in affective computing.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Sentiment analysis and opinion mining
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Understanding the nuances in everyday language is pivotal for advancements in computational linguistics&emotions research. Traditional lexicon-based tools such as LIWC and Pattern have long served as foundational instruments in this domain. LIWC is the most extensively validated word count based text analysis tool in the social sciences and Pattern is an open source Python library offering functionalities for NLP. However, everyday language is inherently spontaneous, richly expressive,&deeply context dependent. To explore the capabilities of LLMs in capturing the valences of daily narratives in Flemish, we first conducted a study involving approximately 25,000 textual responses from 102 Dutch-speaking participants. Each participant provided narratives prompted by the question,"What is happening right now and how do you feel about it?", accompanied by self-assessed valence ratings on a continuous scale from -50 to +50. We then assessed the performance of three Dutch-specific LLMs in predicting these valence scores, and compared their outputs to those generated by LIWC and Pattern. Our findings indicate that, despite advancements in LLM architectures, these Dutch tuned models currently fall short in accurately capturing the emotional valence present in spontaneous, real-world narratives. This study underscores the imperative for developing culturally and linguistically tailored models/tools that can adeptly handle the complexities of natural language use. Enhancing automated valence analysis is not only pivotal for advancing computational methodologies but also holds significant promise for psychological research with ecologically valid insights into human daily experiences. We advocate for increased efforts in creating comprehensive datasets&finetuning LLMs for low-resource languages like Flemish, aiming to bridge the gap between computational linguistics&emotion research.

Problem

• Assessing lexicon-based tools for valence analysis in Flemish
• Comparing LLMs and traditional tools in emotion detection
• Addressing gaps in low-resource language emotion analysis

Innovation

• Used Dutch-specific LLMs for valence analysis
• Compared LLMs with lexicon-based tools LIWC/Pattern
• Advocated culturally tailored models for Flemish