Duluth at SemEval-2026 Task 6: DeBERTa with LLM-Augmented Data for Unmasking Political Question Evasions

2026-04-22

AI Summary

This study addresses the challenge of classifying evasive and ambiguous responses in political interviews by proposing a two-stage classification approach based on DeBERTa-V3-base. The method integrates focal loss, hierarchical learning rate decay, and Boolean discourse features, and—novelty—leverages synthetic data generated by Gemini 3 and Claude Sonnet 4.5 to augment minority-class samples. Evaluated on SemEval-2026 Task 1, the approach achieves a Macro F1 score of 0.76, ranking 8th out of 40 teams (average: 0.70), with notable gains in recall for evasive categories. Error analysis reveals that model misclassifications primarily stem from annotator disagreement on the boundary between “ambiguous” and “clear” responses, underscoring the inherent subjectivity and difficulty of the task.

Technology Category

• Natural Language Processing: Discourse, Pragmatics & Argument Mining
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

This paper presents the Duluth approach to SemEval-2026 Task 6 on CLARITY: Unmasking Political Question Evasions. We address Task 1 (clarity-level classification) and Task 2 (evasion-level classification), both of which involve classifying question--answer pairs from U.S.\ presidential interviews using a two-level taxonomy of response clarity. Our system is based on DeBERTa-V3-base, extended with focal loss, layer-wise learning rate decay, and boolean discourse features. To address class imbalance in the training data, we augment minority classes using synthetic examples generated by Gemini 3 and Claude Sonnet 4.5. Our best configuration achieved a Macro F1 of 0.76 on the Task 1 evaluation set, placing 8th out of 40 teams. The top-ranked system (TeleAI) achieved 0.89, while the mean score across participants was 0.70. Error analysis reveals that the dominant source of misclassification is confusion between Ambivalent and Clear Reply responses, a pattern that mirrors disagreements among human annotators. Our findings demonstrate that LLM-based data augmentation can meaningfully improve minority-class recall on nuanced political discourse tasks.

Problem

• political question evasions
• clarity classification
• evasion classification
• response clarity
• political discourse

Innovation

• DeBERTa
• LLM-augmented data
• focal loss
• class imbalance
• political question evasion