Comparing BERT Sentence-Pair Classification and Few-Shot LLM Prompting for Detecting Threat and Solution Framing in German Climate News

2026-06-24

AI Summary

This study addresses the automated identification of threat and solution framing at the sentence level in German-language climate news, aiming to facilitate large-scale media content analysis. For the first time in this context, it systematically compares a fine-tuned, context-aware BERT model (deepset/gbert-large) against a few-shot prompted open-source large language model (Llama 4 Maverick), the latter augmented with chain-of-thought reasoning, structured output formatting, and confidence scoring. Experimental results show that the BERT-based dual binary classifier achieves an F1 score of 0.83 on both tasks, significantly outperforming the LLM’s 0.78. Ablation studies further confirm that incorporating preceding sentence context is crucial for enhancing BERT’s performance. This work provides an effective approach for frame analysis in German and highlights the impact of contextual modeling and model paradigm choice on frame detection accuracy.

Abstract

News media play a central role in shaping public perceptions of climate change, and whether coverage emphasizes threats or solutions has measurable effects on audience engagement and policy support. Automated detection of these framing patterns at the sentence level would allow researchers to analyze large corpora that are infeasible to code manually. We present a systematic comparison of two approaches for classifying sentences from German-language climate news articles as threat-oriented, solution-oriented, both, or neither. The first approach uses few-shot prompting with an open-weights large language model (Llama 4 Maverick), employing chain-of-thought reasoning and structured output with confidence scoring. The second approach fine-tunes a German BERT model (deepset/gbert-large) for sentence-pair classification, where the preceding sentence provides contextual information for the target sentence. Both approaches implement two independent binary classifiers, one for threat framing and one for solution framing. We evaluate both methods on a corpus of 440 Austrian newspaper articles that were manually coded following a detailed coding scheme developed with domain experts. The fine-tuned BERT classifiers achieve an F1 score of 0.83 for both the threat and solution tasks, while the LLM-based classifiers reach an F1 of 0.78. An ablation study confirms that providing the preceding sentence as context improves BERT classification performance substantially compared to single-sentence input. These results contribute to the growing body of work comparing fine-tuned encoder models with prompted generative models for text classification in computational social science.

Problem

• threat framing
• solution framing
• climate news
• sentence classification
• media framing

Innovation

• few-shot prompting
• sentence-pair classification
• context-aware framing detection
• German BERT
• large language models