Enhancing LLMs for Governance with Human Oversight: Evaluating and Aligning LLMs on Expert Classification of Climate Misinformation for Detecting False or Misleading Claims about Climate Change

2025-01-23

AI Summary

This study investigates the potential of large language models (LLMs) to govern climate change misinformation—a high-stakes, domain-specific challenge requiring nuanced scientific judgment. Method: We construct a manually curated, expert-annotated binary classification dataset of climate claims, grounded in real-world social media content, and conduct comprehensive benchmarking and supervised fine-tuning across leading LLMs—including GPT-4o, GPT-3.5-turbo, Llama-3, and Mixtral. Contribution/Results: Fine-tuned GPT-3.5-turbo achieves performance comparable to climate experts with two decades of experience; open-source models consistently underperform closed-source counterparts; and state-of-the-art closed-source LLMs still lag behind purpose-built human-in-the-loop tools. Critically, this work provides the first empirical validation that expert-annotated data is indispensable for high-fidelity LLM governance in specialized domains. The findings establish a methodological foundation and actionable pipeline for deploying trustworthy AI in climate science, public health, and political discourse.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Application Domains: Misinformation & Fake News

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Climate misinformation is a problem that has the potential to be substantially aggravated by the development of Large Language Models (LLMs). In this study we evaluate the potential for LLMs to be part of the solution for mitigating online dis/misinformation rather than the problem. Employing a public expert annotated dataset and a curated sample of social media content we evaluate the performance of proprietary vs. open source LLMs on climate misinformation classification task, comparing them to existing climate-focused computer-assisted tools and expert assessments. Results show (1) state-of-the-art (SOTA) open-source models substantially under-perform in classifying climate misinformation compared to proprietary models, (2) existing climate-focused computer-assisted tools leveraging expert-annotated datasets continues to outperform many of proprietary models, including GPT-4o, and (3) demonstrate the efficacy and generalizability of fine-tuning GPT-3.5-turbo on expert annotated dataset in classifying claims about climate change at the equivalency of climate change experts with over 20 years of experience in climate communication. These findings highlight 1) the importance of incorporating human-oversight, such as incorporating expert-annotated datasets in training LLMs, for governance tasks that require subject-matter expertise like classifying climate misinformation, and 2) the potential for LLMs in facilitating civil society organizations to engage in various governance tasks such as classifying false or misleading claims in domains beyond climate change such as politics and health science.

Problem

• Climate Change
• Misinformation
• Large Language Models

Innovation

• Large Language Models
• Misinformation Detection
• Expert-reviewed Data