PU-Lie: Lightweight Deception Detection in Imbalanced Diplomatic Dialogues via Positive-Unlabeled Learning

2025-07-12

AI Summary

Deception detection in diplomatic dialogues faces dual challenges: linguistic ambiguity and extreme class imbalance (<5% deceptive instances). To address these, we propose PU-Lie—a lightweight, interpretable model that introduces positive-unlabeled (PU) learning to this task for the first time. PU-Lie freezes the BERT encoder and integrates speaker-aware representations with handcrafted linguistic and strategic features, thereby eliminating reliance on large-scale labeled negative examples. With a parameter count reduced by 650× compared to standard fine-tuned BERT baselines, PU-Lie significantly enhances detection of rare deceptive behaviors. On the Diplomacy dataset, it achieves a macro F1 score of 60%, outperforming prior methods under low-resource, high-stakes conditions. Our approach establishes a novel, efficient, and interpretable paradigm for deception detection in scenarios where annotated negatives are scarce and domain expertise is critical.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Detecting deception in strategic dialogues is a complex and high-stakes task due to the subtlety of language and extreme class imbalance between deceptive and truthful communications. In this work, we revisit deception detection in the Diplomacy dataset, where less than 5% of messages are labeled deceptive. We introduce a lightweight yet effective model combining frozen BERT embeddings, interpretable linguistic and game-specific features, and a Positive-Unlabeled (PU) learning objective. Unlike traditional binary classifiers, PU-Lie is tailored for situations where only a small portion of deceptive messages are labeled, and the majority are unlabeled. Our model achieves a new best macro F1 of 0.60 while reducing trainable parameters by over 650x. Through comprehensive evaluations and ablation studies across seven models, we demonstrate the value of PU learning, linguistic interpretability, and speaker-aware representations. Notably, we emphasize that in this problem setting, accurately detecting deception is more critical than identifying truthful messages. This priority guides our choice of PU learning, which explicitly models the rare but vital deceptive class.

Problem

• Detecting deception in imbalanced diplomatic dialogues
• Addressing extreme class imbalance with PU learning
• Combining BERT embeddings and interpretable features effectively

Innovation

• Uses Positive-Unlabeled learning for imbalance
• Combines BERT embeddings with interpretable features
• Reduces parameters by 650x for efficiency