Large Language Models for Detection of Life-Threatening Texts

2025-06-12

AI Summary

This study addresses the automatic detection of life-threatening text (e.g., suicidal ideation, self-harm, violent threats) to support psychological crisis intervention. To tackle extreme class imbalance, we construct a multi-scenario balanced dataset and systematically fine-tune open-weight LLMs—including Gemma-7B, Mistral-7B, and Llama-2-7B—comparing them against traditional methods (e.g., bag-of-words, BiBERT). Our work provides the first empirical validation of LLMs’ robustness in severely imbalanced settings, revealing that conventional random oversampling yields marginal gains and uncovering intrinsic mechanisms enabling LLMs to adapt to imbalance without explicit rebalancing. Experiments demonstrate that Mistral-7B and Llama-2-7B achieve F1 improvements exceeding 22% over baselines across diverse class distributions, consistently capturing life-threatening semantics without heavy data resampling. These findings establish a new paradigm for high-risk text detection: efficient, lightweight, and inherently robust—offering practical advantages for real-world mental health applications.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Detecting life-threatening language is essential for safeguarding individuals in distress, promoting mental health and well-being, and preventing potential harm and loss of life. This paper presents an effective approach to identifying life-threatening texts using large language models (LLMs) and compares them with traditional methods such as bag of words, word embedding, topic modeling, and Bidirectional Encoder Representations from Transformers. We fine-tune three open-source LLMs including Gemma, Mistral, and Llama-2 using their 7B parameter variants on different datasets, which are constructed with class balance, imbalance, and extreme imbalance scenarios. Experimental results demonstrate a strong performance of LLMs against traditional methods. More specifically, Mistral and Llama-2 models are top performers in both balanced and imbalanced data scenarios while Gemma is slightly behind. We employ the upsampling technique to deal with the imbalanced data scenarios and demonstrate that while this method benefits traditional approaches, it does not have as much impact on LLMs. This study demonstrates a great potential of LLMs for real-world life-threatening language detection problems.

Problem

• Detecting life-threatening texts to prevent harm and save lives
• Comparing LLMs with traditional methods for text detection
• Evaluating LLM performance on balanced and imbalanced datasets

Innovation

• Fine-tuned open-source LLMs for text detection
• Upsampling technique for imbalanced data scenarios
• Compared LLMs with traditional detection methods