Can LLMs Understand the Impact of Trauma? Costs and Benefits of LLMs Coding the Interviews of Firearm Violence Survivors

2026-04-17

AI Summary

Manually coding interviews with gun violence survivors is time-consuming and subjective, yet the accuracy and ethical risks of using large language models (LLMs) to analyze trauma narratives remain unclear. This study presents the first systematic evaluation of open-source LLMs in inductively coding interviews with 21 Black male survivors, employing multiple data preprocessing strategies to assess coding relevance and completeness. Results indicate that while certain LLM configurations can identify key themes, overall performance remains low and highly sensitive to input variations. More critically, built-in safety guardrails frequently filter or erase trauma-related content. These findings highlight both the potential and the profound ethical challenges of deploying LLMs in qualitative research involving marginalized populations, underscoring the need to re-examine the fairness and inclusivity of AI-assisted coding practices.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Firearm violence is a pressing public health issue, yet research into survivors' lived experiences remains underfunded and difficult to scale. Qualitative research, including in-depth interviews, is a valuable tool for understanding the personal and societal consequences of community firearm violence and designing effective interventions. However, manually analyzing these narratives through thematic analysis and inductive coding is time-consuming and labor-intensive. Recent advancements in large language models (LLMs) have opened the door to automating this process, though concerns remain about whether these models can accurately and ethically capture the experiences of vulnerable populations. In this study, we assess the use of open-source LLMs to inductively code interviews with 21 Black men who have survived community firearm violence. Our results demonstrate that while some configurations of LLMs can identify important codes, overall relevance remains low and is highly sensitive to data processing. Furthermore, LLM guardrails lead to substantial narrative erasure. These findings highlight both the potential and limitations of LLM-assisted qualitative coding and underscore the ethical challenges of applying AI in research involving marginalized communities.

Problem

• trauma understanding
• firearm violence survivors
• qualitative coding
• large language models
• ethical AI

Innovation

• large language models
• qualitative coding
• firearm violence survivors
• narrative erasure
• ethical AI