LaMSUM: Amplifying Voices Against Harassment through LLM Guided Extractive Summarization of User Incident Reports

2024-06-22

AI Summary

To address the challenge of manually reviewing large-scale, code-mixed sexual harassment reports in India’s Safe City platform, this paper proposes the first LLM-driven extractive summarization framework tailored to this domain. Methodologically, it introduces a multi-model collaborative architecture integrating Llama, Mistral, and GPT-4o, enhanced by hierarchical text segmentation, prompt-engineered fine-grained extraction decisions, and an ensemble voting mechanism—effectively mitigating LLMs’ abstraction bias and context window limitations. Contributions include: (1) the first explainable and traceable extractive summarization system for code-mixed harassment reports; (2) state-of-the-art performance on the Safe City dataset, significantly outperforming existing baselines; and (3) generation of high-fidelity, structured event overviews that directly inform evidence-based policymaking and targeted anti-harassment interventions.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Citizen reporting platforms like Safe City in India help the public and authorities stay informed about sexual harassment incidents. However, the high volume of data shared on these platforms makes reviewing each individual case challenging. Therefore, a summarization algorithm capable of processing and understanding various Indian code-mixed languages is essential. In recent years, Large Language Models (LLMs) have shown exceptional performance in NLP tasks, including summarization. LLMs inherently produce abstractive summaries by paraphrasing the original text, while the generation of extractive summaries - selecting specific subsets from the original text - through LLMs remains largely unexplored. Moreover, LLMs have a limited context window size, restricting the amount of data that can be processed at once. We tackle these challenge by introducing LaMSUM, a novel multi-level framework designed to generate extractive summaries for large collections of Safe City posts using LLMs. LaMSUM integrates summarization with different voting methods to achieve robust summaries. Extensive evaluation using three popular LLMs (Llama, Mistral and GPT-4o) demonstrates that LaMSUM outperforms state-of-the-art extractive summarization methods for Safe City posts. Overall, this work represents one of the first attempts to achieve extractive summarization through LLMs, and is likely to support stakeholders by offering a comprehensive overview and enabling them to develop effective policies to minimize incidents of unwarranted harassment.

Problem

• Algorithm Development
• Multilingual Analysis
• Sexual Harassment Reporting

Innovation

• LaMSUM
• Multilingual Large Language Models
• Summary Generation