SUMMIR: A Hallucination-Aware Framework for Ranking Sports Insights from LLMs

2026-03-30

AI Summary

This work addresses the challenges of hallucination and insufficient user relevance in sports news insights generated by large language models (LLMs) by introducing SUMMIR, a novel end-to-end framework that jointly optimizes factual reliability and user-centric relevance. SUMMIR uniquely integrates factual consistency verification, hallucination assessment, and user interest modeling, leveraging multiple state-of-the-art LLMs—including GPT-4o and Qwen2.5-72B—and employs multi-stage validation through FactScore and SummaC metrics. Evaluated on a dataset of 7,900 sports news articles spanning four major sports, the proposed approach significantly enhances both factual accuracy and user relevance of generated insights, while also uncovering inherent trade-offs across different LLMs between factual consistency and engaging content generation.

Abstract

With the rapid proliferation of online sports journalism, extracting meaningful pre-game and post-game insights from articles is essential for enhancing user engagement and comprehension. In this paper, we address the task of automatically extracting such insights from articles published before and after matches. We curate a dataset of 7,900 news articles covering 800 matches across four major sports: Cricket, Soccer, Basketball, and Baseball. To ensure contextual relevance, we employ a two-step validation pipeline leveraging both open-source and proprietary large language models (LLMs). We then utilize multiple state-of-the-art LLMs (GPT-4o, Qwen2.5-72B-Instruct, Llama-3.3-70B-Instruct, and Mixtral-8x7B-Instruct-v0.1) to generate comprehensive insights. The factual accuracy of these outputs is rigorously assessed using a FactScore-based methodology, complemented by hallucination detection via the SummaC (Summary Consistency) framework with GPT-4o. Finally, we propose SUMMIR (Sentence Unified Multimetric Model for Importance Ranking), a novel architecture designed to rank insights based on user-specific interests. Our results demonstrate the effectiveness of this approach in generating high-quality, relevant insights, while also revealing significant differences in factual consistency and interestingness across LLMs. This work contributes a robust framework for automated, reliable insight generation from sports news content. The source code is availble here https://github.com/nitish-iitp/SUMMIR.

Problem

• sports insights
• hallucination
• factuality
• LLM
• information extraction

Innovation

• hallucination-aware
• insight ranking
• sports journalism
• fact consistency
• large language models