Logit Reweighting for Topic-Focused Summarization

2025-07-07

AI Summary

To address weak topic focus, high fine-tuning costs, and poor prompt-engineering performance of small models in topic-aware abstractive summarization, this paper proposes a lightweight, fine-tuning-free logit reweighting method applied at generation time. We design three strategies: Constant Shift, Factor Scaling, and Threshold Selection—the last being the core innovation, which optimally balances enhanced topic focus with summary quality. Evaluated on Gemma-2B and Llama-3-8B, our method significantly increases topic keyword usage on the NEWTS dataset while preserving summary fluency and informational completeness. Crucially, it achieves zero training overhead and enables efficient, controllable generation. This work establishes a practical, resource-efficient paradigm for topic-guided summarization in compute-constrained settings.

Technology Category

• Natural Language Processing: Summarization
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Data Visualization & Summarization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Topic discovery and tracking

Abstract

Generating abstractive summaries that adhere to a specific topic remains a significant challenge for language models. While standard approaches, such as fine-tuning, are resource-intensive, simpler methods like prompt engineering often struggle to maintain topical focus, particularly with smaller models. To address this, we propose a lightweight method that enhances topical relevance by directly reweighting the logits of topic-relevant tokens during generation. We evaluate three such reweighting techniques: Constant Shift, which adds a constant value to logits; Factor Scaling, which multiplies them by a factor; and Threshold Selection, which selectively boosts logits that exceed a probability threshold. Experiments on the NEWTS topical summarization dataset, using both Gemma-2B and Llama-3-8B models, show that these techniques effectively increase the use of topic-relevant vocabulary. Notably, the Threshold Selection method successfully improves topical focus without compromising summary quality-a trade-off often seen in other approaches. Our findings demonstrate that directly reweighting logits is a practical and resource-efficient alternative to fine-tuning, offering a promising pathway for precisely controlling the thematic content of generated text.

Problem

• Generating topic-focused abstractive summaries is challenging
• Standard methods are resource-intensive or lack topical focus
• Proposes logit reweighting to enhance topical relevance efficiently

Innovation

• Reweighting logits to enhance topical relevance
• Three techniques: Constant Shift, Factor Scaling, Threshold Selection
• Threshold Selection improves focus without quality loss