Tagging-Augmented Generation: Assisting Language Models in Finding Intricate Knowledge In Long Contexts

2025-10-26

AI Summary

Current large language models exhibit significantly limited question-answering and multi-hop reasoning capabilities on long-context inputs (e.g., 32K tokens). To address this, we propose Tagging-Augmented Generation (TAG), a lightweight, end-to-end data augmentation method that explicitly models cross-paragraph knowledge associations by embedding structured semantic tags directly into the original long text—without chunking or external retrieval. TAG preserves full-text integrity and requires no model architecture modifications or RAG-style embedding/re-ranking modules; instead, it enhances reasoning solely through tag-augmented prompts. On the NoLima and NovelQA benchmarks, TAG improves long-context QA accuracy by up to 17% and multi-hop reasoning by 2.9%, outperforming strong baselines. Our key contribution is the first introduction of fine-grained, controllable semantic tagging as a lightweight context enhancement paradigm—offering a novel, interpretable approach to long-text understanding.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent investigations into effective context lengths of modern flagship large language models (LLMs) have revealed major limitations in effective question answering (QA) and reasoning over long and complex contexts for even the largest and most impressive cadre of models. While approaches like retrieval-augmented generation (RAG) and chunk-based re-ranking attempt to mitigate this issue, they are sensitive to chunking, embedding and retrieval strategies and models, and furthermore, rely on extensive pre-processing, knowledge acquisition and indexing steps. In this paper, we propose Tagging-Augmented Generation (TAG), a lightweight data augmentation strategy that boosts LLM performance in long-context scenarios, without degrading and altering the integrity and composition of retrieved documents. We validate our hypothesis by augmenting two challenging and directly relevant question-answering benchmarks -- NoLima and NovelQA -- and show that tagging the context or even just adding tag definitions into QA prompts leads to consistent performance gains over the baseline -- up to 17% for 32K token contexts, and 2.9% in complex reasoning question-answering for multi-hop queries requiring knowledge across a wide span of text. Additional details are available at https://sites.google.com/view/tag-emnlp.

Problem

• LLMs struggle with question answering in long contexts
• Existing methods require extensive preprocessing and are sensitive
• Proposing lightweight data augmentation to boost long-context performance

Innovation

• Tagging-Augmented Generation boosts LLM long-context performance
• Lightweight data augmentation without altering document integrity
• Adding tag definitions to prompts improves complex reasoning accuracy