GATEAU: Selecting Influential Samples for Long Context Alignment

2024-10-21

AI Summary

Large language models (LLMs) suffer from inefficient sample selection for instruction alignment in extremely long-context scenarios, particularly when modeling long-range dependencies. Method: This paper proposes the first unsupervised sample influence evaluation framework tailored for long-range dependency modeling. It quantifies long-range semantic relevance along two novel dimensions—response generation difficulty and input comprehension difficulty. A joint scoring mechanism is introduced, integrating gradient sensitivity analysis and attention entropy estimation, further enhanced by causal attribution and context perturbation to enable interpretable, annotation-free sample filtering. Results: Evaluated on long-context benchmarks including LooGLE and NarrativeQA, the framework improves average instruction-following and factual consistency accuracy by 4.2%, significantly outperforming existing sample selection paradigms.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Aligning large language models to handle instructions with extremely long contexts has yet to be fully investigated. Previous studies attempt to scale up the available data volume by synthesizing long instruction-following samples, as constructing such a dataset tends to be challenging for annotators. However, a lack of a well-defined strategy for ensuring data quality may introduce low-quality samples and restrict the model performance. Thus, we propose GATEAU, a novel framework to address the unique challenge of long context alignment by identifying the influential samples enriched with long-range dependency relations. Specifically, GATEAU measures the long-range dependencies from two essential aspects: the difficulty of generating target responses due to the long-range dependencies, and the difficulty of understanding long inputs due to such dependencies. Comprehensive experiments indicate that GATEAU effectively identifies influential samples and the model trained on these selected samples exhibits better instruction-following and long-context understanding capabilities.

Problem

• Aligning large language models for long contexts
• Identifying influential samples with long-range dependencies
• Improving model instruction-following and understanding capabilities

Innovation

• Identifies influential long-context samples
• Measures long-range dependency relations
• Enhances model instruction-following capabilities