Unifying Conformal Language Tasks with In-Context Ensembles

2026-09-02

AI Summary

该研究通过引入结合上下文学习和集成方法的Conformal Relevance框架，解决了自然语言处理任务中覆盖与简洁性的平衡问题。

Abstract

Many NLP tasks, such as summarization and extractive question answering, reduce to retrieving relevant content from documents under two constraints: coverage, retaining enough pertinent information to achieve some goal, and conciseness, removing as much irrelevant information as possible. Conformal prediction methods have been used to guarantee coverage, and must be optimized for conciseness through design of a score function. State-of-the-art scoring functions use hand-engineered LLM prompts asking the model to rate the importance of content, but manual prompt engineering is labor-intensive and task-specific. We introduce the Conformal Relevance framework which uses in-context learning example curation and ensembling to create a score function which maintains coverage while improving conciseness with minimal manual input. We demonstrate this framework's application on seven NLP tasks, and also theoretically study the impact of diversity for ensembled conformal scores, giving a complementarity condition that characterizes when ensembling improves worst-case sentence scores, and a saturation bound on ensemble improvement.

Problem

• Conformal Prediction
• Coverage
• Conciseness
• NLP Tasks
• Score Function

Innovation

• Conformal Relevance
• in-context learning
• ensembling
• coverage
• conciseness