V-SYNTHESIS: Task-Agnostic Synthesis of Consistent and Diverse In-Context Demonstrations from Scratch via V-Entropy

2025-06-29

AI Summary

In zero-shot in-context learning (ICL), synthesizing high-quality demonstration examples without annotations remains challenging due to the inherent trade-off between consistency and diversity, often leading to biased or low-fidelity demonstrations. Method: This paper proposes V-Synthesis, a task-agnostic, zero-shot demonstration generation framework. Its core is V-Score—a novel consistency metric—and an entropy-optimized proportional sampling strategy, enabling fully automated ICL demonstration synthesis without any human-provided examples or task-specific priors. Contribution/Results: V-Synthesis establishes the first truly task-agnostic, zero-label-initiated ICL example synthesis method. Evaluated across multiple benchmarks, it improves downstream task performance by an average of 2.0% while substantially reducing computational overhead. Extensive experiments confirm its superior balance of consistency, diversity, and cross-task generalization, offering a new paradigm for resource-constrained ICL.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Computational Photography, Image & Video Synthesis

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

High labeling cost for in-context learning (ICL) demonstrations motivates using large language models (LLMs) for synthesis to reduce overhead. However, existing synthesis methods are mainly task-specific or rely on pre-existing demonstrations. So this paper focuses on synthesizing demonstrations from scratch for arbitrary tasks. A major challenge in synthesizing from scratch is ensuring consistency with the target task, as the lack of labeling guidance could lead to synthesis bias. We first propose a consistency metric called V-Score, which has higher performance and lower computation cost compared with the metrics based on grams or embedding vectors. Furthermore, we introduce V-Synthesis, which leverages V-Score for proportional sampling to ensure both high consistency and diversity of synthesized demonstrations. Experimental results demonstrate that V-Synthesis yields an average performance improvement of 2.0% compared to existing synthesis methods confirming the effectiveness of V-Synthesis.

Problem

• Synthesizing in-context demonstrations from scratch for arbitrary tasks
• Ensuring consistency with target tasks without labeling guidance
• Balancing high consistency and diversity in synthesized demonstrations

Innovation

• Task-agnostic synthesis of in-context demonstrations
• V-Score ensures consistency with target tasks
• Proportional sampling enhances diversity and consistency