Modeling Data Diversity for Joint Instance and Verbalizer Selection in Cold-Start Scenarios

2025-06-30

AI Summary

In cold-start prompt learning, template design, verbalizer selection, and few-shot instance choice are highly sensitive and mutually dependent. Method: This paper introduces the first formal modeling of the coupling between instance label distributions and verbalizer semantic proximity in embedding space. We propose a joint optimization framework that maps PLM token and mask embeddings into a unified low-dimensional space, enabling coordinated selection of semantically diverse yet low-uncertainty verbalizers and instances via dimensionality reduction and clustering. Contribution/Results: Our approach explicitly models verbalizer–instance dependencies while balancing semantic consistency and data diversity. Evaluated on eight benchmark tasks, it significantly reduces prediction uncertainty and outperforms state-of-the-art few-shot prompting methods in generalization performance. The work establishes a novel paradigm for co-optimizing prompt components under cold-start conditions.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Search and Optimization: Learning to Search
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Prompt-based methods leverage the knowledge of pre-trained language models (PLMs) trained with a masked language modeling (MLM) objective; however, these methods are sensitive to template, verbalizer, and few-shot instance selection, particularly in cold-start settings with no labeled data. Existing studies overlook the dependency between instances and verbalizers, where instance-label probabilities depend on verbalizer token proximity in the embedding space. To address this, we propose COLDSELECT, a joint verbalizer and instance selection approach that models data diversity. COLDSELECT maps PLM vocabulary and $h_{[MASK]}$ embeddings into a shared space, applying dimensionality reduction and clustering to ensure efficient and diverse selection. By optimizing for minimal uncertainty and maximal diversity, COLDSELECT captures data relationships effectively. Experiments on eight benchmarks demonstrate COLDSELECT's superiority in reducing uncertainty and enhancing generalization, outperforming baselines in verbalizer and few-shot instance selection for cold-start scenarios.

Problem

• Joint selection of verbalizer and instances in cold-start scenarios
• Addressing sensitivity to template and verbalizer in prompt-based methods
• Modeling data diversity to improve few-shot instance selection

Innovation

• Joint verbalizer and instance selection approach
• Shared space mapping with dimensionality reduction
• Optimizes minimal uncertainty and maximal diversity