Learning-To-Measure: In-context Active Feature Acquisition

2025-10-14

AI Summary

This paper addresses three key challenges in multi-task Active Feature Acquisition (AFA): poor policy generalization across tasks, systematic absence of retrospective data characteristics, and severe scarcity of task labels. To this end, we propose a meta-level AFA framework that enables zero-fine-tuning cross-task policy transfer via in-context learning; guides feature acquisition by maximizing conditional mutual information; and achieves efficient uncertainty quantification through sequence modeling or autoregressive pretraining. An uncertainty-driven greedy acquisition agent is further designed to select features adaptively. Experiments on synthetic and real-world tabular datasets demonstrate that our method significantly outperforms task-specific baselines—especially under extreme label sparsity (<5% labeled instances) and high feature missingness (>80%). To the best of our knowledge, this is the first AFA approach that is truly multi-task通用, requires no fine-tuning, and provides interpretable, principled feature selection.

Technology Category

• Machine Learning: Active Learning
• Search and Optimization: Metareasoning and Metaheuristics
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Active feature acquisition (AFA) is a sequential decision-making problem where the goal is to improve model performance for test instances by adaptively selecting which features to acquire. In practice, AFA methods often learn from retrospective data with systematic missingness in the features and limited task-specific labels. Most prior work addresses acquisition for a single predetermined task, limiting scalability. To address this limitation, we formalize the meta-AFA problem, where the goal is to learn acquisition policies across various tasks. We introduce Learning-to-Measure (L2M), which consists of i) reliable uncertainty quantification over unseen tasks, and ii) an uncertainty-guided greedy feature acquisition agent that maximizes conditional mutual information. We demonstrate a sequence-modeling or autoregressive pre-training approach that underpins reliable uncertainty quantification for tasks with arbitrary missingness. L2M operates directly on datasets with retrospective missingness and performs the meta-AFA task in-context, eliminating per-task retraining. Across synthetic and real-world tabular benchmarks, L2M matches or surpasses task-specific baselines, particularly under scarce labels and high missingness.

Problem

• Adaptively selecting features to improve model performance on test instances
• Learning acquisition policies across tasks with retrospective missing data
• Eliminating per-task retraining through in-context meta-learning approach

Innovation

• Meta-AFA learning across various tasks
• Uncertainty-guided greedy feature acquisition agent
• Autoregressive pre-training for reliable uncertainty quantification