AFABench: A Generic Framework for Benchmarking Active Feature Acquisition

2025-08-20

AI Summary

In real-world scenarios, acquiring full feature sets is often constrained by cost, latency, or privacy, motivating active feature acquisition (AFA)—a paradigm that dynamically selects informative feature subsets to balance predictive performance and acquisition cost. However, the absence of a standardized benchmark hinders fair, reproducible method comparison. To address this, we introduce AFABench, the first general-purpose AFA benchmark framework. First, it integrates diverse real-world and synthetic datasets, including the newly designed AFAContext dataset to evaluate non-myopic strategy robustness. Second, it provides a modular software architecture supporting rapid integration and systematic evaluation of static, greedy, and reinforcement learning–based AFA policies. Third, extensive experiments uncover consistent trade-off patterns between accuracy and cost across methods, empirically validating the long-horizon advantage of reinforcement learning approaches. AFABench establishes reproducible baselines and a standardized evaluation protocol for the AFA research community.

Technology Category

• Machine Learning: Active Learning
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

In many real-world scenarios, acquiring all features of a data instance can be expensive or impractical due to monetary cost, latency, or privacy concerns. Active Feature Acquisition (AFA) addresses this challenge by dynamically selecting a subset of informative features for each data instance, trading predictive performance against acquisition cost. While numerous methods have been proposed for AFA, ranging from greedy information-theoretic strategies to non-myopic reinforcement learning approaches, fair and systematic evaluation of these methods has been hindered by the lack of standardized benchmarks. In this paper, we introduce AFABench, the first benchmark framework for AFA. Our benchmark includes a diverse set of synthetic and real-world datasets, supports a wide range of acquisition policies, and provides a modular design that enables easy integration of new methods and tasks. We implement and evaluate representative algorithms from all major categories, including static, greedy, and reinforcement learning-based approaches. To test the lookahead capabilities of AFA policies, we introduce a novel synthetic dataset, AFAContext, designed to expose the limitations of greedy selection. Our results highlight key trade-offs between different AFA strategies and provide actionable insights for future research. The benchmark code is available at: https://github.com/Linusaronsson/AFA-Benchmark.

Problem

• Standardizing evaluation for Active Feature Acquisition methods
• Addressing lack of diverse datasets for fair AFA comparison
• Testing lookahead capabilities beyond greedy feature selection

Innovation

• Generic benchmark framework for AFA
• Modular design supports new methods
• Novel synthetic dataset tests limitations