Retrieval-Augmented Feature Generation for Domain-Specific Classification

2024-06-17

AI Summary

To address the limited interpretability of feature engineering and the difficulty of integrating domain knowledge in few-shot domain classification, this paper proposes a retrieval-augmented, large language model (LLM)-driven feature generation method. First, relevant prior features are retrieved using BM25 and semantic embeddings; then, chain-of-thought prompting guides the LLM to perform domain-specific reasoning, generating semantically coherent and business-consistent novel features. A feature validity verification and dynamic filtering mechanism is further introduced to ensure quality. This work pioneers the synergistic modeling of information retrieval and LLM-based reasoning for feature construction, overcoming the limitations of traditional black-box feature engineering. Evaluated across multi-domain datasets—including healthcare, economics, and geography—the method achieves an average 8.2% improvement in F1-score. The generated features exhibit strong interpretability and domain adaptability.

Technology Category

• Machine Learning: Feature Construction/Reformulation
• Natural Language Processing: Generation
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Feature generation can significantly enhance learning outcomes, particularly for tasks with limited data. An effective way to improve feature generation is by expanding the current feature space using existing features and enriching the informational content. However, generating new, interpretable features in application fields often requires domain-specific knowledge about the existing features. This paper introduces a new method RAFG for generating reasonable and explainable features specific to domain classification tasks. To generate new features with interpretability in domain knowledge, we perform information retrieval on existing features to identify potential feature associations, and utilize these associations to generate meaningful features. Furthermore, we develop a Large Language Model (LLM)-based framework for feature generation with reasoning to verify and filter features during the generation process. Experiments across several datasets in medical, economic, and geographic domains show that our RAFG method produces high-quality, meaningful features and significantly improves classification performance compared with baseline methods.

Problem

• Limited Data
• Domain-specific Features
• Classification Performance

Innovation

• RAFG method
• Interpretable feature generation
• Domain-specific classification improvement