IOTA: Corrective Knowledge-Guided Prompt Learning via Black-White Box Framework

2026-01-28

AI Summary

This work addresses the limitation of existing parameter-efficient fine-tuning methods that treat pretrained models as black boxes, thereby neglecting their intrinsic knowledge and constraining downstream performance. To overcome this, we propose IOTA, a novel framework that introduces, for the first time, a synergistic black-box and white-box mechanism. IOTA generates interpretable corrective knowledge by contrasting erroneous predictions with correct reasoning paths and integrates this knowledge into prompt-based learning, enabling a dual-driven adaptation strategy guided by both data and knowledge. Combining prompt learning, knowledge distillation, and an explainable prompt generation and selection strategy, IOTA consistently outperforms state-of-the-art methods across 12 image classification benchmarks under both few-shot and easy-to-hard transfer settings, demonstrating the effectiveness and generalizability of the proposed corrective knowledge.

Technology Category

• Machine Learning: Transparent, Interpretable, Explainable ML
• Computer Vision: Interpretability, Explainability, and Transparency
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Recently, adapting pre-trained models to downstream tasks has attracted increasing interest. Previous Parameter-Efficient-Tuning (PET) methods regard the pre-trained model as an opaque Black Box model, relying purely on data-driven optimization and underutilizing their inherent prior knowledge. This oversight limits the models'potential for effective downstream task adaptation. To address these issues, we propose a novel black-whIte bOx prompT leArning framework (IOTA), which integrates a data-driven Black Box module with a knowledge-driven White Box module for downstream task adaptation. Specifically, the White Box module derives corrective knowledge by contrasting the wrong predictions with the right cognition. This knowledge is verbalized into interpretable human prompts and leveraged through a corrective knowledge-guided prompt selection strategy to guide the Black Box module toward more accurate predictions. By jointly leveraging knowledge- and data-driven learning signals, IOTA achieves effective downstream task adaptation. Experimental results on 12 image classification benchmarks under few-shot and easy-to-hard adaptation settings demonstrate the effectiveness of corrective knowledge and the superiority of our method over state-of-the-art methods.

Problem

• Parameter-Efficient Tuning
• Pre-trained Models
• Downstream Task Adaptation
• Prior Knowledge
• Black Box

Innovation

• prompt learning
• corrective knowledge
• black-white box framework
• parameter-efficient tuning
• knowledge-guided adaptation