🤖 AI Summary
Large language models (LLMs) possess opaque, static internal knowledge, while integrating external knowledge often incurs redundant retrieval, noise interference, and excessive computational cost.
Method: We propose IAPEK—a framework featuring (i) Thrust, the first instance-level knowledge sufficiency metric that dynamically determines retrieval necessity via few-shot distributional representation learning; (ii) traceable knowledge modeling, representation-space analysis, and adaptive retrieval gating for on-demand, precise external knowledge invocation; and (iii) quantification of pre-trained language model (PTLM) intrinsic knowledge capacity to jointly optimize retrieval decisions.
Contribution/Results: IAPEK achieves statistically significant cost-efficiency gains across 88% of evaluated tasks, with an average performance improvement of 26%. It establishes a practical, budget-aware paradigm for knowledge-augmented inference—enabling effective low-cost knowledge enhancement without compromising accuracy.
📝 Abstract
Although large-scale pre-trained language models (PTLMs) are shown to encode rich knowledge in their model parameters, the inherent knowledge in PTLMs can be opaque or static, making external knowledge necessary. However, the existing information retrieval techniques could be costly and may even introduce noisy and sometimes misleading knowledge. To address these challenges, we propose the instance-level adaptive propulsion of external knowledge (IAPEK), where we only conduct the retrieval when necessary. To achieve this goal, we propose measuring whether a PTLM contains enough knowledge to solve an instance with a novel metric, Thrust, which leverages the representation distribution of a small number of seen instances. Extensive experiments demonstrate that thrust is a good measurement of PTLM models' instance-level knowledgeability. Moreover, we can achieve significantly higher cost-efficiency with the Thrust score as the retrieval indicator than the naive usage of external knowledge on 88% of the evaluated tasks with 26% average performance improvement. Such findings shed light on the real-world practice of knowledge-enhanced LMs with a limited knowledge-seeking budget due to computation latency or costs.
