Fine-Tuning Language Models to Know What They Know

2026-02-02

AI Summary

This work addresses the limited metacognitive awareness of large language models, which often struggle to accurately assess whether they possess the knowledge required to answer a given question. To bridge this gap, the authors propose Evolution Strategy-based Metacognitive Alignment (ESMA), a novel approach that quantifies metacognitive capability and integrates it into the fine-tuning process. ESMA employs a dual-prompt mechanism to evaluate and enhance the alignment between a model’s internal self-assessment and its external behavioral outputs. Experimental results demonstrate that by adjusting only a small set of critical parameters, ESMA significantly improves both metacognitive accuracy and generalization across seen and unseen tasks, marking a meaningful step toward more self-aware and reliable language models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Metareasoning and Metaheuristics
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Metacognition is a critical component of intelligence, specifically regarding the awareness of one's own knowledge. While humans rely on shared internal memory for both answering questions and reporting their knowledge state, this dependency in LLMs remains underexplored. This study proposes a framework to measure metacognitive ability $d_{\rm{type2}}'$ using a dual-prompt method, followed by the introduction of Evolution Strategy for Metacognitive Alignment (ESMA) to bind a model's internal knowledge to its explicit behaviors. ESMA demonstrates robust generalization across diverse untrained settings, indicating a enhancement in the model's ability to reference its own knowledge. Furthermore, parameter analysis attributes these improvements to a sparse set of significant modifications.

Problem

• metacognition
• language models
• knowledge awareness
• self-knowledge
• type2 performance

Innovation

• metacognition
• dual-prompt method
• Evolution Strategy for Metacognitive Alignment
• language model fine-tuning
• knowledge awareness