Self-Correction Distillation for Structured Data Question Answering

2025-11-11

AI Summary

Small language models (e.g., 8B-parameter models) often generate erroneous structured queries for table QA, knowledge graph QA, and temporal knowledge graph QA. To address this, we propose a self-correcting distillation framework. Our method introduces: (1) an Error Prompting Mechanism (EPM) that enables fine-grained error detection and feedback during inference; and (2) a two-stage knowledge distillation paradigm that systematically transfers both query generation and automatic error correction capabilities from large models (e.g., GPT-4) to small models—first such effort in the literature. Experiments across five benchmarks spanning three structured data modalities demonstrate that our approach significantly outperforms existing distillation methods: small models achieve performance close to GPT-4, while the large model augmented with EPM attains state-of-the-art results on most tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Intelligent Query Processing

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Structured data question answering (QA), including table QA, Knowledge Graph (KG) QA, and temporal KG QA, is a pivotal research area. Advances in large language models (LLMs) have driven significant progress in unified structural QA frameworks like TrustUQA. However, these frameworks face challenges when applied to small-scale LLMs since small-scale LLMs are prone to errors in generating structured queries. To improve the structured data QA ability of small-scale LLMs, we propose a self-correction distillation (SCD) method. In SCD, an error prompt mechanism (EPM) is designed to detect errors and provide customized error messages during inference, and a two-stage distillation strategy is designed to transfer large-scale LLMs'query-generation and error-correction capabilities to small-scale LLM. Experiments across 5 benchmarks with 3 structured data types demonstrate that our SCD achieves the best performance and superior generalization on small-scale LLM (8B) compared to other distillation methods, and closely approaches the performance of GPT4 on some datasets. Furthermore, large-scale LLMs equipped with EPM surpass the state-of-the-art results on most datasets.

Problem

• Improving structured query generation accuracy for small-scale language models
• Distilling error-correction capabilities from large to small language models
• Enhancing structured data question answering performance across multiple benchmarks

Innovation

• Self-correction distillation method for small LLMs
• Error prompt mechanism detects and corrects errors
• Two-stage distillation transfers query and correction skills