CAREF: Calibration-Aware Regularization for Explanation Faithfulness Without Rationale Supervision

2026-05-26

AI Summary

This study addresses the critical challenge of simultaneously improving both prediction accuracy and explanation faithfulness in large language models without requiring human-annotated rationales. The authors propose CAREF, a novel framework that integrates entropy-based calibration regularization with token-level sparsity control within a unified loss function, termed LSCED, enabling unsupervised interpretable fine-tuning. Built upon the Flan-T5 architecture, CAREF employs a parameter-efficient fine-tuning strategy—specifically the CAREF-AQ variant—which updates only 6.43% of the model’s parameters. Evaluated across four natural language explanation benchmarks, the method achieves an average accuracy of 89.04% and an explanation alignment score of 81.00 nBERT, significantly outperforming established baselines such as LoRA and AdaLoRA.

Abstract

We introduce CAREF, a parameter-efficient fine-tuning framework that jointly optimizes predictive accuracy and explanation faithfulness via calibration-aware regularization. At its core, CAREF couples entropy-based calibration with token-level sparsity control through a single unified loss, the Calibration-Aware Regularization for Explanation Faithfulness (LSCED), without requiring rationale supervision. Evaluated on four NLE benchmarks (COS-E, ECQA, ComVE, e-SNLI) with Flan-T5, our lightweight CAREF-AQ variant attains the best average accuracy (89.04) and explanation alignment (81.00 nBERT) using only 6.43% of trainable parameters, outperforming LoRA and AdaLoRA. To our knowledge, CAREF is the first method to unify entropy and sparsity regularization in a single training objective for interpretable LLM fine-tuning.

Problem

• explanation faithfulness
• rationale supervision
• calibration
• sparsity regularization
• interpretable LLM

Innovation

• calibration-aware regularization
• explanation faithfulness
• parameter-efficient fine-tuning
• entropy-based calibration
• token-level sparsity