CLSGen: A Dual-Head Fine-Tuning Framework for Joint Probabilistic Classification and Verbalized Explanation

2026-04-13

AI Summary

This work addresses the challenge that large language models often fail to provide reliable probabilistic estimates in real-world decision-making scenarios, while conventional fine-tuning approaches risk catastrophic forgetting and degradation of linguistic capabilities. To overcome these limitations, the authors propose CLSGen, a novel framework featuring a dual-head fine-tuning architecture that jointly optimizes probability prediction and natural language explanation generation within a binary classification setting. Through a tailored training strategy and carefully constructed data, CLSGen maintains strong generative fluency while significantly enhancing discriminative performance and output reliability. Experimental results demonstrate that CLSGen consistently outperforms existing methods across multiple benchmarks, achieving higher AUROC and F1 scores, with generated explanations that are both highly consistent with predictions and readily interpretable.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

With the recent progress of Large Language Models (LLMs), there is a growing interest in applying these models to solve complex and challenging problems. Modern LLMs, capable of processing long contexts and generating verbalized explanations, offer significant potential in addressing real-world applications. However, a critical hurdle in deploying LLMs for practical decision-making is their inability to provide reliable, quantitative probabilities. While task-specific fine-tuning of LLMs using traditional discriminative objectives (similar to encoder-only models) can yield probability estimates, this often leads to catastrophic forgetting and linguistic collapse. Consequently, the model loses its ability to generate explanations, severely undermining its interpretability and usability. To address this challenge, we propose CLSGen, a novel LLM fine-tuning framework designed for binary classification tasks. The CLSGen framework encompasses a new model architecture, training methodology, and data construction strategy to enable robust probability estimation without sacrificing the model's inherent explanation-generation capabilities. Experimental results across multiple benchmark datasets demonstrate that models fine-tuned with CLSGen outperform existing baselines in classification metrics (AUROC and F1-score). Regarding explanation, the results showed strong alignment between predicted labels and generated justifications, as well as high readability.

Problem

• Large Language Models
• Probability Estimation
• Verbalized Explanation
• Catastrophic Forgetting
• Interpretability

Innovation

• dual-head fine-tuning
• probabilistic classification
• verbalized explanation
• catastrophic forgetting mitigation
• large language models