Distilling Empathy from Large Language Models

2025-07-10

AI Summary

This study addresses the effective transfer of empathic capability from large language models (LLMs) to small language models (SLMs) via knowledge distillation, aiming to enhance empathetic dialogue performance on resource-constrained devices (e.g., smartphones). We propose a two-stage fine-tuning framework: (1) supervised fine-tuning on high-quality empathic dialogues generated by an LLM; and (2) structured empathic prompting—introducing four prompt categories to explicitly guide the SLM in emotion recognition, stance understanding, response adaptation, and expressive warmth. Compared to conventional distillation methods, our approach achieves a 90% win rate in empathic response generation—a 10-percentage-point improvement—and significantly outperforms direct prompting baselines. The core contribution lies in decoupling empathy into prompt-driven, distillable behavioral components and establishing the first dedicated training paradigm for empathic knowledge distillation targeting SLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Emotional Intelligence

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The distillation of knowledge from Large Language Models (LLMs) into Smaller Language Models (SLMs), preserving the capabilities and performance of LLMs while reducing model size, has played a key role in the proliferation of LLMs. Because SLMs are considerably smaller than LLMs, they are often utilized in domains where human interaction is frequent but resources are highly constrained, e.g., smart phones. Therefore, it is crucial to ensure that empathy, a fundamental aspect of positive human interactions, already instilled into LLMs, is retained by SLMs after distillation. In this paper, we develop a comprehensive approach for effective empathy distillation from LLMs into SLMs. Our approach features a two-step fine-tuning process that fully leverages datasets of empathetic dialogue responses distilled from LLMs. We explore several distillation methods beyond basic direct prompting and propose four unique sets of prompts for targeted empathy improvement to significantly enhance the empathy distillation process. Our evaluations demonstrate that SLMs fine-tuned through the two-step fine-tuning process with distillation datasets enhanced by the targeted empathy improvement prompts significantly outperform the base SLM at generating empathetic responses with a win rate of 90%. Our targeted empathy improvement prompts substantially outperform the basic direct prompting with a 10% improvement in win rate.

Problem

• Distill empathy from LLMs to SLMs effectively
• Enhance SLMs' empathetic response generation capability
• Optimize distillation methods for empathy retention

Innovation

• Two-step fine-tuning for empathy distillation
• Targeted empathy improvement prompts
• Leveraging empathetic dialogue datasets