Policy-to-Language: Train LLMs to Explain Decisions with Flow-Matching Generated Rewards

2025-02-18

AI Summary

Addressing the dual challenges of insufficient agent decision interpretability and high reward annotation cost in human–AI coexistence scenarios, this paper proposes a model-agnostic natural language explanation generation framework. Methodologically, it pioneers the deep integration of flow-matching generative models with latent representations of large language models (LLMs), explicitly embedding linguistic explanation cues into the reward modeling process to enable semantically aligned, dense reward auto-generation. Crucially, it operates without human reward annotations and jointly optimizes explanation generation and reinforcement learning objectives end-to-end. Empirically, the approach significantly improves explanation plausibility and faithfulness across diverse RL and LLM benchmarks, while simultaneously enhancing downstream task performance. It demonstrates strong generalization capability and training efficiency, offering a scalable solution for interpretable, annotation-free reward learning.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

As humans increasingly share environments with diverse agents powered by RL, LLMs, and beyond, the ability to explain their policies in natural language will be vital for reliable coexistence. In this paper, we build a model-agnostic explanation generator based on an LLM. The technical novelty is that the rewards for training this LLM are generated by a generative flow matching model. This model has a specially designed structure with a hidden layer merged with an LLM to harness the linguistic cues of explanations into generating appropriate rewards. Experiments on both RL and LLM tasks demonstrate that our method can generate dense and effective rewards while saving on expensive human feedback; it thus enables effective explanations and even improves the accuracy of the decisions in original tasks.

Problem

• Develop model-agnostic explanation generator using LLM
• Generate rewards via flow matching for training
• Enhance decision accuracy and reduce human feedback

Innovation

• LLM-based explanation generator
• Flow-matching generated rewards
• Model-agnostic decision explanation