Faithful Action-unit Causal Reasoning for Counterfactually Faithful Emotion Explanations

2026-06-14

AI Summary

This work addresses the limited causal fidelity of action units (AUs) in existing facial emotion interpretation models, which often fail to ensure that AUs genuinely drive predictions. The authors formulate AU-to-emotion reasoning as a counterfactual consistency problem grounded in a predefined causal graph \( G \), employing do-intervention training: if an AU is causally relevant to an emotion class in \( G \), intervening on it should alter the prediction; otherwise, the prediction should remain invariant. They introduce a trainable and quantifiable objective for optimizing counterfactual fidelity, enabling, for the first time, strict adherence to a prescribed causal structure in emotion explanations. This framework further extends to controllable fidelity in language-based explanations. Experiments show that on the UNBC-PAIN dataset, alignment between invoked AUs and the PSPI pain structure improves from 0.08 to 0.57, while cross-dataset emotion tasks exhibit increased fidelity to causal graph \( G \), rising from 0.50 to 0.84, demonstrating the constructibility of faithful language explanations.

Abstract

Multimodal models can name the action units (AUs) behind a facial emotion, but their AU->emotion rationales are typically plausible rather than faithful: nothing forces the AUs a model invokes to be the AUs that actually drive its prediction. We cast AU->emotion reasoning as a counterfactual-consistency problem between the rationale, the label, and a structural AU->emotion causal graph G, and propose FACR, which grounds the reasoner in an independently induced, polarity-aware G and trains a counterfactual-faithfulness objective: a do-intervention on an AU that G marks causal for a class must move the prediction, while one it marks irrelevant must leave it unchanged. Faithfulness is thereby both trainable and measurable through a matching interventional metric, which we evaluate against a known causal structure, the PSPI pain-AU composition, as no existing affective-reasoning benchmark allows. We are explicit that this metric tests fidelity to the supplied structure rather than its rediscovery: it asks whether the trained reasoner invokes the AUs the structure marks causal, on held-out subjects and a second dataset. Under subject-independent evaluation on UNBC-PAIN, the objective raises the agreement between the invoked AUs and the PSPI composition from a no-objective baseline of 0.08 to 0.57, at a small detection cost; an unfaithfulness control attributes the gain to the objective. On a cross-dataset emotion transfer, the objective likewise raises fidelity to G on a seven-class task (0.50 to 0.84). Finally, we attach a language verbalizer and extend the audit to the generated text: biasing each action unit's emission by its latent activation makes the rationale faithful by construction, so that ablating an AU removes it from the explanation, a property that transfers to a second language-model backbone, whereas a freely generated rationale is unfaithful.

Problem

• action units
• emotion explanation
• causal reasoning
• counterfactual faithfulness
• multimodal models

Innovation

• counterfactual faithfulness
• causal reasoning
• action units
• emotion explanation
• interventional evaluation