TraMP-LLaMA: Generative Interpretability with Decoupled Instruction Tuning for Facial Expression Quality Assessment

2026-06-25

AI Summary

Existing methods for facial expression quality assessment provide only severity scores without explicitly articulating the observable facial motion evidence supporting those predictions, resulting in limited interpretability. To address this gap, this work proposes TraMP-LLaMA—the first multimodal framework to integrate generative interpretability into this task—by fusing RGB appearance and facial landmark trajectories. Through a decoupled instruction-tuning strategy, the model simultaneously predicts severity scores and generates structured textual reports. We introduce PFED5-plus, a new dataset augmented with expert-annotated motion descriptions. Experimental results demonstrate that our approach achieves at least a 4.39% improvement in Spearman’s rank correlation coefficient over the strongest baseline on PFED5-plus, while also producing the highest-quality explanatory reports.

Abstract

Existing facial expression quality assessment (FEQA) methods typically produce only a severity score, without explicitly communicating the observable facial motion evidence that supports the prediction. This limits interpretability and makes it difficult to inspect the basis of model outputs in Parkinson's disease assessment. To address this gap, we propose TraMP-LLaMA, a unified multimodal framework that jointly predicts severity scores and generates structured textual reports from facial motion cues. The framework integrates RGB appearance and landmark trajectory cues, and adopts a decoupled instruction-tuning strategy to reduce task interference between severity prediction and language generation. To support this task, we further extend the PFED5 dataset with expert-guided textual motion descriptions and construct PFED5-plus. Experiments on PFED5-plus show that TraMP-LLaMA outperforms competitive video-language baselines in report generation and achieves the best severity prediction performance among the compared methods under joint multi-expression training, improving Spearman's rank correlation by at least 4.39 percent over all competing methods. The text annotations and code are available at https://github.com/shuchaoduan/TraMP-LLaMA.

Problem

• facial expression quality assessment
• interpretability
• Parkinson's disease
• severity scoring
• facial motion cues

Innovation

• decoupled instruction tuning
• facial expression quality assessment
• multimodal interpretability
• structured report generation
• landmark trajectory