Task-Specific Feature Fusion Method for Multi-Task Affective Behavior Analysis

2026-07-15

AI Summary

This work addresses the performance limitations of unified models in multi-task affective behavior analysis, where discrepancies among valence-arousal regression, expression classification, and action unit detection hinder joint optimization. To overcome this, the authors propose a task-adaptive feature fusion strategy that leverages complementary frame-level features extracted from frozen DINOv2 ViT-L and DINOv3 ConvNeXt-base backbones. Each task is equipped with a dedicated prediction head and tailored fusion mechanism—such as gated or residual fusion—to avoid enforced architectural sharing. The framework further integrates temporal convolutions, LightGBM-based post-processing, and threshold calibration. Evaluated on the ABAW11 validation set, the method achieves an EXPR macro F1 of 0.4222, AU macro F1 of 0.5402, and VA mean CCC of 0.6717, yielding a total score of 1.6341 and demonstrating substantial improvement in multi-task collaborative performance.

Abstract

The 11th Affective Behavior Analysis in-the-wild (ABAW11) Multi-Task Learning Challenge requires a unified system to predict valence-arousal, categorical expressions, and facial action units from the official s-Aff-Wild2 images. Although these tasks are naturally related through facial behavior, our validation experiments show that they benefit from different visual features, temporal processing strategies, fusion mechanisms, and calibration procedures. In this paper, we study task-adaptive feature fusion for ABAW11 multi-task affective behavior analysis. We first adapt two pretrained visual backbones, DINOv2 ViT-L and DINOv3 ConvNeXt-base, on an external expression-oriented facial image set and then freeze them to extract complementary frame-level features from the official ABAW11 data. On top of these frozen features, we systematically compare frame-level prediction heads, temporal convolutional heads, post-hoc temporal smoothing, LightGBM models, feature concatenation, gated fusion, residual fusion, late logit fusion, threshold calibration, and shared MTL structures. The final system selects task-specific fusion and prediction strategies rather than forcing all tasks to share a single architecture. On the ABAW11 validation set, the selected system achieves an EXPR macro-F1 of 0.4222, an AU macro-F1 of 0.5402, and a mean VA CCC of 0.6717, resulting in an overall validation score of 1.6341. The results suggest that task-adaptive fusion of frozen visual features is a simple and effective strategy for ABAW-style multi-task affective behavior analysis.

Problem

• multi-task learning
• affective behavior analysis
• feature fusion
• facial expression recognition
• action unit detection

Innovation

• task-adaptive fusion
• multi-task learning
• frozen feature extraction
• affective behavior analysis
• feature fusion