evTransFER: A Transfer Learning Framework for Event-based Facial Expression Recognition

2025-08-05

AI Summary

To address low accuracy and challenging temporal modeling in event-camera-based facial expression recognition (FER), this paper proposes evTransFER. First, it leverages a face reconstruction pre-trained GAN encoder for cross-task transfer learning. Second, it introduces Temporal-Intensity Encoding (TIE), a novel event representation that explicitly encodes both temporal and intensity information from event streams. Third, it designs the TIE-LSTM architecture to efficiently model long-range spatiotemporal dynamics. Evaluated on the e-CK+ dataset, evTransFER achieves 93.6% accuracy—outperforming the state-of-the-art by 25.9 percentage points. Key contributions are: (i) the first application of reconstruction-pretrained encoders for transfer learning in event-based FER; (ii) the TIE representation, uniquely balancing high temporal resolution with intensity-aware encoding; and (iii) a new paradigm for high-accuracy, computationally efficient FER in the event-domain.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Event-based cameras are bio-inspired vision sensors that asynchronously capture per-pixel intensity changes with microsecond latency, high temporal resolution, and high dynamic range, providing valuable information about the spatio-temporal dynamics of the scene. In the present work, we propose evTransFER, a transfer learning-based framework and architecture for face expression recognition using event-based cameras. The main contribution is a feature extractor designed to encode the spatio-temporal dynamics of faces, built by training an adversarial generative method on a different problem (facial reconstruction) and then transferring the trained encoder weights to the face expression recognition system. We show that this proposed transfer learning method greatly improves the ability to recognize facial expressions compared to training a network from scratch. In addition, we propose an architecture that incorporates an LSTM to capture longer-term facial expression dynamics, and we introduce a new event-based representation, referred to as TIE, both of which further improve the results. We evaluate the proposed framework on the event-based facial expression database e-CK+ and compare it to state-of-the-art methods. The results show that the proposed framework evTransFER achieves a 93.6% recognition rate on the e-CK+ database, significantly improving the accuracy (25.9% points or more) when compared to state-of-the-art performance for similar problems.

Problem

• Develops transfer learning for event-based facial expression recognition
• Encodes spatio-temporal dynamics via adversarial generative training
• Improves accuracy with LSTM and new event representation TIE

Innovation

• Transfer learning for event-based facial expression recognition
• Adversarial generative method for facial reconstruction
• LSTM and TIE representation for dynamic expression capture