Tutor-Student Reinforcement Learning: A Dynamic Curriculum for Robust Deepfake Detection

2026-03-25

AI Summary

Standard supervised training treats all samples equally in deepfake detection, hindering the learning of robust and generalizable features. To address this limitation, this work proposes a Teacher-Student Reinforcement Learning (TSRL) framework that, for the first time, formulates dynamic curriculum learning as a Markov Decision Process. In this framework, a teacher agent constructs state representations by integrating sample-wise historical dynamics—such as exponentially moving averaged (EMA) loss and forgetting counts—with visual features, and adaptively adjusts per-sample loss weights through Proximal Policy Optimization (PPO) in a continuous action space. By prioritizing high-value samples, TSRL significantly enhances model generalization to unseen forgery techniques, demonstrating the efficacy of reinforcement learning–driven adaptive curriculum design in deepfake detection.

Technology Category

• Machine Learning: Reinforcement Learning
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Standard supervised training for deepfake detection treats all samples with uniform importance, which can be suboptimal for learning robust and generalizable features. In this work, we propose a novel Tutor-Student Reinforcement Learning (TSRL) framework to dynamically optimize the training curriculum. Our method models the training process as a Markov Decision Process where a ``Tutor'' agent learns to guide a ``Student'' (the deepfake detector). The Tutor, implemented as a Proximal Policy Optimization (PPO) agent, observes a rich state representation for each training sample, encapsulating not only its visual features but also its historical learning dynamics, such as EMA loss and forgetting counts. Based on this state, the Tutor takes an action by assigning a continuous weight (0-1) to the sample's loss, thereby dynamically re-weighting the training batch. The Tutor is rewarded based on the Student's immediate performance change, specifically rewarding transitions from incorrect to correct predictions. This strategy encourages the Tutor to learn a curriculum that prioritizes high-value samples, such as hard-but-learnable examples, leading to a more efficient and effective training process. We demonstrate that this adaptive curriculum improves the Student's generalization capabilities against unseen manipulation techniques compared to traditional training methods. Code is available at https://github.com/wannac1/TSRL.

Problem

• deepfake detection
• curriculum learning
• robustness
• generalization
• supervised training

Innovation

• Tutor-Student Reinforcement Learning
• Dynamic Curriculum Learning
• Deepfake Detection
• Proximal Policy Optimization
• Adaptive Sample Weighting