Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization

2025-10-08

AI Summary

Action Quality Assessment (AQA) faces dual challenges in real-world scenarios: poor generalization and catastrophic forgetting, primarily caused by non-stationary quality distribution shifts. To address continual AQA, we propose MAGR++, an adaptive manifold alignment graph regularization framework. MAGR++ employs full-parameter fine-tuning, a two-stage feature correction mechanism, and a learnable manifold projector that maps historical features into the current embedding space. Coupled with graph regularization, it jointly aligns both local neighborhood structures and global distribution manifolds—stabilizing shallow representations while enhancing deep feature adaptability. Evaluated on four newly established Continual AQA (CAQA) benchmarks, MAGR++ achieves substantial improvements: +3.6% average correlation over state-of-the-art methods in offline evaluation and +12.2% in online continual learning. To our knowledge, MAGR++ is the first method to systematically resolve feature manifold shift and overfitting in continual AQA.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Life-Long and Continual Learning
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Action Quality Assessment (AQA) quantifies human actions in videos, supporting applications in sports scoring, rehabilitation, and skill evaluation. A major challenge lies in the non-stationary nature of quality distributions in real-world scenarios, which limits the generalization ability of conventional methods. We introduce Continual AQA (CAQA), which equips AQA with Continual Learning (CL) capabilities to handle evolving distributions while mitigating catastrophic forgetting. Although parameter-efficient fine-tuning of pretrained models has shown promise in CL for image classification, we find it insufficient for CAQA. Our empirical and theoretical analyses reveal two insights: (i) Full-Parameter Fine-Tuning (FPFT) is necessary for effective representation learning; yet (ii) uncontrolled FPFT induces overfitting and feature manifold shift, thereby aggravating forgetting. To address this, we propose Adaptive Manifold-Aligned Graph Regularization (MAGR++), which couples backbone fine-tuning that stabilizes shallow layers while adapting deeper ones with a two-step feature rectification pipeline: a manifold projector to translate deviated historical features into the current representation space, and a graph regularizer to align local and global distributions. We construct four CAQA benchmarks from three datasets with tailored evaluation protocols and strong baselines, enabling systematic cross-dataset comparison. Extensive experiments show that MAGR++ achieves state-of-the-art performance, with average correlation gains of 3.6% offline and 12.2% online over the strongest baseline, confirming its robustness and effectiveness. Our code is available at https://github.com/ZhouKanglei/MAGRPP.

Problem

• Addresses evolving quality distributions in action assessment
• Mitigates catastrophic forgetting during continual learning updates
• Aligns feature manifolds to stabilize representation learning

Innovation

• Full-parameter fine-tuning for effective representation learning
• Adaptive manifold projection to rectify feature deviations
• Graph regularization aligning local and global distributions