Gen2Balance: Generative Balancing for Long-Tailed Video Action Recognition

2026-06-21

AI Summary

This work addresses the challenge of poor recognition performance on rare actions in video action recognition due to long-tailed data distributions. To mitigate this issue, the authors propose the first approach that leverages text-to-video generative models for data balancing. Specifically, they construct action-semantic-guided, diverse textual prompts to synthesize videos and introduce a two-stage training strategy to alleviate domain shift between real and generated data. Remarkably, the method achieves substantial performance gains with only partial data balancing while significantly reducing computational overhead. On the UCF-LT and K100-LT benchmarks, it outperforms the current best baselines by 5.1% and 7.0%, respectively, and yields a striking 31.9% improvement on rare action categories in RareAct. Notably, it attains 79% of the full performance gain at merely 27% of the computational cost.

Abstract

We address the problem of training on long-tailed data for video action recognition. We propose to augment the training set using a text-to-video generative model, conditioned on diverse text prompts grounded in action profiles and training exemplars. Our approach, called Gen2Balance, converts an imbalanced training set into a balanced combination of real and generated video clips. To effectively learn from such data, we employ a two-stage training strategy that mitigates domain shift and yields significant improvements. We evaluate on long-tailed versions of standard benchmarks: UCF-101 (UCF-LT) and a 100-class subset of Kinetics (K100-LT) selected to prioritise temporally challenging actions. Gen2Balance improves accuracy over the strongest baselines for long-tailed learning by 5.1% and 7.0% on the respective datasets. On rare actions from the RareAct dataset (e.g., cut keyboard), Gen2Balance improves accuracy by 31.9%, demonstrating effectiveness for scarce actions. By varying the amount of synthetic data added, we show that partial balancing already achieves 79% of the performance gains at 27% of the compute cost on K100-LT, highlighting the practical scalability of Gen2Balance.

Problem

• long-tailed
• video action recognition
• data imbalance
• rare actions

Innovation

• text-to-video generation
• long-tailed learning
• action recognition
• data balancing
• domain shift mitigation