Problem
Research questions and friction points this paper is trying to address.
Fine-Grained Action Recognition
Human Action Recognition
Visual-Pose Fusion
Action Differentiation
Fine-Grained Cues
Fine-Grained Action Recognition with Cross-Attentive Latent Sparse Experts
🤖 AI Summary
This work addresses the challenge of fine-grained human action recognition, where visual similarity among actions impedes discriminability. To this end, we propose a structured multimodal fusion framework that jointly models three complementary representations: RGB appearance, pose heatmap geometry, and skeletal graph topology. Our approach employs pairwise cross-attention to enable symmetric inter-stream interaction and introduces a stream-wise latent sparse Mixture-of-Experts (MoE) mechanism that dynamically routes inputs to a shared subset of experts based on content, augmented with load-balancing regularization. Notably, our method achieves state-of-the-art performance on Gym99, Gym288, and Diving48 without relying on textual supervision or large-scale vision-language pretraining. On the long-tailed Gym288 benchmark, it improves mean class accuracy by 7.6 percentage points, from 68.6% to 76.2%.
📝 Abstract
Fine-grained human action recognition (FHAR) must distinguish visually similar actions that differ mainly in body configuration, timing, or local appearance. RGB representations retain visual context but often suppress joint-level geometry, whereas skeleton representations encode kinematics but discard dense spatial detail. We introduce FineX, which factorizes fine-grained cues into RGB appearance, pose heatmap geometry, and skeletal-graph topology. Pairwise cross-attention enables symmetric, stream-preserving information exchange, followed by a streamwise latent sparse Mixture-of-Experts that routes each representation to a content-dependent subset of shared experts, regularized by a load-balancing objective. FineX achieves state-of-the-art results on Gym99, Gym288, and Diving48. On the long-tailed Gym288, it raises mean class accuracy from 68.6% to 76.2% (+7.6 points) without textual supervision or large-scale vision-language pre-training, demonstrating the benefit of structured visual-pose-graph fusion and conditional expert refinement for FHAR.
Innovation
Methods, ideas, or system contributions that make the work stand out.
fine-grained action recognition
cross-attention
latent sparse mixture-of-experts
pose heatmap
skeletal graph
🔎 Similar Papers
No similar papers found.
