HieroAction: Hierarchically Guided VLM for Fine-Grained Action Analysis

2025-08-23

AI Summary

Existing action assessment methods typically produce a single scalar score, lacking interpretable reasoning and failing to meet real-world demands for fine-grained analysis in domains such as sports science, clinical rehabilitation, and robotics. To address this, we propose a structured action assessment framework. First, we design a stepwise action reasoning mechanism that enables progressive evaluation—from holistic action recognition to fine-grained sub-action decomposition. Second, we introduce hierarchical policy learning that synergistically integrates vision-language models, domain-customized chain-of-thought prompting, and reinforcement learning–based reward optimization to support multi-level semantic modeling. Evaluated on multiple benchmark datasets, our method significantly improves both scoring accuracy and interpretability. It is the first approach to achieve high-accuracy, fine-grained human action assessment with fully transparent, traceable reasoning—thereby enabling trustworthy, decision-ready insights for critical application domains.

Technology Category

• Knowledge Representation and Reasoning: Action, Change, and Causality
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Evaluating human actions with clear and detailed feedback is important in areas such as sports, healthcare, and robotics, where decisions rely not only on final outcomes but also on interpretable reasoning. However, most existing methods provide only a final score without explanation or detailed analysis, limiting their practical applicability. To address this, we introduce HieroAction, a vision-language model that delivers accurate and structured assessments of human actions. HieroAction builds on two key ideas: (1) Stepwise Action Reasoning, a tailored chain of thought process designed specifically for action assessment, which guides the model to evaluate actions step by step, from overall recognition through sub action analysis to final scoring, thus enhancing interpretability and structured understanding; and (2) Hierarchical Policy Learning, a reinforcement learning strategy that enables the model to learn fine grained sub action dynamics and align them with high level action quality, thereby improving scoring precision. The reasoning pathway structures the evaluation process, while policy learning refines each stage through reward based optimization. Their integration ensures accurate and interpretable assessments, as demonstrated by superior performance across multiple benchmark datasets. Code will be released upon acceptance.

Problem

• Evaluating human actions with detailed feedback in sports, healthcare, and robotics
• Providing interpretable reasoning beyond final action scores
• Addressing lack of structured analysis in existing assessment methods

Innovation

• Stepwise Action Reasoning for interpretable assessment
• Hierarchical Policy Learning for fine-grained dynamics
• Reinforcement learning for reward-based optimization