LaDy: Lagrangian-Dynamic Informed Network for Skeleton-based Action Segmentation via Spatial-Temporal Modulation

2026-03-25

AI Summary

This work addresses the limitations of existing skeleton-based action segmentation methods, which often neglect the physical dynamics of human motion and consequently struggle to distinguish actions with similar kinematics but differing dynamic intentions, while also exhibiting imprecise boundary localization. To overcome these issues, this study is the first to explicitly incorporate Lagrangian dynamics into the task by modeling generalized coordinates and generalized forces, and introduces an energy consistency loss to enforce physical plausibility. Furthermore, a dynamics-driven spatiotemporal modulation mechanism is proposed to enhance both action semantic discriminability and boundary awareness. Extensive experiments on multiple challenging benchmarks demonstrate state-of-the-art performance, validating the effectiveness of integrating physical dynamics for improving segmentation accuracy.

Technology Category

• Computer Vision: Segmentation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Skeleton-based Temporal Action Segmentation (STAS) aims to densely parse untrimmed skeletal sequences into frame-level action categories. However, existing methods, while proficient at capturing spatio-temporal kinematics, neglect the underlying physical dynamics that govern human motion. This oversight limits inter-class discriminability between actions with similar kinematics but distinct dynamic intents, and hinders precise boundary localization where dynamic force profiles shift. To address these, we propose the Lagrangian-Dynamic Informed Network (LaDy), a framework integrating principles of Lagrangian dynamics into the segmentation process. Specifically, LaDy first computes generalized coordinates from joint positions and then estimates Lagrangian terms under physical constraints to explicitly synthesize the generalized forces. To further ensure physical coherence, our Energy Consistency Loss enforces the work-energy theorem, aligning kinetic energy change with the work done by the net force. The learned dynamics then drive a Spatio-Temporal Modulation module: Spatially, generalized forces are fused with spatial representations to provide more discriminative semantics. Temporally, salient dynamic signals are constructed for temporal gating, thereby significantly enhancing boundary awareness. Experiments on challenging datasets show that LaDy achieves state-of-the-art performance, validating the integration of physical dynamics for action segmentation. Code is available at https://github.com/HaoyuJi/LaDy.

Problem

• Skeleton-based Action Segmentation
• Lagrangian Dynamics
• Physical Dynamics
• Temporal Boundary Localization
• Action Discriminability

Innovation

• Lagrangian dynamics
• skeleton-based action segmentation
• physical consistency
• spatio-temporal modulation
• generalized forces