Grammar-Guided Hierarchical Parsing for Long-form Audio Activity Recognition

2026-06-26

AI Summary

This work addresses the limitations of existing approaches in long-form audio activity recognition, which suffer from inconsistent cross-level modeling and reliance on multi-level supervisory labels. The authors reformulate the task as a hierarchical parsing problem grounded in event-level evidence, introducing a hierarchical activity grammar to enforce compositional structure and temporal ordering constraints. They further propose a grammar-guided dynamic programming decoding mechanism that, using only event-level detection posteriors, enables end-to-end generation of temporally coherent and semantically interpretable activity–sub-activity–event parse trees—without requiring supervision from high-level activity or sub-activity annotations. Evaluated on the MultiAct dataset, the method achieves substantial improvements in temporal consistency, as measured by Edit Score, while enabling explainable inference of hierarchical activity structures.

Abstract

Long-form audio exhibits an inherent hierarchy: fine-grained events form sub-activities, which in turn constitute higher-level activities. Prior work often models these levels separately, leading to cross-level inconsistencies and requiring supervision at multiple levels. We formulate the problem as hierarchical parsing from event-level evidence: given detected event segments with class posteriors, we infer an order-consistent Act-Sub-Event parse tree. We propose Hierarchical Activity Grammar, encoding hierarchical composition and temporal-order constraints, and perform grammar-guided decoding that combines event evidence with a grammar prior. This yields a temporally grounded parse tree from which sub-activity segmentation and activity classification are derived, without requiring sub-activity or activity labels for training. Experiments on the long-form MultiAct audio dataset demonstrate improved temporal-order consistency (Edit score) and produces interpretable hierarchies.

Problem

• long-form audio
• hierarchical parsing
• activity recognition
• temporal consistency
• cross-level inconsistency

Innovation

• hierarchical parsing
• grammar-guided decoding
• long-form audio activity recognition
• temporal-order consistency
• unsupervised hierarchy inference