Dynamic Group Detection using VLM-augmented Temporal Groupness Graph

2025-09-04

AI Summary

This work addresses the problem of detecting dynamically evolving human crowds in video, overcoming the limitation of existing methods that assume static crowd configurations. The proposed method introduces a temporal modeling framework that jointly leverages vision-language model (VLM)-enhanced local appearance features and global scene context. Specifically, CLIP is employed to extract semantically robust frame-level crowdness features; a cross-frame crowdness graph is then constructed to explicitly model member identity consistency and temporal structural evolution; finally, graph optimization enforces globally temporally consistent crowd partitioning. Evaluated on multiple public benchmarks, the approach achieves significant improvements over state-of-the-art methods and, for the first time, enables high-precision detection of complex dynamic crowd behaviors—including splitting, merging, and reconfiguration. This work establishes a novel paradigm for social behavior analysis in video understanding.

Technology Category

• Computer Vision: Language and Vision
• Humans and AI: Crowd Sourcing and Human Computation
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This paper proposes dynamic human group detection in videos. For detecting complex groups, not only the local appearance features of in-group members but also the global context of the scene are important. Such local and global appearance features in each frame are extracted using a Vision-Language Model (VLM) augmented for group detection in our method. For further improvement, the group structure should be consistent over time. While previous methods are stabilized on the assumption that groups are not changed in a video, our method detects dynamically changing groups by global optimization using a graph with all frames' groupness probabilities estimated by our groupness-augmented CLIP features. Our experimental results demonstrate that our method outperforms state-of-the-art group detection methods on public datasets. Code: https://github.com/irajisamurai/VLM-GroupDetection.git

Problem

• Detecting dynamically changing human groups in videos
• Integrating local appearance and global context features
• Ensuring temporal consistency across video frames

Innovation

• VLM-augmented groupness features extraction
• Temporal graph optimization for dynamic groups
• Global context integration with local features