TR-LLM: Integrating Trajectory Data for Scene-Aware LLM-Based Human Action Prediction

2024-10-05

arXiv.org

AI Summary

To address degraded behavioral prediction performance under real-world occlusion and partial observability, this paper proposes a trajectory-augmented multimodal prediction framework that mitigates inherent limitations of large language models (LLMs) in spatial perception and physical constraint modeling. The method explicitly incorporates fine-grained human trajectories as geometric priors into the LLM’s action reasoning pipeline: a trajectory encoder extracts spatiotemporal constraints, while instruction tuning and physics-guided decoding jointly align linguistic semantics with motion priors. Evaluated on benchmarks including HomeAction, the framework achieves a 23.6% improvement in action prediction accuracy and boosts F1-score by over 31% under occluded or low-information conditions. These gains demonstrate substantially enhanced robustness and interpretability in partially observable scenarios, establishing the first integration of explicit trajectory-based physical priors into LLM-driven behavioral forecasting.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Accurate prediction of human behavior is crucial for AI systems to effectively support real-world applications, such as autonomous robots anticipating and assisting with human tasks. Real-world scenarios frequently present challenges such as occlusions and incomplete scene observations, which can compromise predictive accuracy. Thus, traditional video-based methods often struggle due to limited temporal and spatial perspectives. Large Language Models (LLMs) offer a promising alternative. Having been trained on a large text corpus describing human behaviors, LLMs likely encode plausible sequences of human actions in a home environment. However, LLMs, trained primarily on text data, lack inherent spatial awareness and real-time environmental perception. They struggle with understanding physical constraints and spatial geometry. Therefore, to be effective in a real-world spatial scenario, we propose a multimodal prediction framework that enhances LLM-based action prediction by integrating physical constraints derived from human trajectories. Our experiments demonstrate that combining LLM predictions with trajectory data significantly improves overall prediction performance. This enhancement is particularly notable in situations where the LLM receives limited scene information, highlighting the complementary nature of linguistic knowledge and physical constraints in understanding and anticipating human behavior.

Problem

• Enhance human action prediction using LLMs and trajectory data.
• Address challenges like occlusions and incomplete scene observations.
• Combine linguistic knowledge with physical constraints for better predictions.

Innovation

• Integrates trajectory data with LLMs
• Enhances LLM predictions using physical constraints
• Improves action prediction in limited scene information