Decision-Aware Uncertainty Evaluation of Vision-Language Model-Based Early Action Anticipation for Human-Robot Interaction

2026-03-09

AI Summary

This study addresses the risk in human-robot collaboration arising from unreliable uncertainty estimation in vision-language models (VLMs) during early action prediction under partial and ambiguous observations. The work presents the first uncertainty evaluation framework tailored for VLM-based early action prediction in human-robot interaction, introducing a temporal-prefix-based calibration protocol and selective prediction metrics. It systematically uncovers calibration biases and failure modes inherent in VLMs when operating with incomplete visual input. Experimental results provide critical evidence regarding the reliability of VLM uncertainty estimates in short-duration action recognition, thereby establishing a foundation for confidence-gated, trustworthy decision-making in interactive robotic systems.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Intelligent Robots: Human-Robot Interaction
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Robots in shared workspaces must interpret human actions from partial, ambiguous observations, where overconfident early predictions can lead to unsafe or disruptive interaction. This challenge is amplified in egocentric views, where viewpoint changes and occlusions increase perceptual noise and ambiguity. As a result, downstream human-robot interaction modules require not only an action hypothesis but also a trustworthy estimate of confidence under partial observation. Recent vision-language model-based approaches have been proposed for short-term action recognition due to their open-vocabulary and context-aware reasoning, but their uncertainty reliability in the temporal-prefix regime is largely uncharacterized. We present the first systematic evaluation of uncertainty in vision-language model-based short-term action recognition for human-robot interaction. We introduce a temporal-prefix evaluation protocol and metrics for calibration and selective prediction. We also characterize miscalibration patterns and failure modes under partial observations. Our study provides the missing reliability evidence needed to use vision-language model predictions in confidence-gated human-robot interaction modules.

Problem

• uncertainty evaluation
• vision-language models
• early action anticipation
• human-robot interaction
• partial observation

Innovation

• vision-language models
• uncertainty evaluation
• early action anticipation
• temporal-prefix calibration
• human-robot interaction