SafeGesture: Evaluating Fine-Grained Hand Gesture Understanding in Vision-Language Models through Scenario-Conditioned Safety Interpretation

2026-08-17

AI Summary

This study addresses the insufficient fine-grained gesture understanding and reasoning capabilities of Vision-Language Models (VLMs) in safety-critical scenarios. We introduce SafeGesture, a benchmark comprising 4,800 gesture-scene paired samples, alongside a scene-conditioned safety interpretation evaluation framework. Our analysis reveals that safety reasoning, rather than gesture recognition, constitutes the primary bottleneck, characterized by significant label bias and uncertainty. Evaluations demonstrate that although visual inputs substantially enhance performance, mainstream models such as GPT-4o achieve a safety accuracy of only 53.3%, with no model exceeding 56.2%. These findings underscore the critical limitations of current VLMs in aligning perception with reasoning for safety-critical gesture interpretation.

Abstract

Open-weight and frontier vision-language models (VLMs) perform well on general image understanding, but their ability to interpret fine-grained hand gestures in safety-critical operational contexts remains largely unexamined. We introduce SafeGesture, a benchmark that evaluates whether a model can infer scenario-appropriate safety actions from hand gestures. It pairs six HaGRID gestures with eight operational scenarios for 4,800 items and evaluates Qwen2.5-VL-7B, LLaVA-NeXT-7B, InternVL2-8B, Phi-3.5-Vision, and GPT-4o. Results reveal a perception-reasoning decoupling: GPT-4o achieves 98.4% gesture accuracy but 53.3% safety accuracy, while Qwen2.5-VL reaches 84.9% and 39.5%, yielding gaps of 45.0 and 45.4 percentage points. Four of five models rarely or never use the uncertainty label, and failure directions differ substantially across models. Accuracy also obscures label bias: a scenario-majority policy with no visual input reaches 58.3%, above every evaluated model, while only GPT-4o exceeds this prior under macro-F1. Visual input improves safety accuracy by 11.2 to 30.2 percentage points, but providing the ground-truth gesture as text improves performance by only 0.4 to 3.2 points, and no model exceeds 56.2%. These results indicate that the main bottleneck is scenario-conditioned safety reasoning rather than gesture recognition.

Problem

• Vision-Language Models
• Hand Gesture Understanding
• Safety Reasoning
• Scenario-Conditioned Interpretation
• Fine-Grained Evaluation

Innovation

• SafeGesture benchmark
• scenario-conditioned safety reasoning
• perception-reasoning decoupling
• fine-grained hand gesture understanding
• vision-language model evaluation