Semantic-Metric Bayesian Risk Fields: Learning Robot Safety from Human Videos with a VLM Prior

2025-12-08

AI Summary

This work addresses the challenge of modeling humans’ continuous, context- and space-dependent perception of safety risk for robotic systems. We propose a semantic-metric joint Bayesian risk field framework that integrates commonsense priors from pre-trained vision-language models (VLMs) with a learnable, pixel-wise likelihood network based on Vision Transformers (ViT). Given RGB images and object-level text descriptions as input, the framework generates interpretable, dense 2D/3D risk maps. Our key contribution is the first incorporation of VLM-derived priors into Bayesian risk modeling, enabling explainable risk estimation, cross-scenario generalization, and online adaptation. Experiments demonstrate that the generated risk fields accurately capture human safety preferences across downstream tasks—including visual-motor planning, value prediction, and trajectory optimization—highlighting strong practical utility and broad scalability potential.

Technology Category

• Natural Language Processing: Safety and Robustness
• Computer Vision: Language and Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Humans interpret safety not as a binary signal but as a continuous, context- and spatially-dependent notion of risk. While risk is subjective, humans form rational mental models that guide action selection in dynamic environments. This work proposes a framework for extracting implicit human risk models by introducing a novel, semantically-conditioned and spatially-varying parametrization of risk, supervised directly from safe human demonstration videos and VLM common sense. Notably, we define risk through a Bayesian formulation. The prior is furnished by a pretrained vision-language model. In order to encourage the risk estimate to be more human aligned, a likelihood function modulates the prior to produce a relative metric of risk. Specifically, the likelihood is a learned ViT that maps pretrained features, to pixel-aligned risk values. Our pipeline ingests RGB images and a query object string, producing pixel-dense risk images. These images that can then be used as value-predictors in robot planning tasks or be projected into 3D for use in conventional trajectory optimization to produce human-like motion. This learned mapping enables generalization to novel objects and contexts, and has the potential to scale to much larger training datasets. In particular, the Bayesian framework that is introduced enables fast adaptation of our model to additional observations or common sense rules. We demonstrate that our proposed framework produces contextual risk that aligns with human preferences. Additionally, we illustrate several downstream applications of the model; as a value learner for visuomotor planners or in conjunction with a classical trajectory optimization algorithm. Our results suggest that our framework is a significant step toward enabling autonomous systems to internalize human-like risk. Code and results can be found at https://riskbayesian.github.io/bayesian_risk/.

Problem

• Extracts implicit human risk models from safe demonstration videos
• Learns continuous, context-dependent risk using VLM prior and Bayesian formulation
• Enables robots to plan human-like motion by predicting pixel-dense risk images

Innovation

• Bayesian risk fields with VLM prior for safety
• Pixel-dense risk images from RGB and object queries
• Generalizable mapping for novel objects and contexts