Chemist Eye: A Visual Language Model-Powered System for Safety Monitoring and Robot Decision-Making in Self-Driving Laboratories

2025-08-07

AI Summary

To address multi-source safety challenges in Self-Driving Laboratories (SDLs)—including PPE non-compliance, lithium-battery fire hazards, and sudden personnel emergencies—this paper proposes a distributed multimodal safety monitoring system. The system integrates RGB-depth-thermal visual sensing and, for the first time in laboratory safety, incorporates Vision-Language Models (VLMs) to enable semantic-level risk understanding and robot-autonomous evasive decision-making. Leveraging real-time communication and third-party alarm integration, it supports cross-platform dynamic response. Evaluated in a real-world SDL environment, the system achieves 97% risk detection accuracy and 95% correct evasive decision rate. This work establishes the first VLM-driven semantic safety perception and human-robot collaborative emergency response framework for laboratories, significantly enhancing the real-time performance, robustness, and interpretability of intelligent safety management.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Safety and Robustness

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Large-scale security measurements
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The integration of robotics and automation into self-driving laboratories (SDLs) can introduce additional safety complexities, in addition to those that already apply to conventional research laboratories. Personal protective equipment (PPE) is an essential requirement for ensuring the safety and well-being of workers in laboratories, self-driving or otherwise. Fires are another important risk factor in chemical laboratories. In SDLs, fires that occur close to mobile robots, which use flammable lithium batteries, could have increased severity. Here, we present Chemist Eye, a distributed safety monitoring system designed to enhance situational awareness in SDLs. The system integrates multiple stations equipped with RGB, depth, and infrared cameras, designed to monitor incidents in SDLs. Chemist Eye is also designed to spot workers who have suffered a potential accident or medical emergency, PPE compliance and fire hazards. To do this, Chemist Eye uses decision-making driven by a vision-language model (VLM). Chemist Eye is designed for seamless integration, enabling real-time communication with robots. Based on the VLM recommendations, the system attempts to drive mobile robots away from potential fire locations, exits, or individuals not wearing PPE, and issues audible warnings where necessary. It also integrates with third-party messaging platforms to provide instant notifications to lab personnel. We tested Chemist Eye with real-world data from an SDL equipped with three mobile robots and found that the spotting of possible safety hazards and decision-making performances reached 97 % and 95 %, respectively.

Problem

• Monitors safety hazards like PPE compliance and fires in self-driving labs
• Uses vision-language model for real-time robot decision-making and warnings
• Enhances situational awareness by integrating multi-sensor cameras and alerts

Innovation

• Distributed safety monitoring with multi-sensor stations
• Vision-language model-driven robot decision-making
• Real-time hazard alerts and robot redirection