ChEmREF: Evaluating Language Model Readiness for Chemical Emergency Response

2025-11-13

AI Summary

Emergency responders require rapid, reliable decision support during hazardous materials (HAZMAT) incidents. Method: We propose ChEmREF—the first multi-task evaluation framework tailored for chemical emergency response—integrating chemical representation conversion, emergency response generation, and domain-specific question answering across 1,035 chemicals, grounded in authoritative sources including the Emergency Response Guidebook (ERG) and PubChem. Evaluation employs precise matching, LLM-as-a-judge scoring, and multiple-choice accuracy. Contribution/Results: Experiments show state-of-the-art LLMs achieve 68.0%, 52.7%, and 63.9% performance on the three tasks, respectively, demonstrating their auxiliary potential in public safety-critical scenarios. However, significant reliability gaps persist, underscoring the necessity of human oversight. ChEmREF establishes a benchmark for rigorous, task-aligned assessment of LLMs in chemical emergency response.

Technology Category

• Reasoning under Uncertainty: Uncertainty Representations
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Emergency responders managing hazardous material HAZMAT incidents face critical, time-sensitive decisions, manually navigating extensive chemical guidelines. We investigate whether today's language models can assist responders by rapidly and reliably understanding critical information, identifying hazards, and providing recommendations.We introduce the Chemical Emergency Response Evaluation Framework (ChEmREF), a new benchmark comprising questions on 1,035 HAZMAT chemicals from the Emergency Response Guidebook and the PubChem Database. ChEmREF is organized into three tasks: (1) translation of chemical representation between structured and unstructured forms (e.g., converting C2H6O to ethanol), (2) emergency response generation (e.g., recommending appropriate evacuation distances) and (3) domain knowledge question answering from chemical safety and certification exams. Our best evaluated models received an exact match of 68.0% on unstructured HAZMAT chemical representation translation, a LLM Judge score of 52.7% on incident response recommendations, and a multiple-choice accuracy of 63.9% on HAMZAT examinations.These findings suggest that while language models show potential to assist emergency responders in various tasks, they require careful human oversight due to their current limitations.

Problem

• Evaluating language models' ability to assist chemical emergency response decisions
• Testing models on chemical representation translation and hazard identification
• Assessing reliability of AI recommendations for HAZMAT incident management

Innovation

• Introduces ChEmREF benchmark for chemical emergency response
• Evaluates models on chemical translation and response tasks
• Shows models need human oversight despite potential