A Smart-Glasses for Emergency Medical Services via Multimodal Multitask Learning

2025-11-17

AI Summary

To address the challenge of rapid multimodal information fusion and critical decision-making by emergency medical personnel under high-stress conditions, this paper introduces EMSGlass—a smart-glasses system for prehospital emergency care. Methodologically, we propose EMSNet, the first multimodal, multitask model tailored to prehospital settings, jointly encoding textual instructions, real-time vital signs, and scene images. We further design EMSServe, a low-latency inference framework supporting asynchronous modality processing, modality-aware model partitioning, and feature caching for adaptive deployment across heterogeneous edge hardware. Experiments demonstrate significant accuracy gains over unimodal baselines and 1.9×–11.7× faster end-to-end inference. A user study confirms reduced decision latency and improved operational efficiency. Our core contributions include: (1) the first lightweight, multimodal cognitive system specifically designed for emergency response; (2) a novel multitask collaborative architecture; and (3) an efficient edge inference paradigm for time-critical healthcare applications.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Emergency Medical Technicians (EMTs) operate in high-pressure environments, making rapid, life-critical decisions under heavy cognitive and operational loads. We present EMSGlass, a smart-glasses system powered by EMSNet, the first multimodal multitask model for Emergency Medical Services (EMS), and EMSServe, a low-latency multimodal serving framework tailored to EMS scenarios. EMSNet integrates text, vital signs, and scene images to construct a unified real-time understanding of EMS incidents. Trained on real-world multimodal EMS datasets, EMSNet simultaneously supports up to five critical EMS tasks with superior accuracy compared to state-of-the-art unimodal baselines. Built on top of PyTorch, EMSServe introduces a modality-aware model splitter and a feature caching mechanism, achieving adaptive and efficient inference across heterogeneous hardware while addressing the challenge of asynchronous modality arrival in the field. By optimizing multimodal inference execution in EMS scenarios, EMSServe achieves 1.9x -- 11.7x speedup over direct PyTorch multimodal inference. A user study evaluation with six professional EMTs demonstrates that EMSGlass enhances real-time situational awareness, decision-making speed, and operational efficiency through intuitive on-glass interaction. In addition, qualitative insights from the user study provide actionable directions for extending EMSGlass toward next-generation AI-enabled EMS systems, bridging multimodal intelligence with real-world emergency response workflows.

Problem

• Develops smart glasses to assist EMTs in high-pressure decision-making environments
• Integrates multimodal data for real-time understanding of emergency medical incidents
• Optimizes inference speed for efficient emergency response through specialized serving framework

Innovation

• Multimodal multitask learning integrates text vitals images
• Modality-aware model splitter enables adaptive efficient inference
• Feature caching mechanism handles asynchronous modality arrival