Experimental study on surveillance video-based indoor occupancy measurement with occupant-centric control

2026-03-27

AI Summary

This study addresses the challenges of accuracy and stability in vision-based indoor occupancy estimation under real-world conditions, as well as the lack of empirical validation regarding its impact on HVAC control performance. To this end, the authors propose an occupancy-aware pipeline that integrates YOLOv8 for object detection, multi-object tracking, and—novelty—the incorporation of the DeepSeek large language model (LLM) into the post-processing stage to enhance temporal consistency and estimation accuracy. The proposed method is embedded within an OpenStudio–EnergyPlus model predictive control framework. Evaluated on real surveillance data, it achieves an accuracy of 0.8824 and an F1-score of 0.9320, while enabling a 17.94% energy saving in HVAC operation. These results demonstrate the feasibility and advantages of LLM-enhanced occupant-centric building control.

Abstract

Accurate occupancy information is essential for closed-loop occupant-centric control (OCC) in smart buildings. However, existing vision-based occupancy measurement methods often struggle to provide stable and accurate measurements in real indoor environments, and their implications for downstream HVAC control remain insufficiently studied. To achieve Net Zero emissions by 2050, this paper presents an experimental study of large language models (LLMs)-enhanced vision-based indoor occupancy measurement and its impact on OCC-enabled HVAC operation. Detection-only, tracking-based, and LLM-based refinement pipelines are compared under identical conditions using real surveillance data collected from a research laboratory in China, with frame-level manual ground-truth annotations. Results show that tracking-based methods improve temporal stability over detection-only measurement, while LLM-based refinement further improves occupancy measurement performance and reduces false unoccupied prediction. The best-performing pipeline, YOLOv8+DeepSeek, achieves an accuracy of 0.8824 and an F1-score of 0.9320. This pipeline is then integrated into an HVAC supervisory model predictive control framework in OpenStudio-EnergyPlus. Experimental results demonstrate that the proposed framework can support more efficient OCC operation, achieving a substantial HVAC energy-saving potential of 17.94%. These findings provide an effective methodology and practical foundation for future research in AI-enhanced smart building operations.

Problem

• occupancy measurement
• vision-based sensing
• occupant-centric control
• HVAC energy efficiency
• indoor environment

Innovation

• LLM-enhanced occupancy measurement
• occupant-centric control
• vision-based indoor sensing
• HVAC energy saving
• model predictive control