Solar Multimodal Transformer: Intraday Solar Irradiance Predictor using Public Cameras and Time Series

2025-02-28

AI Summary

To enable precise intra-day solar irradiance forecasting for power dispatch and trading decisions, this paper proposes a lightweight multimodal Transformer model that jointly encodes single-frame public webcam images and adaptively scaled, normalized irradiance time series. Its key contributions are: (1) a novel image–time-series joint encoding architecture enabling heterogeneous modality alignment; (2) a self-supervised irradiance time-series normalization strategy that significantly enhances cross-scenario generalization; and (3) a lightweight attention mechanism tailored for deployment across diverse camera specifications. Experimental results demonstrate a 25.95% improvement in forecasting accuracy over the commercial service Solcast, while maintaining strong robustness under real-world complex lighting conditions and high device transferability across heterogeneous camera hardware.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Accurate intraday solar irradiance forecasting is crucial for optimizing dispatch planning and electricity trading. For this purpose, we introduce a novel and effective approach that includes three distinguishing components from the literature: 1) the uncommon use of single-frame public camera imagery; 2) solar irradiance time series scaled with a proposed normalization step, which boosts performance; and 3) a lightweight multimodal model, called Solar Multimodal Transformer (SMT), that delivers accurate short-term solar irradiance forecasting by combining images and scaled time series. Benchmarking against Solcast, a leading solar forecasting service provider, our model improved prediction accuracy by 25.95%. Our approach allows for easy adaptation to various camera specifications, offering broad applicability for real-world solar forecasting challenges.

Problem

• Accurate intraday solar irradiance forecasting for energy optimization.
• Utilizing public camera imagery and time series for predictions.
• Improving prediction accuracy by 25.95% over existing services.

Innovation

• Uses single-frame public camera imagery
• Normalizes solar irradiance time series
• Lightweight multimodal Solar Multimodal Transformer