ISALux: Illumination and Segmentation Aware Transformer Employing Mixture of Experts for Low Light Image Enhancement

2025-08-25

AI Summary

To address insufficient joint modeling of illumination and semantic priors in low-light image enhancement (LLIE), this paper proposes an illumination-semantic dual-stream Transformer framework. Methodologically, it introduces: (1) the Hierarchical Illumination-Semantic Association Multi-Head Self-Attention (HISA-MSA) mechanism to explicitly capture cross-scale correlations between illumination distribution and semantic structure; (2) a Mixture-of-Experts (MoE) gating mechanism integrated with dual-stream self-attention for dynamic fusion of illumination correction and semantic guidance; and (3) Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning, mitigating overfitting under limited training data. The method achieves state-of-the-art performance on benchmark datasets—including LOL, DARK, and SID—demonstrating significant improvements in both quantitative metrics (PSNR/SSIM) and perceptual quality. Ablation studies validate the effectiveness and complementary nature of each component.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We introduce ISALux, a novel transformer-based approach for Low-Light Image Enhancement (LLIE) that seamlessly integrates illumination and semantic priors. Our architecture includes an original self-attention block, Hybrid Illumination and Semantics-Aware Multi-Headed Self- Attention (HISA-MSA), which integrates illumination and semantic segmentation maps for en- hanced feature extraction. ISALux employs two self-attention modules to independently process illumination and semantic features, selectively enriching each other to regulate luminance and high- light structural variations in real-world scenarios. A Mixture of Experts (MoE)-based Feed-Forward Network (FFN) enhances contextual learning, with a gating mechanism conditionally activating the top K experts for specialized processing. To address overfitting in LLIE methods caused by distinct light patterns in benchmarking datasets, we enhance the HISA-MSA module with low-rank matrix adaptations (LoRA). Extensive qualitative and quantitative evaluations across multiple specialized datasets demonstrate that ISALux is competitive with state-of-the-art (SOTA) methods. Addition- ally, an ablation study highlights the contribution of each component in the proposed model. Code will be released upon publication.

Problem

• Enhancing low-light images using illumination and semantic priors
• Addressing overfitting in low-light enhancement with LoRA adaptations
• Integrating Mixture of Experts for specialized contextual processing

Innovation

• Transformer with illumination and semantics integration
• Mixture of Experts for contextual learning enhancement
• Low-rank adaptations to prevent dataset overfitting