FAIR Universe Weak Lensing ML Uncertainty Challenge: Handling Uncertainties and Distribution Shifts for Precision Cosmology

2026-04-15

AI Summary

This work addresses critical limitations in weak gravitational lensing analysis—namely, the scarcity of realistic simulations, inadequate modeling of systematic errors leading to distributional shifts, and the absence of a unified evaluation benchmark—which collectively hinder precise cosmological parameter inference. To bridge this gap, we introduce the first benchmark dataset for weak lensing that incorporates multiple realistic sources of systematic error and launch a machine learning challenge focused on uncertainty quantification, data efficiency, and robustness to distribution shifts. By integrating higher-order statistics, cosmological simulations, and systematic error modeling, we establish a two-stage evaluation framework and provide a standardized platform to foster collaboration between the physics and machine learning communities. This initiative aims to advance the development of reproducible, high-precision cosmological analysis methods and lay the groundwork for the reliable deployment of machine learning in upcoming large-scale sky surveys.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Adversarial Attacks & Robustness
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Security and privacy of machine learning and AI applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Weak gravitational lensing, the correlated distortion of background galaxy shapes by foreground structures, is a powerful probe of the matter distribution in our universe and allows accurate constraints on the cosmological model. In recent years, high-order statistics and machine learning (ML) techniques have been applied to weak lensing data to extract the nonlinear information beyond traditional two-point analysis. However, these methods typically rely on cosmological simulations, which poses several challenges: simulations are computationally expensive, limiting most realistic setups to a low training data regime; inaccurate modeling of systematics in the simulations create distribution shifts that can bias cosmological parameter constraints; and varying simulation setups across studies make method comparison difficult. To address these difficulties, we present the first weak lensing benchmark dataset with several realistic systematics and launch the FAIR Universe Weak Lensing Machine Learning Uncertainty Challenge. The challenge focuses on measuring the fundamental properties of the universe from weak lensing data with limited training set and potential distribution shifts, while providing a standardized benchmark for rigorous comparison across methods. Organized in two phases, the challenge will bring together the physics and ML communities to advance the methodologies for handling systematic uncertainties, data efficiency, and distribution shifts in weak lensing analysis with ML, ultimately facilitating the deployment of ML approaches into upcoming weak lensing survey analysis.

Problem

• weak lensing
• machine learning
• distribution shift
• systematic uncertainties
• cosmological parameter estimation

Innovation

• weak lensing
• machine learning
• distribution shifts
• systematic uncertainties
• benchmark dataset