VITA: Variational Pretraining of Transformers for Climate-Robust Crop Yield Forecasting

2025-08-05

AI Summary

Existing crop yield forecasting models suffer significant performance degradation under extreme climatic conditions, primarily due to asymmetric data distributions: abundant pretraining meteorological data versus scarce fine-tuning yield observations. To address this, we propose VITA—a Transformer-based framework integrated with a variational autoencoder. VITA introduces meteorological variable proxy tasks and feature-masking self-supervised pretraining, enabling efficient fine-tuning using only basic meteorological statistics. This design facilitates deep, robust representation learning of atmospheric states. Empirically evaluated across 763 counties in the U.S. Corn Belt, VITA achieves state-of-the-art (SOTA) accuracy in both normal and extreme years—demonstrating statistically significant gains during extreme years (p ≈ 0.01). Moreover, VITA requires substantially less data than mainstream alternatives (e.g., GNN-RNN hybrids), effectively mitigating the small-sample challenge in extreme-event modeling.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Accurate crop yield forecasting is essential for global food security. However, current AI models systematically underperform when yields deviate from historical trends. This issue arises from key data challenges, including a major asymmetry between rich pretraining weather datasets and the limited data available for fine-tuning. We introduce VITA (Variational Inference Transformer for Asymmetric data), a variational pretraining framework that addresses this asymmetry. Instead of relying on input reconstruction, VITA uses detailed weather variables as proxy targets during pretraining and learns to predict rich atmospheric states through self-supervised feature masking. This allows the model to be fine-tuned using only basic weather statistics during deployment. Applied to 763 counties in the U.S. Corn Belt, VITA achieves state-of-the-art performance in predicting corn and soybean yields across all evaluation scenarios. While it consistently delivers superior performance under normal conditions, its advantages are particularly pronounced during extreme weather years, with statistically significant improvements (paired t-test, $p approx 0.01$). Importantly, VITA outperforms prior frameworks like GNN-RNN using less data, making it more practical for real-world use--particularly in data-scarce regions. This work highlights how domain-aware AI design can overcome data limitations and support resilient agricultural forecasting in a changing climate.

Problem

• Addresses AI model underperformance in extreme crop yield deviations
• Resolves asymmetry between pretraining and fine-tuning weather data
• Improves crop yield forecasting with limited fine-tuning data

Innovation

• Variational pretraining with weather proxy targets
• Self-supervised feature masking for atmospheric states
• Fine-tuning using basic weather statistics only