LeCropFollow: Latent Space Planning for Navigation in Unstructured Crop Fields

2026-06-30

AI Summary

This work addresses the challenge of navigation failure faced by agricultural robots in unstructured field environments—such as those with missing or irregular crop rows—by proposing a visual navigation framework based on learned latent-space representations. Rather than relying on explicit geometric modeling or conventional feature compression, the method performs trajectory planning directly within a high-dimensional latent manifold that preserves rich semantic content and uncertainty. It integrates a self-supervised semantic heatmap extractor with a TD-MPC2 reinforcement learning planner to enable zero-shot sim-to-real transfer. Field experiments demonstrate that the approach reduces semantic navigation failure rates by 2.4× in maize fields with missing rows compared to keypoint-based baselines, while maintaining state-of-the-art performance in regularly planted rows.

Abstract

Unstructured navigational features, such as irregular planting or discontinuities, remain the primary failure mode for under-canopy agricultural robots. Existing geometric approaches often fail in these scenarios because they compress high-dimensional visual data into deterministic spatial references, effectively discarding the uncertainty and semantic context required to navigate ambiguous terrain. To address this, we present LeCropFollow, a visual navigation framework that bypasses explicit geometric modeling in favor of a learned latent representation. By integrating a self-supervised semantic heatmap extractor with TD-MPC2, a Model-Based Reinforcement Learning (MBRL) planner, our system optimizes trajectories directly within a latent manifold. The framework operates over the uncompressed heatmap signal, preserving the semantic context that geometric reductions discard. We demonstrate that this representational shift enables zero-shot transfer from simplified simulation to the physical world without fine-tuning. Extensive field experiments in late-stage corn fields show that LeCropFollow matches state-of-the-art baselines in unstructured rows but significantly outperforms them in plantation gaps, achieving a 2.4x reduction in semantic failures compared to keypoint-based methods. These results suggest that latent planning offers a robust alternative to geometric estimation for operations in heterogeneous agricultural environments. Code, models, and data available: https://felipe-tommaselli.github.io/lecropfollow .

Problem

• unstructured crop fields
• visual navigation
• geometric modeling
• semantic context
• agricultural robots

Innovation

• latent space planning
• visual navigation
• model-based reinforcement learning
• self-supervised semantic representation
• zero-shot sim-to-real transfer