End-to-end Autonomous Vehicle Following System using Monocular Fisheye Camera

2025-11-24

AI Summary

Existing vehicle platooning systems rely heavily on lane markings and expensive sensors, suffering from poor generalization. To address this, we propose an end-to-end monocular fisheye vision-based car-following method. Our approach abandons conventional modular pipelines and instead integrates monocular fisheye perception, semantic segmentation, temporal modeling, and adaptive sampling into a unified framework. Key innovations include: (1) incorporating semantic mask constraints into multi-frame temporal fusion to mitigate causal confusion; and (2) designing a dynamic sampling mechanism for robust lead-vehicle trajectory tracking. The method significantly reduces hardware dependency while maintaining high fidelity. In real-world closed-loop road experiments, it achieves stable car-following under challenging conditions—including absent lane markings, partial occlusions, and varying road curvature—outperforming state-of-the-art multi-stage methods. The proposed system demonstrates strong generalization, low-cost deployment feasibility, and practical robustness.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Data transparency and provenance
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The increase in vehicle ownership has led to increased traffic congestion, more accidents, and higher carbon emissions. Vehicle platooning is a promising solution to address these issues by improving road capacity and reducing fuel consumption. However, existing platooning systems face challenges such as reliance on lane markings and expensive high-precision sensors, which limits their general applicability. To address these issues, we propose a vehicle following framework that expands its capability from restricted scenarios to general scenario applications using only a camera. This is achieved through our newly proposed end-to-end method, which improves overall driving performance. The method incorporates a semantic mask to address causal confusion in multi-frame data fusion. Additionally, we introduce a dynamic sampling mechanism to precisely track the trajectories of preceding vehicles. Extensive closed-loop validation in real-world vehicle experiments demonstrates the system's ability to follow vehicles in various scenarios, outperforming traditional multi-stage algorithms. This makes it a promising solution for cost-effective autonomous vehicle platooning. A complete real-world vehicle experiment is available at https://youtu.be/zL1bcVb9kqQ.

Problem

• Developing autonomous vehicle following using monocular fisheye camera only
• Overcoming reliance on lane markings and expensive high-precision sensors
• Expanding vehicle platooning capability from restricted to general scenarios

Innovation

• End-to-end autonomous vehicle following using monocular fisheye camera
• Semantic mask addresses causal confusion in multi-frame fusion
• Dynamic sampling mechanism precisely tracks preceding vehicle trajectories