Depth-Enhanced YOLO-SAM2 Detection for Reliable Ballast Insufficiency Identification

2026-02-21

AI Summary

This study addresses the low recall and insufficient reliability of existing RGB-based methods for detecting insufficient railway ballast. To overcome these limitations, we propose a highly robust detection framework that fuses RGB-D data. Our approach begins with a sleeper-aligned depth correction pipeline, which enhances depth accuracy through polynomial modeling, RANSAC, and temporal smoothing. Subsequently, we integrate YOLOv8 for object localization with SAM2 for instance segmentation, leveraging axis-aligned and rotated bounding box sampling along with geometric criteria to classify ballast conditions. Evaluated on real-world top-view RGB-D datasets, the proposed method significantly improves recall from 0.49 to 0.80 and achieves an F1 score exceeding 0.80, outperforming current state-of-the-art approaches.

Technology Category

• Intelligent Robots: State Estimation
• Computer Vision: Other Foundations of Computer Vision
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

This paper presents a depth-enhanced YOLO-SAM2 framework for detecting ballast insufficiency in railway tracks using RGB-D data. Although YOLOv8 provides reliable localization, the RGB-only model shows limited safety performance, achieving high precision (0.99) but low recall (0.49) due to insufficient ballast, as it tends to over-predict the sufficient class. To improve reliability, we incorporate depth-based geometric analysis enabled by a sleeper-aligned depth-correction pipeline that compensates for RealSense spatial distortion using polynomial modeling, RANSAC, and temporal smoothing. SAM2 segmentation further refines region-of-interest masks, enabling accurate extraction of sleeper and ballast profiles for geometric classification. Experiments on field-collected top-down RGB-D data show that depth-enhanced configurations substantially improve the detection of insufficient ballast. Depending on bounding-box sampling (AABB or RBB) and geometric criteria, recall increases from 0.49 to as high as 0.80, and F1-score improves from 0.66 to over 0.80. These results demonstrate that integrating depth correction with YOLO-SAM2 yields a more robust and reliable approach for automated railway ballast inspection, particularly in visually ambiguous or safety-critical scenarios.

Problem

• ballast insufficiency
• railway inspection
• RGB-D data
• object detection
• depth enhancement

Innovation

• depth correction
• YOLO-SAM2
• ballast insufficiency detection
• geometric analysis
• RGB-D railway inspection