Beyond Chamfer Distance: Granular Order-aware Evaluation Metric For Online Mapping

2026-05-21

AI Summary

Existing online map construction evaluation relies on Chamfer-distance-based mAP, which ignores point sequence order and employs coarse geometric granularity, making it difficult to distinguish the performance of advanced methods. This work proposes a fine-grained evaluation framework that jointly accounts for sequence sensitivity and geometric precision. It introduces Sequence Optimal Subpattern Assignment (SOSPA), a similarity metric satisfying metric axioms for single-instance comparison, and integrates a soft-matching Polyline Detection and Localization (PLD) mechanism for multi-instance joint assessment. The framework overcomes the limitations of conventional hard-threshold mAP, effectively differentiating state-of-the-art methods on nuScenes, revealing that current model bottlenecks primarily stem from detection capability, and enabling interpretable error decomposition.

Abstract

Online map estimation is a crucial component of autonomous driving systems that reduces the reliance on costly high-definition maps. State-of-the-art (SOTA) methods commonly predict map elements as ordered sequences of points that form polylines and polygons. The evaluation of these methods relies predominantly on mean average precision (mAP) based on thresholded Chamfer distance (CD). This framework lacks sensitivity to point ordering and provides limited granularity in assessing geometric quality, making it difficult to distinguish which methods truly excel over others. In this work, we address these limitations on two fronts. For the single-instance similarity measure, we introduce sequence optimal sub-pattern assignment (SOSPA), an order-aware metric that enables fine-grained evaluation of individual geometries while satisfying all metric axioms. For the multi-instance evaluation framework, we propose polyline localisation and detection (PLD), a soft metric that jointly captures detection quality and geometric accuracy, replacing the hard thresholding of mAP with a principled soft assignment. Through evaluations on nuScenes, we demonstrate that PLD effectively ranks SOTA online mapping methods (MapTRv2, StreamMapNet, MapTracker) while providing a decomposed error analysis. This analysis identifies detection capability as the dominant bottleneck in current methods, revealing a performance trend that mAP fails to capture. Code for evaluation using our metrics will be released.

Problem

• online mapping
• evaluation metric
• Chamfer distance
• point ordering
• geometric quality

Innovation

• SOSPA
• PLD
• order-aware metric
• online mapping evaluation
• soft assignment