🤖 AI Summary
Existing evaluation methods for segmentation models struggle to handle label noise, inconsistency, or unreliability. This work proposes an Adaptive Resolution Label Aggregation (ARLA) mechanism—the first evaluation framework specifically designed for such challenging scenarios. ARLA dynamically aligns and rescales both predictions and labels during inference, leveraging multi-scale aggregation and domain knowledge to adaptively match label fidelity with prevailing noise levels. By doing so, it effectively extracts a clear performance signal from noisy annotations. In flood prediction tasks, the method substantially mitigates issues arising from inconsistent labeling in forested regions and erroneous labels in cloud-covered areas, significantly enhancing evaluation reliability.
📝 Abstract
Labels are critical for both training and evaluating deep learning segmentation models, but are often inconsistent, noisy, or ambiguous at class boundaries. Many approaches have been developed to support training models on weak labels, but few to none currently exist to facilitate evaluating models on unreliable labels. We therefore introduce a method called "Adaptive Resolution Label Aggregation", or "ARLA", which dynamically adapts the resolution of both the label and the model prediction at inference time before the evaluation metrics are computed. We demonstrate how ARLA can be used to better analyse model behaviour with a practical application to a real flood prediction model, where ARLA was able to overcome issues with inconsistent labelling of forested areas and errors in labels within regions of heavy cloud cover. Our work presents a new approach to evaluating segmentation models, with adjustable parameters to adapt the aggregated resolution to the precision of the label or the level of label noise. Fundamentally, ARLA exploits the information encapsulated by a label but minimises the label error, extracting from the noise a clearer signal of a model's true performance.