RecourseBench: A Modular Framework for Reproducible Algorithmic Recourse Evaluation

2026-06-14

AI Summary

Existing attribution methods lack a unified, scalable, and reproducible evaluation framework, hindering systematic comparison. To address this gap, this work proposes the first modular benchmarking framework for attribution, decoupling the pipeline into five interoperable layers—data, preprocessing, model, attribution method, and evaluation—and enabling flexible integration through abstract interfaces and a dynamic registration mechanism. The framework introduces an innovative four-tier categorization system coupled with an automated testing protocol that rigorously validates whether each implemented method reproduces results from its original publication. An interactive web interface further supports multidimensional configuration and comparative analysis. Currently integrating 28 state-of-the-art attribution methods, the framework establishes the first automated, quantitative guarantee of method-level reproducibility in the field.

Abstract

Algorithmic recourse methods provide counterfactual explanations that inform individuals of the actions required to overturn an unfavorable model decision. Despite rapid methodological progress, principled comparison remains elusive; existing frameworks are often difficult to extend and lack both interoperability and systematic verification that integrated methods faithfully reproduce their originally reported results. We introduce \emph{RecourseBench}, a unified evaluation framework built around three commitments namely, modularity, reproducibility, and interactivity. The framework decomposes the pipeline into five fully decoupled layers -- Data, Preprocessing, Model, Recourse Method, and Evaluation -- governed by abstract interfaces and a dynamic registry. To address the reproducibility gap in prior benchmarks, we introduce a four-tier classification system in which every integrated method is validated by an automated test suite against its originally reported results. We further provide an interactive web interface for flexible, configuration-driven comparison across methods, datasets, and model architectures. Our framework currently integrates 28 state-of-the-art recourse methods and, to our knowledge, constitutes the first recourse benchmark to explicitly enforce method-level reproducibility through automated, quantitative testing.

Problem

• algorithmic recourse
• reproducibility
• evaluation framework
• counterfactual explanations
• benchmarking

Innovation

• algorithmic recourse
• reproducibility
• modular framework
• counterfactual explanations
• benchmarking