Multistage Defer Trees for Hybrid Interpretability: If at First You Can't Succeed, Tree Again

2026-06-29

AI Summary

This work addresses the inherent trade-off between accuracy and interpretability in machine learning: while single decision trees are transparent but limited in performance, ensemble models achieve high accuracy at the cost of opacity. The authors propose a multi-stage deferred decision tree sequence that employs a sparse decision tree backbone to deliver fully interpretable predictions for the majority of samples, deferring only a small fraction of difficult cases to subsequent trees or black-box models. This hybrid architecture establishes, for the first time, a collaborative mechanism where sparse trees serve as the primary predictor and black-box components act as auxiliary fallbacks. Evaluated across multiple datasets, the approach matches the accuracy of complex ensemble methods while ensuring that most predictions follow concise, transparent decision paths, thereby significantly advancing the accuracy–interpretability frontier.

Abstract

Recent work has shown that well-optimized individual decision trees can match complex black box models in some settings, primarily in noisy domains. For the remaining settings, however, complex ensembled compositions of trees often achieve higher accuracy at the cost of interpretability, leaving practitioners with difficult modeling decisions along an accuracy-interpretability tradeoff. Ideally, we would like to classify as much of the data as possible with one or a small number of trees, achieving interpretability for most samples while maintaining state-of-the-art accuracy. We introduce Multistage Defer Trees: a sequence of sparse decision trees that each make predictions for most samples, while deferring a small proportion to the next tree in the sequence or, ultimately, to a black box. We demonstrate that we can train this model class to match the performance of complex tree-based ensembles while routing most samples through only one or a small number of sparse decision trees. We discuss a range of techniques for training these models while maintaining simplicity. Our method expands the accuracy--interpretability frontier in settings where single-tree methods remain insufficient, demonstrating that even when complex models are necessary, they need not be fully opaque.

Problem

• interpretability
• accuracy-interpretability tradeoff
• decision trees
• black box models
• hybrid models

Innovation

• Multistage Defer Trees
• Interpretability
• Decision Trees
• Accuracy-Interpretability Tradeoff
• Hybrid Models