Zero-Shot Decision Tree Construction via Large Language Models

2025-01-27

AI Summary

To address the challenge of decision tree construction under label scarcity, this paper proposes Zero-Shot Decision Trees (ZSDT), the first method enabling full decision tree induction without any labeled training data. Leveraging only the zero-shot reasoning capabilities of large language models (LLMs), ZSDT autonomously performs attribute discretization, class probability estimation, and Gini impurity computation—strictly adhering to the CART splitting criterion to produce a complete, interpretable tree structure. The approach integrates prompt-engineered probabilistic semantic parsing with classical statistical logic, ensuring trustworthy knowledge transfer from LLMs to interpretable machine learning. Evaluated on multi-class tabular datasets, ZSDT significantly outperforms existing zero-shot baselines and matches the accuracy of supervised decision trees, while preserving end-to-end transparency and structural integrity. This work establishes a novel paradigm for trustworthy AI modeling in low-resource settings.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

This paper introduces a novel algorithm for constructing decision trees using large language models (LLMs) in a zero-shot manner based on Classification and Regression Trees (CART) principles. Traditional decision tree induction methods rely heavily on labeled data to recursively partition data using criteria such as information gain or the Gini index. In contrast, we propose a method that uses the pre-trained knowledge embedded in LLMs to build decision trees without requiring training data. Our approach leverages LLMs to perform operations essential for decision tree construction, including attribute discretization, probability calculation, and Gini index computation based on the probabilities. We show that these zero-shot decision trees can outperform baseline zero-shot methods and achieve competitive performance compared to supervised data-driven decision trees on tabular datasets. The decision trees constructed via this method provide transparent and interpretable models, addressing data scarcity while preserving interpretability. This work establishes a new baseline in low-data machine learning, offering a principled, knowledge-driven alternative to data-driven tree construction.

Problem

• Decision Tree
• Limited Data
• Machine Learning

Innovation

• Pre-trained Lexicons
• Zero-shot Learning
• Interpretable Decision Trees