Hierarchical Text Classification Using Black Box Large Language Models

2025-08-06

AI Summary

Hierarchical text classification (HTC) faces challenges of data scarcity and high model complexity. Method: This paper investigates few-shot and zero-shot HTC using black-box large language models (LLMs) via API calls—bypassing reliance on extensive labeled data and computational training resources. We propose and comparatively evaluate three prompting strategies: (i) direct leaf-label prediction, (ii) direct hierarchical-label prediction, and (iii) top-down multi-step prediction. Contribution/Results: Experiments show that direct hierarchical-label prediction achieves the best performance on deep category hierarchies, substantially outperforming conventional supervised methods. Few-shot settings consistently improve accuracy, demonstrating LLMs’ strong generalization capability for HTC. Moreover, we empirically identify a positive correlation between performance gains and API invocation costs as hierarchy depth increases—revealing a critical accuracy-cost trade-off. These findings provide empirical grounding and strategic guidance for efficient deployment of LLM-based HTC systems.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Hierarchical Text Classification (HTC) aims to assign texts to structured label hierarchies; however, it faces challenges due to data scarcity and model complexity. This study explores the feasibility of using black box Large Language Models (LLMs) accessed via APIs for HTC, as an alternative to traditional machine learning methods that require extensive labeled data and computational resources. We evaluate three prompting strategies -- Direct Leaf Label Prediction (DL), Direct Hierarchical Label Prediction (DH), and Top-down Multi-step Hierarchical Label Prediction (TMH) -- in both zero-shot and few-shot settings, comparing the accuracy and cost-effectiveness of these strategies. Experiments on two datasets show that a few-shot setting consistently improves classification accuracy compared to a zero-shot setting. While a traditional machine learning model achieves high accuracy on a dataset with a shallow hierarchy, LLMs, especially DH strategy, tend to outperform the machine learning model on a dataset with a deeper hierarchy. API costs increase significantly due to the higher input tokens required for deeper label hierarchies on DH strategy. These results emphasize the trade-off between accuracy improvement and the computational cost of prompt strategy. These findings highlight the potential of black box LLMs for HTC while underscoring the need to carefully select a prompt strategy to balance performance and cost.

Problem

• Explores using black box LLMs for hierarchical text classification
• Evaluates prompting strategies for accuracy and cost-effectiveness
• Highlights trade-off between accuracy improvement and computational cost

Innovation

• Uses black box LLMs via APIs for HTC
• Evaluates three prompting strategies for accuracy
• Balances performance and cost in HTC