Lower Bounds for Greedy Teaching Set Constructions

2025-05-06

AI Summary

This paper addresses two central open problems in computational learning theory: (i) whether the minimal teaching dimension $mathrm{TS}_{min}$ of a concept class is $O(d)$, where $d$ is its VC dimension; and (ii) the validity of the Simon–Zilles (2015) recursive teaching dimension conjecture. To overcome limitations of greedy algorithms in teaching set construction, we establish the first tight lower bounds on their performance: for batch size $k=1$, the greedy algorithm achieves only $O(log|C|)$; for $k=2$, it attains merely $Omega(loglog|C|)$. We extend these bounds to $k = O(d)$, proving that greedy strategies cannot resolve the $O(d)$ conjecture—thereby highlighting the necessity of modeling higher-order interactions. Our analysis integrates combinatorial reasoning, information-theoretic arguments, and the recursive teaching framework, and contrasts the greedy approach with the halving method to rigorously characterize its inherent bottlenecks.

Technology Category

• Machine Learning: Learning Theory
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

A fundamental open problem in learning theory is to characterize the best-case teaching dimension $operatorname{TS}_{min}$ of a concept class $mathcal{C}$ with finite VC dimension $d$. Resolving this problem will, in particular, settle the conjectured upper bound on Recursive Teaching Dimension posed by [Simon and Zilles; COLT 2015]. Prior work used a natural greedy algorithm to construct teaching sets recursively, thereby proving upper bounds on $operatorname{TS}_{min}$, with the best known bound being $O(d^2)$ [Hu, Wu, Li, and Wang; COLT 2017]. In each iteration, this greedy algorithm chooses to add to the teaching set the $k$ labeled points that restrict the concept class the most. In this work, we prove lower bounds on the performance of this greedy approach for small $k$. Specifically, we show that for $k = 1$, the algorithm does not improve upon the halving-based bound of $O(log(|mathcal{C}|))$. Furthermore, for $k = 2$, we complement the upper bound of $Oleft(log(log(|mathcal{C}|)) ight)$ from [Moran, Shpilka, Wigderson, and Yuhudayoff; FOCS 2015] with a matching lower bound. Most consequentially, our lower bound extends up to $k le lceil c d ceil$ for small constant $c>0$: suggesting that studying higher-order interactions may be necessary to resolve the conjecture that $operatorname{TS}_{min} = O(d)$.

Problem

• Characterize best-case teaching dimension for finite VC dimension.
• Prove lower bounds on greedy teaching set algorithm performance.
• Investigate necessity of higher-order interactions for resolving conjecture.

Innovation

• Greedy algorithm constructs teaching sets recursively
• Lower bounds proven for small k values
• Higher-order interactions may resolve conjecture