Logical perspectives on learning statistical objects

2025-04-01

AI Summary

This paper investigates how learnability of a base function class propagates to derived statistical function classes—such as expectation operators—focusing on tight sample complexity characterizations in both PAC and online learning frameworks. Methodologically, it introduces, for the first time in learning theory, “structure randomization” from model theory, integrating combinatorial dimensions (e.g., VC dimension, Littlestone dimension) with structural properties of the base class to derive upper bounds on the sample complexity of statistical classes. The approach yields significantly improved bounds, especially for statistical classes induced by hypothesis classes definable via logical formulas. Its contributions include: (i) a unified treatment of PAC and online learning; (ii) revelation of deep connections between semantic properties of logical formulas and statistical learnability; and (iii) the first computable framework for quantifying complexity of such statistical classes.

Technology Category

• Machine Learning: Statistical Relational/Logic Learning
• Reasoning under Uncertainty: Stochastic Optimization
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

We consider the relationship between learnability of a ``base class'' of functions on a set X and learnability of a class of statistical functions derived from the base class. For example, we refine results showing that learnability of a family of functions implies learnability of the family of functions mapping a function in the class to its expectation under a distribution. We will look at both Probably Approximately Correct (PAC) learning, where example inputs and outputs are chosen at random, and online learning, where the examples are chosen adversarially. We establish improved bounds on the sample complexity of learning for statistical classes, stated in terms of combinatorial dimensions of the base class. We do this by adapting techniques introduced in model theory for ``randomizing a structure''. We give particular attention to classes derived from logical formulas, and relate learnability of the statistical classes to properties of the formula. Finally, we provide bounds on the complexity of learning the statistical classes built on top of a logic-based hypothesis class.

Problem

• Relationship between base class and statistical functions learnability
• Improved bounds on sample complexity for statistical classes
• Learnability bounds for logic-based statistical hypothesis classes

Innovation

• Refines learnability of statistical function classes
• Adapts model theory randomization techniques
• Bounds complexity for logic-based statistical learning