An MRP Formulation for Supervised Learning: Generalized Temporal Difference Learning Models

2024-04-23

arXiv.org

AI Summary

Traditional supervised learning assumes i.i.d. data, contradicting inherent dependencies in real-world datasets. Method: This paper reformulates supervised learning as a policy evaluation problem on a Markov reward process (MRP), introducing a generalized temporal difference (TD) learning framework that explicitly relaxes the i.i.d. assumption. Under linear function approximation, we establish rigorous convergence guarantees and prove that TD estimators achieve superior statistical efficiency over ordinary least squares (OLS) under dependent noise. Contribution/Results: Our theoretical analysis reveals TD’s intrinsic adaptability to non-i.i.d. structures. Empirical evaluation on regression and deep image classification tasks demonstrates faster convergence, improved generalization, and practical efficacy. This work establishes a novel paradigm and foundational theory for supervised learning with dependent data.

Technology Category

• Machine Learning: Learning Theory
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

In traditional statistical learning, data points are usually assumed to be independently and identically distributed (i.i.d.) following an unknown probability distribution. This paper presents a contrasting viewpoint, perceiving data points as interconnected and employing a Markov reward process (MRP) for data modeling. We reformulate the typical supervised learning as an on-policy policy evaluation problem within reinforcement learning (RL), introducing a generalized temporal difference (TD) learning algorithm as a resolution. Theoretically, our analysis draws connections between the solutions of linear TD learning and ordinary least squares (OLS). We also show that under specific conditions, particularly when noises are correlated, the TD's solution proves to be a more effective estimator than OLS. Furthermore, we establish the convergence of our generalized TD algorithms under linear function approximation. Empirical studies verify our theoretical results, examine the vital design of our TD algorithm and show practical utility across various datasets, encompassing tasks such as regression and image classification with deep learning.

Problem

• Reformulating supervised learning as reinforcement learning policy evaluation
• Developing generalized TD learning algorithm for interconnected data
• Proving convergence and superiority over OLS under correlated noise

Innovation

• Markov reward process for data modeling
• Generalized temporal difference learning algorithm
• Novel generalized Bellman operator with convergence