Sketched Linear Contrastive Learning: Approximation, Optimization, and Statistical Scaling

2026-06-25

AI Summary

This work investigates the scaling laws of contrastive learning under varying model sizes, data volumes, and computational budgets, addressing a critical theoretical gap in the field. Focusing on linear contrastive learning with paired Gaussian latent variables and observed two-view sketches, the study employs full-batch gradient descent to train a bilinear scoring function. By introducing a Gaussian-negative quadratic surrogate loss, a power-law spectral alignment assumption, and a contrastive source condition, it establishes—for the first time—an explicit trivariate statistical scaling law that jointly accounts for sketch dimensionality, sample size, and optimization duration. This law reveals the distinct influence of two-view interactions on optimization dynamics and noise sensitivity. Through a bias–variance–cross-term decomposition, the authors derive a risk expression comprising irreducible risk, approximation error, optimization bias, and variance, offering a principled theoretical foundation for balancing model capacity, data quantity, and computational effort.

Abstract

Scaling laws describe how learning performance varies with model size, data size, and compute. While recent theoretical work has established scaling laws for sketched linear regression, much less is understood for contrastive representation learning. In this paper, we study a sketched linear model for contrastive learning under a paired Gaussian latent-variable setup. The learner observes only sketched views of two correlated variables and trains a bilinear contrastive score by full-batch empirical gradient descent. We analyze a Gaussian-negative quadratic contrastive surrogate under aligned power-law spectra and a contrastive source condition, where we derive a risk decomposition into irreducible risk, approximation error, GD bias, GD variance, and a cross term. The cross term is controlled by the bias and variance and therefore does not affect the upper-bound scaling. Our main theorem gives an explicit scaling law with respect to sketch dimension $M$, sample size $N$, and effective optimization horizon $L_{\mathrm{eff}}γ$. Compared with standard linear-regression scaling laws, the contrastive setting must learn interactions between two views, and this changes how optimization and finite-sample noise scale with model size, data, and training time. This provides a first theoretical step toward understanding scaling behavior in contrastive learning and gives guidance for balancing model size, data, and optimization compute.

Problem

• contrastive learning
• scaling laws
• sketched linear model
• representation learning
• statistical scaling

Innovation

• sketched contrastive learning
• scaling laws
• risk decomposition
• bilinear representation
• gradient descent bias-variance