Unveiling Multiple Descents in Unsupervised Autoencoders

2024-06-17

AI Summary

Dual and triple descent phenomena—previously observed exclusively in supervised learning—have not been systematically characterized in unsupervised autoencoders (AEs). Method: We conduct a comprehensive theoretical and empirical study of linear and nonlinear AEs, analyzing reconstruction error and downstream task performance (anomaly detection, domain adaptation) under varying bottleneck dimensions and feature/sample noise levels. Experiments span synthetic and real-world datasets across diverse architectures. Results: We are the first to rigorously identify and reproduce multi-scale descent behaviors—occurring at model-level, epoch-level, and sample-level—in unsupervised AEs. These phenomena manifest robustly across architectures and datasets. Crucially, moderate over-parameterization not only reduces reconstruction error but also improves downstream generalization. Our work provides the first theoretical explanation and empirical validation of multi-descent in nonlinear AEs, establishing a principled link between bottleneck design and generalization behavior.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

The phenomenon of double descent has challenged the traditional bias-variance trade-off in supervised learning but remains unexplored in unsupervised learning, with some studies arguing for its absence. In this study, we first demonstrate analytically that double descent does not occur in linear unsupervised autoencoders (AEs). In contrast, we show for the first time that both double and triple descent can be observed with nonlinear AEs across various data models and architectural designs. We examine the effects of partial sample and feature noise and highlight the importance of bottleneck size in influencing the double descent curve. Through extensive experiments on both synthetic and real datasets, we uncover model-wise, epoch-wise, and sample-wise double descent across several data types and architectures. Our findings indicate that over-parameterized models not only improve reconstruction but also enhance performance in downstream tasks such as anomaly detection and domain adaptation, highlighting their practical value in complex real-world scenarios.

Problem

• Double descent in unsupervised autoencoders
• Impact of bottleneck size on descent curves
• Over-parameterized models enhance downstream tasks

Innovation

• Nonlinear autoencoders demonstrate multiple descents
• Examine noise effects on double descent
• Over-parameterized models enhance downstream tasks