A Priori Generalizability Estimate for a CNN

2025-02-24

AI Summary

This work addresses the challenges of evaluating CNN generalization capability—namely, its reliance on labeled data and lack of interpretable diagnostics. We propose an unsupervised prior assessment method based on full-network truncated singular value decomposition (SVD). By analyzing the left and right singular vectors of weight matrices, we introduce two novel metrics: Right Projection Ratio (RPR) and Left Projection Ratio (LPR), enabling sample-level performance prediction without ground-truth labels. Key contributions include: (i) the first application of full-network SVD for generalization diagnostics; (ii) the RPR metric, which requires only unlabeled data; and (iii) empirical evidence showing RPR strongly correlates with class imbalance and exhibits significant negative correlation (p < 0.01) with actual model performance across both image classification and segmentation tasks. The method provides an efficient, interpretable, and label-free tool for pre-deployment generalization risk assessment.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Security and Privacy: Large-scale security measurements

Abstract

We formulate truncated singular value decompositions of entire convolutional neural networks. We demonstrate the computed left and right singular vectors are useful in identifying which images the convolutional neural network is likely to perform poorly on. To create this diagnostic tool, we define two metrics: the Right Projection Ratio and the Left Projection Ratio. The Right (Left) Projection Ratio evaluates the fidelity of the projection of an image (label) onto the computed right (left) singular vectors. We observe that both ratios are able to identify the presence of class imbalance for an image classification problem. Additionally, the Right Projection Ratio, which only requires unlabeled data, is found to be correlated to the model's performance when applied to image segmentation. This suggests the Right Projection Ratio could be a useful metric to estimate how likely the model is to perform well on a sample.

Problem

• Estimates CNN generalizability using singular value decompositions.
• Identifies poor performance images via projection ratios.
• Correlates Right Projection Ratio with model segmentation performance.

Innovation

• Truncated SVD for CNN analysis
• Right Projection Ratio metric
• Left Projection Ratio metric