Structured Output Regularization: a framework for few-shot transfer learning

2025-10-09

AI Summary

To address poor model adaptability and overfitting in few-shot transfer learning, this paper proposes a structured output regularization framework. Specifically, we freeze the internal parameters of pretrained backbones (e.g., DenseNet121, EfficientNet-B4) and fine-tune only the top classification layer. Crucially, we jointly impose group Lasso and ℓ₁ regularization on the output-layer weights—thereby explicitly enforcing both inter-channel group sparsity and global weight sparsity—without significantly increasing the number of trainable parameters. This dual regularization enhances discriminative learning of domain-specific patterns while mitigating overfitting. Evaluated on three few-shot medical image classification benchmarks, our method achieves performance on par with state-of-the-art approaches. Results demonstrate its effectiveness, strong generalization capability, and architecture-agnostic scalability across diverse backbone networks.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Traditional transfer learning typically reuses large pre-trained networks by freezing some of their weights and adding task-specific layers. While this approach is computationally efficient, it limits the model's ability to adapt to domain-specific features and can still lead to overfitting with very limited data. To address these limitations, we propose Structured Output Regularization (SOR), a simple yet effective framework that freezes the internal network structures (e.g., convolutional filters) while using a combination of group lasso and $L_1$ penalties. This framework tailors the model to specific data with minimal additional parameters and is easily applicable to various network components, such as convolutional filters or various blocks in neural networks enabling broad applicability for transfer learning tasks. We evaluate SOR on three few shot medical imaging classification tasks and we achieve competitive results using DenseNet121, and EfficientNetB4 bases compared to established benchmarks.

Problem

• Addressing overfitting in few-shot transfer learning scenarios
• Enhancing model adaptation to domain-specific features with limited data
• Reducing parameter overhead while maintaining competitive performance across tasks

Innovation

• Freezes internal network structures during transfer
• Uses group lasso and L1 penalties combination
• Tailors model with minimal additional parameters