Task-Specific Adaptation with Restricted Model Access

2025-02-02

AI Summary

In gray-box settings where model weights and architecture are inaccessible, conventional fine-tuning is infeasible due to parameter opacity and privacy constraints. Method: This paper proposes a lightweight, gradient-only fine-tuning paradigm that requires no internal parameter access. It introduces (1) dual lightweight learnable adapters at input and output layers for low-overhead task adaptation; (2) a progressive exposure mechanism to optimally balance performance and privacy; and (3) gradient passthrough combined with multimodal alignment (text–image/video/sketch) to enable cross-architecture and cross-modal transfer. Results: Evaluated across multiple backbone models and mainstream benchmarks, our method matches full-parameter fine-tuning in accuracy while reducing GPU memory consumption by over 60%. Crucially, it eliminates weight leakage and mitigates reverse-engineering attacks, significantly enhancing deployment security and hardware compatibility.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The emergence of foundational models has greatly improved performance across various downstream tasks, with fine-tuning often yielding even better results. However, existing fine-tuning approaches typically require access to model weights and layers, leading to challenges such as managing multiple model copies or inference pipelines, inefficiencies in edge device optimization, and concerns over proprietary rights, privacy, and exposure to unsafe model variants. In this paper, we address these challenges by exploring"Gray-box"fine-tuning approaches, where the model's architecture and weights remain hidden, allowing only gradient propagation. We introduce a novel yet simple and effective framework that adapts to new tasks using two lightweight learnable modules at the model's input and output. Additionally, we present a less restrictive variant that offers more entry points into the model, balancing performance with model exposure. We evaluate our approaches across several backbones on benchmarks such as text-image alignment, text-video alignment, and sketch-image alignment. Results show that our Gray-box approaches are competitive with full-access fine-tuning methods, despite having limited access to the model.

Problem

• Federated Learning
• Model Fine-tuning
• Privacy Preservation

Innovation

• Gray-box Fine-tuning
• Model Adaptability
• Privacy Protection