Learning Transferable Features for Implicit Neural Representations

2024-09-15

arXiv.org

AI Summary

Implicit Neural Representations (INRs) suffer from poor generalization and struggle to rapidly adapt to unseen tasks. To address this, we propose STRAINER—a novel framework that systematically investigates the transferability of INR features for the first time. Our method introduces a layer-wise affine-driven shared encoder coupled with task-specific decoders, enabling cross-task feature reuse, and incorporates a lightweight, data-driven prior embedding to support effective transfer initialization. STRAINER significantly improves both the efficiency and fidelity of INR fitting on in-distribution novel signals: in image reconstruction, it achieves an early-stage signal-to-noise ratio gain of approximately 10 dB and markedly accelerates convergence. Moreover, it unifies support for in-domain and cross-domain signal fitting, as well as inverse problem solving.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Implicit neural representations (INRs) have demonstrated success in a variety of applications, including inverse problems and neural rendering. An INR is typically trained to capture one signal of interest, resulting in learned neural features that are highly attuned to that signal. Assumed to be less generalizable, we explore the aspect of transferability of such learned neural features for fitting similar signals. We introduce a new INR training framework, STRAINER that learns transferrable features for fitting INRs to new signals from a given distribution, faster and with better reconstruction quality. Owing to the sequential layer-wise affine operations in an INR, we propose to learn transferable representations by sharing initial encoder layers across multiple INRs with independent decoder layers. At test time, the learned encoder representations are transferred as initialization for an otherwise randomly initialized INR. We find STRAINER to yield extremely powerful initialization for fitting images from the same domain and allow for $approx +10dB$ gain in signal quality early on compared to an untrained INR itself. STRAINER also provides a simple way to encode data-driven priors in INRs. We evaluate STRAINER on multiple in-domain and out-of-domain signal fitting tasks and inverse problems and further provide detailed analysis and discussion on the transferability of STRAINER's features. Our demo can be accessed at https://kushalvyas.github.io/strainer.html .

Problem

• Implicit Neural Representations
• Generalization
• Task-specific Adaptation

Innovation

• STRAINER
• Implicit Neural Representations (INRs)
• Feature Sharing