The Inlet Rank Collapse in Implicit Neural Representations: Diagnosis and Unified Remedy

2026-02-02

AI Summary

This work addresses the challenge that implicit neural representations (INRs) struggle to recover fine-grained details under limited training, a limitation rooted in the insufficient expansion of input coordinates into a high-dimensional embedding space at the first layer, leading to rank collapse in the initial representation. By decomposing the neural tangent kernel (NTK) layer by layer, this study is the first to reveal—through a structural lens—the “entry-level rank collapse” phenomenon and provides a unified explanation for the mechanisms underlying positional encoding, SIREN, and batch normalization. Building on this insight, the authors propose a rank-expanding initialization strategy that explicitly controls the initial representation rank without altering network architecture or increasing computational cost. Experiments demonstrate that, with this initialization alone, a standard MLP achieves high-fidelity signal reconstruction, confirming the critical role of initial rank propagation in enhancing INR performance.

Technology Category

• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Implicit Neural Representations (INRs) have revolutionized continuous signal modeling, yet they struggle to recover fine-grained details within finite training budgets. While empirical techniques, such as positional encoding (PE), sinusoidal activations (SIREN), and batch normalization (BN), effectively mitigate this, their theoretical justifications are predominantly post hoc, focusing on the global NTK spectrum only after modifications are applied. In this work, we reverse this paradigm by introducing a structural diagnostic framework. By performing a layer-wise decomposition of the NTK, we mathematically identify the ``Inlet Rank Collapse'': a phenomenon where the low-dimensional input coordinates fail to span the high-dimensional embedding space, creating a fundamental rank deficiency at the first layer that acts as an expressive bottleneck for the entire network. This framework provides a unified perspective to re-interpret PE, SIREN, and BN as different forms of rank restoration. Guided by this diagnosis, we derive a Rank-Expanding Initialization, a minimalist remedy that ensures the representation rank scales with the layer width without architectural modifications or computational overhead. Our results demonstrate that this principled remedy enables standard MLPs to achieve high-fidelity reconstructions, proving that the key to empowering INRs lies in the structural optimization of the initial rank propagation to effectively populate the latent space.

Problem

• Implicit Neural Representations
• Inlet Rank Collapse
• Neural Tangent Kernel
• Representation Rank
• Expressive Bottleneck

Innovation

• Inlet Rank Collapse
• Neural Tangent Kernel
• Implicit Neural Representations
• Rank-Expanding Initialization
• Structural Diagnosis