Optimizing Rank for High-Fidelity Implicit Neural Representations

2025-12-16

AI Summary

Implicit Neural Representations (INRs) based on MLPs exhibit limited high-frequency reconstruction capability, conventionally attributed to an intrinsic spectral bias; however, this work identifies weight matrix rank degeneration during training—not architectural limitations—as the fundamental bottleneck. Method: We systematically establish rank degeneration as the critical mechanism impeding high-frequency signal representation and propose a high-rank optimization strategy using optimizers such as Muon, which preserves high-rank, near-orthogonal weight updates without modifying the network architecture. Contribution/Results: Our approach requires only standard ReLU MLPs—no coordinate encoding or specialized activation functions—and achieves up to 9 dB PSNR improvement over state-of-the-art methods across natural images, medical imaging, and novel view synthesis tasks, demonstrating superior signal fidelity and generalizability.

Technology Category

• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Implicit Neural Representations (INRs) based on vanilla Multi-Layer Perceptrons (MLPs) are widely believed to be incapable of representing high-frequency content. This has directed research efforts towards architectural interventions, such as coordinate embeddings or specialized activation functions, to represent high-frequency signals. In this paper, we challenge the notion that the low-frequency bias of vanilla MLPs is an intrinsic, architectural limitation to learn high-frequency content, but instead a symptom of stable rank degradation during training. We empirically demonstrate that regulating the network's rank during training substantially improves the fidelity of the learned signal, rendering even simple MLP architectures expressive. Extensive experiments show that using optimizers like Muon, with high-rank, near-orthogonal updates, consistently enhances INR architectures even beyond simple ReLU MLPs. These substantial improvements hold across a diverse range of domains, including natural and medical images, and novel view synthesis, with up to 9 dB PSNR improvements over the previous state-of-the-art. Our project page, which includes code and experimental results, is available at: (https://muon-inrs.github.io).

Problem

• Challenges low-frequency bias in MLPs for high-fidelity signals
• Addresses rank degradation during training to improve INR performance
• Enhances implicit neural representations across diverse domains

Innovation

• Regulating network rank during training improves fidelity
• Using high-rank near-orthogonal optimizers like Muon enhances architectures
• Simple MLPs become expressive with rank control across domains