On Transferring Transferability: Towards a Theory for Size Generalization

2025-05-29

AI Summary

Non-Euclidean structures—such as graphs, sets, and point clouds—exhibit poor generalization across varying input sizes, posing a fundamental challenge for learning invariant representations. Method: This work establishes the first rigorous theoretical framework for cross-dimensional transferability, formalizing size transferability as continuity in a limiting space and constructing dimension-agnostic generalization models via data- and task-driven equivalence identification. We propose design principles for transferable architectures and apply them to adapt mainstream graph neural networks as well as set- and point-cloud models. Contribution/Results: We derive necessary and sufficient conditions for transferability and prove their theoretical validity. Extensive experiments on multiple benchmarks demonstrate that the adapted models achieve significantly improved generalization performance on out-of-distribution size tasks, strongly corroborating the theoretical analysis.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Many modern learning tasks require models that can take inputs of varying sizes. Consequently, dimension-independent architectures have been proposed for domains where the inputs are graphs, sets, and point clouds. Recent work on graph neural networks has explored whether a model trained on low-dimensional data can transfer its performance to higher-dimensional inputs. We extend this body of work by introducing a general framework for transferability across dimensions. We show that transferability corresponds precisely to continuity in a limit space formed by identifying small problem instances with equivalent large ones. This identification is driven by the data and the learning task. We instantiate our framework on existing architectures, and implement the necessary changes to ensure their transferability. Finally, we provide design principles for designing new transferable models. Numerical experiments support our findings.

Problem

• Develops theory for size generalization in learning models
• Explores transferability across varying input dimensions
• Provides design principles for transferable model architectures

Innovation

• General framework for dimension transferability
• Continuity in limit space ensures transferability
• Design principles for transferable models