Hyperparameter Trajectory Inference with Conditional Lagrangian Optimal Transport

2026-03-02

AI Summary

This work addresses the challenge of adapting deployed neural networks to evolving user preferences without costly retraining, which is typically required for hyperparameter adjustment. To this end, the paper introduces the Hyperparameter Trajectory Inference (HTI) task, which aims to learn the conditional distribution of a neural network’s outputs with respect to its hyperparameters, thereby enabling the construction of a surrogate model that approximates the original network under unseen hyperparameter settings. Methodologically, it pioneers the integration of conditional Lagrangian optimal transport into HTI, leveraging the manifold hypothesis and the principle of least action to jointly learn the Lagrangian governing hyperparameter dynamics, the optimal transport map, and geodesic trajectories—ensuring physically plausible inference paths. Experiments demonstrate that the proposed approach outperforms existing surrogates in reconstructing network outputs across diverse hyperparameter spectra.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Learning with Manifolds
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Neural networks (NNs) often have critical behavioural trade-offs that are set at design time with hyperparameters-such as reward weights in reinforcement learning or quantile targets in regression. Post-deployment, however, user preferences can evolve, making initial settings undesirable, necessitating potentially expensive retraining. To circumvent this, we introduce the task of Hyperparameter Trajectory Inference (HTI): to learn, from observed data, how a NN's conditional output distribution changes with its hyperparameters, and construct a surrogate model that approximates the NN at unobserved hyperparameter settings. HTI requires extending existing trajectory inference approaches to incorporate conditions, exacerbating the challenge of ensuring inferred paths are feasible. We propose an approach based on conditional Lagrangian optimal transport, jointly learning the Lagrangian function governing hyperparameter-induced dynamics along with the associated optimal transport maps and geodesics between observed marginals, which form the surrogate model. We incorporate inductive biases based on the manifold hypothesis and least-action principles into the learned Lagrangian, improving surrogate model feasibility. We empirically demonstrate that our approach reconstructs NN outputs across various hyperparameter spectra better than other alternatives.

Problem

• Hyperparameter Trajectory Inference
• Neural Networks
• Conditional Output Distribution
• Surrogate Modeling
• Optimal Transport

Innovation

• Hyperparameter Trajectory Inference
• Conditional Lagrangian Optimal Transport
• Surrogate Modeling
• Manifold Hypothesis
• Least-Action Principle