Exploring possible vector systems for faster training of neural networks with preconfigured latent spaces

2025-12-08

AI Summary

To address slow training and excessive embedding storage overhead in ultra-fine-grained classification with millions of classes, this paper proposes a fast neural training framework based on a preconfigured latent space. The method replaces the conventional learnable classifier head with semantically orthogonal and geometrically uniform target vectors—preconstructed in a low-dimensional space using structured vector systems (e.g., Aₙ root lattices). Coupled with an encoder–ViT architecture, it enables end-to-end training without an explicit classification layer. Evaluated on ImageNet-1K and large-scale datasets containing 500K–600K classes, the approach accelerates convergence by up to 2.3× while compressing the embedding vector repository to less than 10% of that required by standard methods. It thus achieves a favorable trade-off among training efficiency, generalization performance, and deployment practicality.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Computer Vision: Representation Learning for Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The overall neural network (NN) performance is closely related to the properties of its embedding distribution in latent space (LS). It has recently been shown that predefined vector systems, specifically An root system vectors, can be used as targets for latent space configurations (LSC) to ensure the desired LS structure. One of the main LSC advantage is the possibility of training classifier NNs without classification layers, which facilitates training NNs on datasets with extremely large numbers of classes. This paper provides a more general overview of possible vector systems for NN training along with their properties and methods for vector system construction. These systems are used to configure LS of encoders and visual transformers to significantly speed up ImageNet-1K and 50k-600k classes LSC training. It is also shown that using the minimum number of LS dimensions for a specific number of classes results in faster convergence. The latter has potential advantages for reducing the size of vector databases used to store NN embeddings.

Problem

• Explores vector systems for faster neural network training with preconfigured latent spaces.
• Uses vector systems to configure latent spaces, speeding up training on large-class datasets.
• Investigates minimal latent space dimensions for faster convergence and smaller embedding databases.

Innovation

• Predefined vector systems configure latent spaces
• Training classifiers without classification layers enabled
• Minimizing latent dimensions accelerates convergence speed