Asynchronous Perception Machine For Efficient Test-Time-Training

2024-10-27

Neural Information Processing Systems

AI Summary

This work addresses key limitations of test-time training (TTT)—including high computational overhead, reliance on pretraining or data augmentation, and poor generalization to out-of-distribution (OOD) images—by proposing the Asynchronous Perceptron Machine (APM), a lightweight, pretraining-free architecture requiring no proxy tasks or data augmentation. APM processes image patches in arbitrary order and asynchronously per block, enabling semantic distillation via a novel single-representation learning mechanism that yields semantic-aware representations in a single forward pass. It provides the first empirical validation of Hinton’s GLOM hypothesis that “input perception is a field.” Moreover, APM supports single-pass semantic clustering and enables hardware co-design for interpolation-aware, asynchronous inference. Empirically, it matches state-of-the-art TTT methods on OOD detection and achieves zero-shot semantic clustering on 2D image benchmarks. The code is publicly available.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Learning to Search
• Computer Vision: Representation Learning for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

In this work, we propose Asynchronous Perception Machine (APM), a computationally-efficient architecture for test-time-training (TTT). APM can process patches of an image one at a time in any order asymmetrically and still encode semantic-awareness in the net. We demonstrate APM's ability to recognize out-of-distribution images without dataset-specific pre-training, augmentation or any-pretext task. APM offers competitive performance over existing TTT approaches. To perform TTT, APM just distills test sample's representation once. APM possesses a unique property: it can learn using just this single representation and starts predicting semantically-aware features. APM demostrates potential applications beyond test-time-training: APM can scale up to a dataset of 2D images and yield semantic-clusterings in a single forward pass. APM also provides first empirical evidence towards validating GLOM's insight, i.e. input percept is a field. Therefore, APM helps us converge towards an implementation which can do both interpolation and perception on a shared-connectionist hardware. Our code is publicly available at this link: https://rajatmodi62.github.io/apm_project_page/.

Problem

• Processes image patches asynchronously for efficient test-time training
• Recognizes out-of-distribution images without dataset-specific pre-training
• Enables semantic clustering of 2D images in single forward pass

Innovation

• Asynchronous patch processing for semantic encoding
• Single-representation distillation for test-time training
• Semantic clustering without dataset-specific pre-training