Exploring the Hierarchical Reasoning Model for Small Natural-Image Classification Without Augmentation

2025-10-03

AI Summary

This study investigates the feasibility of hierarchical reasoning models (HRMs) for classifying small natural image datasets—MNIST, CIFAR-10, and CIFAR-100—without data augmentation. We propose a novel HRM architecture integrating dual Transformer modules (fₗ, fₕ), deep supervision, rotary position embedding (RoPE), RMSNorm, and a DEQ-inspired single-step implicit training scheme optimized via cosine learning rate scheduling and label smoothing. The design prioritizes training stability and generalization while maintaining lightweight structure. Experiments show HRM achieves 98.0% accuracy on MNIST but only 65.0% and 29.7% on CIFAR-10 and CIFAR-100, respectively—substantially underperforming lightweight CNNs—revealing critical overfitting and generalization bottlenecks on complex natural images. Our core contribution is a systematic empirical validation of implicit hierarchical reasoning in the no-augmentation regime, clarifying both its promise and fundamental limitations for such settings.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Representation Learning for Vision
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

This paper asks whether the Hierarchical Reasoning Model (HRM) with the two Transformer-style modules $(f_L,f_H)$, one step (DEQ-style) training, deep supervision, Rotary Position Embeddings, and RMSNorm can serve as a practical image classifier. It is evaluated on MNIST, CIFAR-10, and CIFAR-100 under a deliberately raw regime: no data augmentation, identical optimizer family with one-epoch warmup then cosine-floor decay, and label smoothing. HRM optimizes stably and performs well on MNIST ($approx 98%$ test accuracy), but on small natural images it overfits and generalizes poorly: on CIFAR-10, HRM reaches 65.0% after 25 epochs, whereas a two-stage Conv--BN--ReLU baseline attains 77.2% while training $sim 30 imes$ faster per epoch; on CIFAR-100, HRM achieves only 29.7% test accuracy despite 91.5% train accuracy, while the same CNN reaches 45.3% test with 50.5% train accuracy. Loss traces and error analyses indicate healthy optimization but insufficient image-specific inductive bias for HRM in this regime. It is concluded that, for small-resolution image classification without augmentation, HRM is not competitive with even simple convolutional architectures as the HRM currently exist but this does not exclude possibilities that modifications to the model may allow it to improve greatly.

Problem

• Evaluating HRM's effectiveness for small natural-image classification without data augmentation
• Comparing HRM performance against simple CNN baselines on MNIST and CIFAR datasets
• Investigating HRM's overfitting issues and insufficient inductive biases for natural images

Innovation

• Hierarchical Reasoning Model with Transformer modules
• One step DEQ-style training with deep supervision
• Rotary Position Embeddings and RMSNorm techniques