What Survives When You Compress a Recursive Reasoner for the Edge?

2026-06-24

AI Summary

This work addresses the challenge of deploying recursive reasoning models on edge devices, where quantization errors accumulate across recurrent steps, leading to catastrophic degradation in global reasoning capability—a failure mode that conventional compression techniques cannot mitigate. Through systematic analysis under extreme compression, the study reveals that while local prediction capacity remains intact, global consistency is lost. To remedy this, the authors propose a retraining-free channel-wise calibration method for INT4 quantization and introduce an unsupervised metric to assess inference trajectory fidelity. Integrating flash-aware streaming embeddings with linear attention, their approach achieves full-depth accuracy at INT8 with a 6× reduction in FLOPs—compatible with 8MB SoCs—and enables INT4 deployment on 4MB microcontrollers. The embedding compression alone eliminates a 99.4MB storage bottleneck.

Abstract

Recursive reasoning models can solve complex structured tasks with only a few million parameters by repeatedly updating a latent state. Deploying these models on edge hardware requires significant compression, but unlike conventional sequence models, quantization errors compound across recursive reasoning cycles rather than across output tokens. As a result, standard intuitions about compression fail to apply. In this work, we ask what survives when recursive reasoners are compressed. Across a full precision sweep, three tasks, and two recursive architectures, we find that aggressive compression preserves local prediction but destroys global reasoning: cell accuracy holds while puzzle-exact accuracy collapses to zero under naive INT4 pruning, distillation, and linear attention alike. Token-level objectives, including quantization-aware training, cannot repair it. The collapse is architectural -- it strikes MLP-mixing recursion but not attention on the same task -- and we reverse it with per-channel calibrated INT4 without retraining. We also introduce carry-trajectory fidelity, the cosine similarity to the full-precision reasoning path, as a label-free signal that predicts this damage and its recovery before a task evaluation. The combined result is a deployment recipe: flash-streamed embeddings remove a 99.4MB bottleneck, INT8 at one cycle matches full-depth accuracy at 6x fewer FLOPs (8MB SoC), and calibrated INT4 fits a 4MB microcontroller.

Problem

• recursive reasoning
• model compression
• quantization error
• edge deployment
• global reasoning collapse

Innovation

• recursive reasoning
• quantization error compounding
• carry-trajectory fidelity
• per-channel calibrated INT4
• edge deployment