Consistency Deep Equilibrium Models

2026-02-03

AI Summary

This work proposes Consistency Deep Equilibrium Models (C-DEQ), a novel approach that integrates consistency distillation into the deep equilibrium (DEQ) framework to address the inherent trade-off between inference efficiency and accuracy caused by the reliance on iterative solvers with limited steps. By interpreting the iterative process as evolution along the trajectory of an ordinary differential equation toward a fixed point, C-DEQ trains intermediate states to directly map to the equilibrium solution. This design enables flexible few-step inference, substantially reducing computational overhead while achieving 2× to 20× higher accuracy than conventional DEQs across multiple tasks.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Deep Equilibrium Models (DEQs) have emerged as a powerful paradigm in deep learning, offering the ability to model infinite-depth networks with constant memory usage. However, DEQs incur significant inference latency due to the iterative nature of fixed-point solvers. In this work, we introduce the Consistency Deep Equilibrium Model (C-DEQ), a novel framework that leverages consistency distillation to accelerate DEQ inference. We cast the DEQ iterative inference process as evolution along a fixed ODE trajectory toward the equilibrium. Along this trajectory, we train C-DEQs to consistently map intermediate states directly to the fixed point, enabling few-step inference while preserving the performance of the teacher DEQ. At the same time, it facilitates multi-step evaluation to flexibly trade computation for performance gains. Extensive experiments across various domain tasks demonstrate that C-DEQs achieves consistent 2-20$\times$ accuracy improvements over implicit DEQs under the same few-step inference budget.

Problem

• Deep Equilibrium Models
• inference latency
• fixed-point solvers
• iterative inference

Innovation

• Consistency Distillation
• Deep Equilibrium Models
• Fixed-Point Solvers
• ODE Trajectory
• Few-Step Inference