NRGPT: An Energy-based Alternative for GPT

2025-12-18

AI Summary

This work addresses the long-standing disjunction between generative pre-trained transformers (GPTs) and energy-based modeling (EBM). We propose the first lightweight, unified GPT-EBM framework. Methodologically, we augment a standard GPT backbone with a learnable energy function, recasting token generation as implicit gradient-driven dynamical exploration over an energy landscape—while preserving the original autoregressive training procedure. We theoretically establish that, under mild conditions, inference corresponds to gradient descent on the learned energy surface. Key contributions include: (i) the first structured integration of GPT and EBM architectures; (ii) significantly delayed overfitting during long-horizon training; (iii) state-of-the-art or competitive performance on Shakespeare, ListOPS, and OpenWebText benchmarks; and (iv) interpretable inference trajectories via explicit energy evolution. The framework bridges expressive generative modeling with principled energy-based reasoning, enabling both improved generalization and transparent decoding dynamics.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Generative Pre-trained Transformer (GPT) architectures are the most popular design for language modeling. Energy-based modeling is a different paradigm that views inference as a dynamical process operating on an energy landscape. We propose a minimal modification of the GPT setting to unify it with the EBM framework. The inference step of our model, which we call eNeRgy-GPT (NRGPT), is conceptualized as an exploration of the tokens on the energy landscape. We prove, and verify empirically, that under certain circumstances this exploration becomes gradient descent, although they don't necessarily lead to the best performing models. We demonstrate that our model performs well for simple language (Shakespeare dataset), algebraic ListOPS tasks, and richer settings such as OpenWebText language modeling. We also observe that our models may be more resistant to overfitting, doing so only during very long training.

Problem

• Unify GPT with energy-based modeling framework
• Explore tokens on energy landscape during inference
• Test model on language and algebraic tasks

Innovation

• Energy-based modeling modifies GPT architecture
• Inference conceptualized as token exploration on energy landscape
• Demonstrated performance across diverse language tasks