Contextually Guided Transformers via Low-Rank Adaptation

2025-06-06

AI Summary

Conventional large language models (LLMs) rely on explicit prompts to guide behavior, incurring substantial computational overhead and limiting inference efficiency. Method: We propose a zero-prompt adaptive inference framework that eliminates external prompts entirely. Instead, it dynamically maintains a context summary at each sequence position and enables prefix-driven automatic specialization via learnable weights. Crucially, the framework directly encodes contextual information into model weights—a novel architectural design—combined with variational latent-space regularization and low-rank adaptation (LoRA) to ensure smooth, consistent context representations. It further introduces dynamic weight modulation and context-aware sequence modeling. Contribution/Results: Our method significantly outperforms prompt tuning and in-context learning baselines on both synthetic context-learning tasks and standard language modeling benchmarks. It is the first to empirically demonstrate that efficient, generalizable adaptive inference can be achieved without any prompts, establishing a new paradigm for prompt-free LLM adaptation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large Language Models (LLMs) based on Transformers excel at text processing, but their reliance on prompts for specialized behavior introduces computational overhead. We propose a modification to a Transformer architecture that eliminates the need for explicit prompts by learning to encode context into the model's weights. Our Contextually Guided Transformer (CGT) model maintains a contextual summary at each sequence position, allowing it to update the weights on the fly based on the preceding context. This approach enables the model to self-specialize, effectively creating a tailored model for processing information following a given prefix. We demonstrate the effectiveness of our method on synthetic in-context learning tasks and language modeling benchmarks. Furthermore, we introduce techniques for enhancing the interpretability of the learned contextual representations, drawing connections to Variational Autoencoders and promoting smoother, more consistent context encoding. This work offers a novel direction for efficient and adaptable language modeling by integrating context directly into the model's architecture.

Problem

• Eliminates need for explicit prompts in Transformers
• Encodes context into model weights dynamically
• Enhances interpretability of contextual representations

Innovation

• Contextually Guided Transformer encodes context into weights
• Low-Rank Adaptation enables dynamic weight updates
• Interpretability enhanced via Variational Autoencoder connections