Efficient Knowledge Feeding to Language Models: A Novel Integrated Encoder-Decoder Architecture

2025-02-07

AI Summary

This work addresses two key limitations: (1) retrieval-augmented generation (RAG) suffers from token-length constraints and retrieval inaccuracy, and (2) in-context learning (ICL) relies heavily on numerous annotated examples. To overcome these, we propose the In-Context Vector (ICV) mechanism—a task-aware, encoder-decoder-based approach that encodes external knowledge into compact latent vectors and directly modulates the internal hidden states of large language models (LLMs), enabling example-free, low-overhead knowledge injection. Crucially, ICV replaces explicit prompts with implicit vector representations, eliminating dependence on external retrieval systems and lengthy prompts. Experiments demonstrate that ICV significantly outperforms standard ICL and fine-tuning baselines on question answering and information retrieval tasks. It achieves competitive performance with minimal parameter overhead—orders of magnitude smaller than LLaMA-3, Gemma, or Phi-3—and substantially reduces computational cost, memory footprint, and input sequence length.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This paper introduces a novel approach to efficiently feeding knowledge to language models (LLMs) during prediction by integrating retrieval and generation processes within a unified framework. While the Retrieval-Augmented Generation (RAG) model addresses gaps in LLMs' training data and knowledge limits, it is hindered by token limit restrictions and dependency on the retrieval system's accuracy. Our proposed architecture incorporates in-context vectors (ICV) to overcome these challenges. ICV recasts in-context learning by using latent embeddings of LLMs to create a vector that captures essential task information. This vector is then used to shift the latent states of the LLM, enhancing the generation process without adding demonstration examples to the prompt. ICV directly integrates information into the model, enabling it to process this information more effectively. Our extensive experimental evaluation demonstrates that ICV outperforms standard in-context learning and fine-tuning across question-answering, information retrieval, and other tasks. This approach mitigates the limitations of current RAG models and offers a more robust solution for handling extensive and diverse datasets. Despite leveraging a fraction of the parameters, our ICV-enhanced model achieves competitive performance against models like LLaMA-3, Gemma, and Phi-3, significantly reducing computational costs and memory requirements. ICV reduces prompt length, is easy to control, surpasses token limitations, and is computationally efficient compared to fine-tuning.

Problem

• Enhance knowledge integration in language models
• Overcome token limits in retrieval-augmented generation
• Reduce computational costs in model predictions

Innovation

• Integrated Encoder-Decoder Architecture
• In-Context Vectors (ICV)
• Efficient Knowledge Feeding