Vector-ICL: In-context Learning with Continuous Vector Representations

2024-10-08

arXiv.org

AI Summary

This work investigates whether large language models (LLMs) can generalize in-context learning (ICL) capabilities—traditionally confined to discrete text—to continuous vector inputs. To this end, we propose Vector-ICL, a novel paradigm that aligns continuous vectors from multimodal black-box encoders into the LLM’s embedding space via lightweight, learnable projectors, enabling tuning-free, cross-modal vector-level ICL. Our key contribution is demonstrating that standard, language-modeling-pretrained LLMs—combined with modality-agnostic projectors—can achieve zero-shot or few-shot generalization to unseen continuous vectors, bypassing tokenization constraints entirely. Evaluated across eight diverse tasks—including text reconstruction, numerical regression, molecular property prediction, and fMRI decoding—Vector-ICL consistently outperforms conventional few-shot ICL baselines and domain-specific models, substantiating the feasibility and promise of LLMs as universal vector processors.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) have shown remarkable in-context learning (ICL) capabilities on textual data. We explore whether these capabilities can be extended to continuous vectors from diverse domains, obtained from black-box pretrained encoders. By aligning input data with an LLM's embedding space through lightweight projectors, we observe that LLMs can effectively process and learn from these projected vectors, which we term Vector-ICL. In particular, we find that pretraining projectors with general language modeling objectives enables Vector-ICL, while task-specific finetuning further enhances performance. In our experiments across various tasks and modalities, including text reconstruction, numerical function regression, text classification, summarization, molecule captioning, time-series classification, graph classification, and fMRI decoding, Vector-ICL often surpasses both few-shot ICL and domain-specific model or tuning. We further conduct analyses and case studies, indicating the potential of LLMs to process vector representations beyond traditional token-based paradigms.

Problem

• Extending in-context learning to continuous vector representations
• Aligning diverse domain vectors with LLM embedding space
• Enabling LLMs to process vectors beyond token paradigms

Innovation

• Aligning vectors with LLM embedding via lightweight projectors
• Pretraining projectors with general language modeling objectives
• Enhancing performance through task-specific finetuning of projectors