Position: Vector Prompt Interfaces Should Be Exposed to Enable Customization of Large Language Models

2026-03-04

AI Summary

This work addresses the limitations of large language models in practical deployment, where textual prompts often fail to enable efficient, stable, and inference-only customization. To overcome this, the paper proposes opening vector prompts as a standardized user interface, establishing a novel customization paradigm. Through vector prompt tuning, attention mechanism analysis, and security evaluation under black-box threat models, experiments demonstrate that vector prompts consistently improve performance with enhanced supervision signals, whereas textual prompts saturate early. Moreover, vector prompts induce globally dense attention patterns, revealing superior controllability and greater potential for model customization compared to conventional textual prompting.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

As large language models (LLMs) transition from research prototypes to real-world systems, customization has emerged as a central bottleneck. While text prompts can already customize LLM behavior, we argue that text-only prompting does not constitute a suitable control interface for scalable, stable, and inference-only customization. This position paper argues that model providers should expose \emph{vector prompt inputs} as part of the public interface for customizing LLMs. We support this position with diagnostic evidence showing that vector prompt tuning continues to improve with increasing supervision whereas text-based prompt optimization saturates early, and that vector prompts exhibit dense, global attention patterns indicative of a distinct control mechanism. We further discuss why inference-only customization is increasingly important under realistic deployment constraints, and why exposing vector prompts need not fundamentally increase model leakage risk under a standard black-box threat model. We conclude with a call to action for the community to rethink prompt interfaces as a core component of LLM customization.

Problem

• customization
• large language models
• prompt interfaces
• vector prompts
• inference-only customization

Innovation

• vector prompt
• LLM customization
• inference-only adaptation
• prompt interface
• black-box deployment