Do Prompts Reshape Representations? An Empirical Study of Prompting Effects on Embeddings

2025-10-22

AI Summary

This study investigates how prompting affects the quality of internal representations in large language models (LLMs) during zero-shot classification, and how this relates to prompt-task relevance. Methodologically, we construct diverse prompt templates and employ representation probing to systematically assess their impact on the separability and semantic structure of hidden-layer embeddings. Our results reveal that prompting substantially reshapes model representations; however, representation quality does not monotonically improve with increasing prompt-task relevance—in fact, highly relevant prompts sometimes degrade performance. This finding challenges the implicit assumption that “more relevant prompts yield better representations,” exposing the non-intuitive nature of prompt mechanisms in in-context learning. The work provides novel theoretical insights and empirical evidence for understanding zero-shot generalization in LLMs, highlighting the complex, non-linear relationship between prompt design, internal representation geometry, and downstream task performance.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Representation Learning for Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Prompting is a common approach for leveraging LMs in zero-shot settings. However, the underlying mechanisms that enable LMs to perform diverse tasks without task-specific supervision remain poorly understood. Studying the relationship between prompting and the quality of internal representations can shed light on how pre-trained embeddings may support in-context task solving. In this empirical study, we conduct a series of probing experiments on prompt embeddings, analyzing various combinations of prompt templates for zero-shot classification. Our findings show that while prompting affects the quality of representations, these changes do not consistently correlate with the relevance of the prompts to the target task. This result challenges the assumption that more relevant prompts necessarily lead to better representations. We further analyze potential factors that may contribute to this unexpected behavior.

Problem

• Investigates how prompting affects language model embedding representations
• Examines if relevant prompts consistently improve representation quality
• Analyzes factors behind unexpected prompt-representation quality relationships

Innovation

• Probing experiments analyze prompt embedding effects
• Prompt relevance does not consistently improve representations
• Empirical study challenges assumptions about prompt quality