Black-box Optimization of LLM Outputs by Asking for Directions

2025-10-19

AI Summary

This work addresses adversarial attacks against black-box large language models (LLMs) that expose only textual outputs—no logits, confidence scores, or model parameters are accessible. We propose a novel attack paradigm that leverages the model’s own natural-language confidence expressions (e.g., “I am highly confident,” “this may be inaccurate”) as surrogate signals for gradient estimation and optimization. By modeling such linguistic confidence feedback as differentiable proxies and integrating prompt engineering, reinforcement learning, and adaptive search strategies, our method enables efficient adversarial search solely via text-based interfaces. It unifies application across vision-language model evasion, jailbreaking, and prompt injection. Experiments on diverse mainstream black-box LLMs demonstrate substantially improved attack success rates. Notably, we uncover a security paradox: stronger models—with richer, more nuanced confidence articulation—are empirically more vulnerable to this attack. The implementation is publicly available.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Adversarial Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We present a novel approach for attacking black-box large language models (LLMs) by exploiting their ability to express confidence in natural language. Existing black-box attacks require either access to continuous model outputs like logits or confidence scores (which are rarely available in practice), or rely on proxy signals from other models. Instead, we demonstrate how to prompt LLMs to express their internal confidence in a way that is sufficiently calibrated to enable effective adversarial optimization. We apply our general method to three attack scenarios: adversarial examples for vision-LLMs, jailbreaks and prompt injections. Our attacks successfully generate malicious inputs against systems that only expose textual outputs, thereby dramatically expanding the attack surface for deployed LLMs. We further find that better and larger models exhibit superior calibration when expressing confidence, creating a concerning security paradox where model capability improvements directly enhance vulnerability. Our code is available at this [link](https://github.com/zj-jayzhang/black_box_llm_optimization).

Problem

• Exploits LLMs' natural language confidence expression for black-box optimization
• Generates adversarial examples without requiring continuous model outputs
• Demonstrates improved model calibration increases vulnerability to attacks

Innovation

• Prompting LLMs to express calibrated internal confidence
• Enabling adversarial optimization via natural language confidence
• Expanding attack surface for text-only output systems