What Does It Mean to Break a Distillation Defense?

2026-06-23

AI Summary

Black-box large language models are vulnerable to distillation attacks, yet existing defenses lack a unified threat model, hindering objective security evaluation. This work presents the first systematic three-dimensional threat model for distillation defense, characterizing adversary capabilities through query budget, data budget, and interface interaction modality. Focusing on output perturbation–based defenses, the study integrates formal modeling, adversarial evaluation, and case studies to rigorously assess robustness. It reveals that the effectiveness of a given defense varies dramatically under different threat assumptions—ranging from trivially bypassed to highly resilient—highlighting the critical importance of explicitly defining and stress-testing adversary capability dimensions. These findings establish a reproducible benchmark for future research and policy development in securing black-box language models against distillation attacks.

Abstract

Black-box LLMs (accessible only via API) are vulnerable to distillation attacks, in which an attacker queries the model and trains a student on its outputs. A recent line of work proposes output perturbation defenses that modify the teacher's output to reduce student performance while preserving utility for legitimate users. As a relatively new family of approaches, output perturbation defenses lack a shared threat model, making it difficult to compare them, reason about composing them with other attacks, or evaluate their robustness against realistic adversaries. This underspecification matters beyond technical evaluation: when defenses are deployed to protect intellectual property or justify regulatory compliance, an imprecise threat model can create a false sense of security. We propose a threat model framework that describes attackers along three dimensions: a query budget, a data budget, and an interface profile that captures how attackers interact with the API. Using antidistillation sampling as a case study, we show that whether the defense is considered effective depends on the assumed threat model. We argue that future work on distillation defenses, along with any governance or policy frameworks built around them, should explicitly specify and stress-test attacker capabilities along our three dimensions.

Problem

• distillation defense
• threat model
• output perturbation
• black-box LLMs
• adversarial robustness

Innovation

• distillation defense
• threat model
• output perturbation
• black-box LLM
• antidistillation sampling