Asking Back: Interaction-Layer Antidistillation Watermarks

2026-05-15

AI Summary

This work addresses the vulnerability of existing watermarking methods to knowledge distillation and paraphrasing attacks, which often exploit dependencies on output tokens or internal model structures and lack control over the attacker’s training process. The authors propose the first approach that embeds watermarks into the interactive behavioral layer of large language models by intermittently prompting the model—via system instructions—to exhibit specific behavioral markers, such as follow-up questions or low-frequency linguistic variants. These markers are unintentionally inherited by distilled models under black-box conditions. Evaluated across 63 LoRA-based distilled models using LLM-as-judge auditing (Cohen’s κ = 0.84/0.78) and non-parametric statistical tests, the method achieves behavioral fidelity rates of 45.2%–88.9%. Notably, against DIPPER paraphrasing attacks, certain distilled models (e.g., OLMo) retain watermarks at rates exceeding those of the teacher model, while user studies indicate negligible impact on user experience (<0.22 Likert scale points).

Abstract

Detecting unauthorized knowledge distillation from a deployed LLM API is hard because the defender controls neither the attacker's training pipeline nor the next-token logits. Existing defenses operate on the teacher's output tokens -- biasing the next-token distribution (green-list watermarks, cryptographic schemes, antidistillation sampling) or rewriting outputs after generation. Recent work shows a paraphrasing attacker can strip these signals without losing the underlying knowledge. We propose interaction-layer antidistillation watermarks, which move the trace one layer higher, into the teacher's interaction behavior: the defender wraps the teacher with a system prompt that intermittently induces a behavioral marker -- an explicit follow-up question, a low-frequency variant, or a declarative restatement. An oblivious distiller inherits the behavior, and the defender audits via black-box queries with a human-validated LLM-as-judge (Cohen's kappa = 0.84/0.78 on strong/style rubrics). Across 63 LoRA-distilled students under a Llama-3.3-70B-Instruct teacher (35,343 judged samples), behavioral watermarks transfer at 88.9% (Gemma) / 80.9% (OLMo) / 45.2% (Qwen) relative fidelity (H1, H2). Under non-adaptive DIPPER paraphrasing, robustness decomposes into a teacher-self ceiling (about 66.4%) and student-relative retention of 21-112%, with OLMo preserving the watermark above the teacher itself (H3, F-Amp). Low-density (about 20%) explicit and implicit declarative variants transfer above per-family baseline (H4, F-Style). An N=20 in-lab study (pre-registered Latin-square) shows all marker variants within 0.22 Likert step of baseline; TOST, Friedman, and Bonferroni-Wilcoxon support H5. The interaction layer is a viable design locus for antidistillation watermarking, complementary to token-, model-, and reasoning-trace-layer defenses.

Problem

• knowledge distillation
• watermarking
• large language models
• model stealing
• API security

Innovation

• interaction-layer watermarking
• antidistillation
• behavioral markers
• LLM-as-judge
• knowledge distillation detection