Shuffling is Not Enough: Breaking Permutation-Based Model Confidentiality in Hybrid FHE Inference

2026-09-11

AI Summary

本文指出混合FHE推理中仅通过乱序不足以保护模型机密性，使用d+1次查询即可精确恢复线性层，导致模型完全可区分。

Abstract

Hybrid fully homomorphic encryption~(FHE) inference improves the practicality of private inference by letting the server evaluate linear layers homomorphically while the client decrypts and applies nonlinearities. Recent schemes attempt to protect model confidentiality by returning noisy, output-permuted responses and appealing to shuffle-model differential privacy~(DP). We show that this protection fails in the correctness regime required by hybrid FHE systems. For a $d$-input linear layer, $d+1$ admissible queries suffice for exact recovery of a permutation-invariant layer summary, hence for perfect model distinguishability. We further show that input DP is orthogonal to model confidentiality and that the local-DP premise required for shuffle amplification cannot hold under correctness-bounded noise. We recover all linear layers of a \safhire{}-style ResNet-20 end-to-end from TFHE transcripts with zero error, using $d+1$ queries per layer for a total of $5{,}712$ direct queries. Under the same query model, we also confirm exact per-layer recovery on pretrained ImageNet-scale CNNs and ViT-B/16. The leaked spectra enable fingerprinting, lineage attribution, and improved logit-based extraction, while suppressing them destroys inference utility.

Problem

• Hybrid FHE
• Model Confidentiality
• Permutation-based Protection
• Differential Privacy
• Linear Layer Recovery

Innovation

• Hybrid FHE
• Permutation-based Confidentiality
• Model Distinguishability
• Shuffle Model Differential Privacy
• Linear Layer Recovery