FEnc$^2$: Unifying Data Packing for Efficient Private Inference via Convolution and Architecture-Aware Fragment Encoding

2026-06-15

AI Summary

This work addresses the high computational and memory overheads that hinder the practical deployment of fully homomorphic encryption (FHE) in privacy-preserving machine learning, which stem from the inherent complexity of cryptographic operations and inefficient ciphertext packing. To overcome these limitations, the authors propose FEnc², a fragment-based unified encoding framework that treats encrypted tensor layout as a first-class design dimension in FHE systems. By jointly optimizing spatial locality and feature grouping through convolution-aware encoding and architecture-aware ciphertext compression, FEnc² substantially improves slot utilization while reducing rotation complexity and ciphertext count. Built upon the CKKS scheme and compatible with NTT and key-switching accelerations, FEnc² achieves up to 228.83× (GPU) and 226.06× (CPU) speedup for LeNet inference on MNIST, and 4.55× (GPU) and 9.43× (CPU) acceleration for MobileNet inference on ImageNet.

Abstract

Fully Homomorphic Encryption (FHE) enables privacy-preserving machine learning but incurs extreme computational and memory overhead. These costs come not only from expensive low-level primitives, including Number Theoretic Transform (NTT), rotation, and key-switching, but also from inefficient ciphertext packing at the application level. Existing packing strategies typically preserve either neighboring data elements or feature grouping, but not both, leading to wasted ciphertext slots, excessive rotations, and inflated ciphertext counts. We propose FEnc2, a unified and principled fragment-based encoding framework for CKKS-based private convolutional neural network inference. FEnc2 optimizes slot utilization, rotation complexity, and ciphertext density through two components: 1)Conv-aware Encoding, which analytically selects an optimal fragment size to decouple spatial dependencies and jointly minimize inner-outer rotations across layers, and 2)Arch-aware Ct Compression, which restores ciphertext density after feature- or channel-reduction layers. Together, these transformations reshape encrypted workload structure and reduce homomorphic operations by one to two orders of magnitude. With full memory capacity utilized, i.e., at maximum batch size, FEnc2 achieves end-to-end latency speedups over the state-of-the-art Orion of up to 228.83x on GPU and 226.06x on CPU for LeNet on MNIST, and up to 4.55x on GPU and 9.43x on CPU for MobileNet on ImageNet. FEnc2 is hardware-agnostic yet architecturally transformative: by optimizing encrypted tensor layout before execution, it reduces ciphertext count and workload pressure on hardware, complementing primitive-level optimizations such as NTT and keyswitch accelerators. These results show that application-level data layout is a first-order architectural design dimension for encrypted inference and an important enabler for next-generation FHE systems.

Problem

• Fully Homomorphic Encryption
• Private Inference
• Data Packing
• Ciphertext Encoding
• Convolutional Neural Networks

Innovation

• fragment-based encoding
• convolution-aware packing
• ciphertext compression
• private inference
• Fully Homomorphic Encryption