🤖 AI Summary
Copyright protection and provenance tracking for large language models (LLMs) remain challenging, particularly under black-box access constraints. Method: This paper proposes HuRef—a human-readable, parameter-agnostic fingerprinting technique for base model identification. HuRef introduces the first formal definition and extraction of three parameter-direction invariants in Transformer architectures that are robust to fine-tuning and RLHF yet sensitive to adversarial perturbations. It encodes fingerprints as natural images synthesized by StyleGAN2 and augments them with zero-knowledge proofs to ensure verifiable authenticity upon release. Contribution/Results: Extensive experiments demonstrate HuRef’s uniqueness and robustness across multiple LLMs. It withstands structural attacks—including dimension permutation and weight matrix rotation—while enabling reliable copyright attribution in black-box settings, thus establishing a practical, trustworthy framework for LLM intellectual property protection.
📝 Abstract
Protecting the copyright of large language models (LLMs) has become crucial due to their resource-intensive training and accompanying carefully designed licenses. However, identifying the original base model of an LLM is challenging due to potential parameter alterations. In this study, we introduce HuRef, a human-readable fingerprint for LLMs that uniquely identifies the base model without interfering with training or exposing model parameters to the public. We first observe that the vector direction of LLM parameters remains stable after the model has converged during pretraining, with negligible perturbations through subsequent training steps, including continued pretraining, supervised fine-tuning, and RLHF, which makes it a sufficient condition to identify the base model. The necessity is validated by continuing to train an LLM with an extra term to drive away the model parameters' direction and the model becomes damaged. However, this direction is vulnerable to simple attacks like dimension permutation or matrix rotation, which significantly change it without affecting performance. To address this, leveraging the Transformer structure, we systematically analyze potential attacks and define three invariant terms that identify an LLM's base model. Due to the potential risk of information leakage, we cannot publish invariant terms directly. Instead, we map them to a Gaussian vector using an encoder, then convert it into a natural image using StyleGAN2, and finally publish the image. In our black-box setting, all fingerprinting steps are internally conducted by the LLMs owners. To ensure the published fingerprints are honestly generated, we introduced Zero-Knowledge Proof (ZKP). Experimental results across various LLMs demonstrate the effectiveness of our method. The code is available at https://github.com/LUMIA-Group/HuRef.