Verifiable Fine-Tuning for LLMs: Zero-Knowledge Training Proofs Bound to Data Provenance and Policy

2025-10-19

AI Summary

Existing parameter-efficient fine-tuning (PEFT) of large language models lacks verifiable guarantees regarding training data provenance, update integrity, and policy compliance. Method: We propose the first end-to-end verifiable PEFT framework, integrating zero-knowledge proofs (ZKPs), data commitment ledgers, verifiable sampling, PEFT-specific arithmetic circuits, semantic constraints for AdamW optimizer behavior, and recursive proof aggregation—executed within a trusted execution environment for millisecond-scale verification. Contribution/Results: Our framework is the first to jointly verify data provenance, sampling privacy, optimizer semantics, and policy quotas under ZKP. It supports federated learning and probabilistic auditing. Experiments on English and bilingual instruction datasets show no utility degradation, zero policy violations, negligible index leakage, and proof generation/verification overheads suitable for practical deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust

Application Category

• Security and Privacy: Data transparency and provenance
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models are often adapted through parameter efficient fine tuning, but current release practices provide weak assurances about what data were used and how updates were computed. We present Verifiable Fine Tuning, a protocol and system that produces succinct zero knowledge proofs that a released model was obtained from a public initialization under a declared training program and an auditable dataset commitment. The approach combines five elements. First, commitments that bind data sources, preprocessing, licenses, and per epoch quota counters to a manifest. Second, a verifiable sampler that supports public replayable and private index hiding batch selection. Third, update circuits restricted to parameter efficient fine tuning that enforce AdamW style optimizer semantics and proof friendly approximations with explicit error budgets. Fourth, recursive aggregation that folds per step proofs into per epoch and end to end certificates with millisecond verification. Fifth, provenance binding and optional trusted execution property cards that attest code identity and constants. On English and bilingual instruction mixtures, the method maintains utility within tight budgets while achieving practical proof performance. Policy quotas are enforced with zero violations, and private sampling windows show no measurable index leakage. Federated experiments demonstrate that the system composes with probabilistic audits and bandwidth constraints. These results indicate that end to end verifiable fine tuning is feasible today for real parameter efficient pipelines, closing a critical trust gap for regulated and decentralized deployments.

Problem

• Ensuring verifiable data provenance and policy compliance in LLM fine-tuning
• Providing zero-knowledge proofs for training process integrity and auditability
• Closing trust gaps in regulated and decentralized model deployments

Innovation

• Zero-knowledge proofs verify model training process
• Commitments bind data sources to training manifests
• Recursive aggregation enables efficient proof verification