Training Foundation Models as Data Compression: On Information, Model Weights and Copyright Law

2024-07-18

arXiv.org

AI Summary

This paper addresses the core legal question of whether large language model weights constitute copyright-infringing copies or derivative works of training data. Methodologically, it introduces the novel “training-as-compression” theoretical framework, modeling weights as information-theoretic compressions of training data and integrating entropy, reconstruction fidelity, deep learning training dynamics, and copyright law principles. It rigorously demonstrates— for the first time—that model weights may satisfy the legal criteria for copyright infringement under specific conditions, particularly when they enable high-fidelity reconstruction of copyrighted material. Based on this insight, the paper proposes a dual-dimensional copyright risk assessment methodology grounded in information entropy and reconstruction fidelity. The resulting paradigm provides a rigorous, actionable foundation for adjudicating AI-generated content ownership, designing copyright-compliant training protocols, and informing evidence-based copyright policy development in the age of foundation models.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Large Vision Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

The training process of foundation models as for other classes of deep learning systems is based on minimizing the reconstruction error over a training set. For this reason, they are susceptible to the memorization and subsequent reproduction of training samples. In this paper, we introduce a training-as-compressing perspective, wherein the model's weights embody a compressed representation of the training data. From a copyright standpoint, this point of view implies that the weights could be considered a reproduction or a derivative work of a potentially protected set of works. We investigate the technical and legal challenges that emerge from this framing of the copyright of outputs generated by foundation models, including their implications for practitioners and researchers. We demonstrate that adopting an information-centric approach to the problem presents a promising pathway for tackling these emerging complex legal issues.

Problem

• Explores copyright implications of foundation models' data compression.
• Investigates if model weights constitute derivative works under copyright law.
• Proposes information-centric solutions for legal challenges in model outputs.

Innovation

• Training models as data compression technique
• Model weights as compressed data representation
• Information-centric approach for legal challenges