When GPT Spills the Tea: Comprehensive Assessment of Knowledge File Leakage in GPTs

2025-05-30

AI Summary

Prior research on GPT-like LLM agents has focused narrowly on adversarial prompting, overlooking stealthy knowledge file leakage across multi-layered data flows (client–server–database). Method: This work pioneers the integration of Data Security Posture Management (DSPM) into LLM security assessment, establishing an end-to-end knowledge file leakage analysis workflow comprising attack surface modeling, red-teaming, and large-scale data flow analysis (650K+ metadata entries, 11K+ data flows). Contribution/Results: We identify five novel leakage vectors; notably, a privilege escalation vulnerability triggered by Code Interpreter achieves 95.95% raw file download success. Empirical analysis reveals that 28.80% of leaked files contain copyright-protected content. We propose actionable, supply-chain-oriented mitigation strategies for both agent builders and platform providers—advancing both theoretical foundations and practical paradigms for secure knowledge management in LLM agents.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Knowledge files have been widely used in large language model (LLM) agents, such as GPTs, to improve response quality. However, concerns about the potential leakage of knowledge files have grown significantly. Existing studies demonstrate that adversarial prompts can induce GPTs to leak knowledge file content. Yet, it remains uncertain whether additional leakage vectors exist, particularly given the complex data flows across clients, servers, and databases in GPTs. In this paper, we present a comprehensive risk assessment of knowledge file leakage, leveraging a novel workflow inspired by Data Security Posture Management (DSPM). Through the analysis of 651,022 GPT metadata, 11,820 flows, and 1,466 responses, we identify five leakage vectors: metadata, GPT initialization, retrieval, sandboxed execution environments, and prompts. These vectors enable adversaries to extract sensitive knowledge file data such as titles, content, types, and sizes. Notably, the activation of the built-in tool Code Interpreter leads to a privilege escalation vulnerability, enabling adversaries to directly download original knowledge files with a 95.95% success rate. Further analysis reveals that 28.80% of leaked files are copyrighted, including digital copies from major publishers and internal materials from a listed company. In the end, we provide actionable solutions for GPT builders and platform providers to secure the GPT data supply chain.

Problem

• Assessing knowledge file leakage risks in GPTs
• Identifying five vectors for sensitive data extraction
• Proposing solutions to secure GPT data supply chain

Innovation

• DSPM-inspired workflow for leakage assessment
• Identifies five knowledge file leakage vectors
• Code Interpreter enables file download vulnerability