Evaluating LLM-Generated Legal Explanations for Regulatory Compliance in Social Media Influencer Marketing

2025-10-09

AI Summary

Regulatory oversight of sponsored content transparency in influencer marketing remains challenging due to the lack of legally grounded, interpretable detection methods. Method: This study proposes a legal-knowledge-driven large language model (LLM) regulatory framework. It introduces the first compliance dataset for influencer marketing, annotated by law students; establishes a taxonomy of LLM legal reasoning errors; and integrates statutory text into prompt engineering to systematically evaluate explanation quality. Experiments employ GPT-5-nano and Gemini-2.5-flash-lite with three prompting strategies. Contribution/Results: The framework achieves an F1-score of 0.93 on classification tasks. It identifies recurrent explanatory deficiencies—e.g., omitted citations and ambiguous references—and demonstrates that statutory text injection significantly enhances legal interpretability, though detection accuracy improves only marginally. Crucially, it establishes a verifiable, attributable, and rule-of-law–compliant foundation for automated regulatory enforcement.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: AI & Law, Justice, Regulation & Governance

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The rise of influencer marketing has blurred boundaries between organic content and sponsored content, making the enforcement of legal rules relating to transparency challenging. Effective regulation requires applying legal knowledge with a clear purpose and reason, yet current detection methods of undisclosed sponsored content generally lack legal grounding or operate as opaque "black boxes". Using 1,143 Instagram posts, we compare gpt-5-nano and gemini-2.5-flash-lite under three prompting strategies with controlled levels of legal knowledge provided. Both models perform strongly in classifying content as sponsored or not (F1 up to 0.93), though performance drops by over 10 points on ambiguous cases. We further develop a taxonomy of reasoning errors, showing frequent citation omissions (28.57%), unclear references (20.71%), and hidden ads exhibiting the highest miscue rate (28.57%). While adding regulatory text to the prompt improves explanation quality, it does not consistently improve detection accuracy. The contribution of this paper is threefold. First, it makes a novel addition to regulatory compliance technology by providing a taxonomy of common errors in LLM-generated legal reasoning to evaluate whether automated moderation is not only accurate but also legally robust, thereby advancing the transparent detection of influencer marketing content. Second, it features an original dataset of LLM explanations annotated by two students who were trained in influencer marketing law. Third, it combines quantitative and qualitative evaluation strategies for LLM explanations and critically reflects on how these findings can support advertising regulatory bodies in automating moderation processes on a solid legal foundation.

Problem

• Evaluating LLM-generated legal explanations for regulatory compliance in influencer marketing
• Assessing legal reasoning errors in automated detection of undisclosed sponsored content
• Developing transparent AI moderation systems with solid legal foundation for advertising

Innovation

• Using LLMs to classify sponsored content on Instagram
• Developing taxonomy of legal reasoning errors in AI
• Combining quantitative and qualitative evaluation methods