Train Yourself as an LLM: Exploring Effects of AI Literacy on Persuasion via Role-playing LLM Training

2026-04-02

AI Summary

This study addresses the growing persuasive power of large language models (LLMs) and its potential risks to public decision-making, noting that existing interventions often treat users as passive recipients. To counter this, the authors propose LLMimic—an interactive, role-playing AI literacy tutorial that, for the first time, leverages gamification to immerse users in simulating the three core stages of LLM training: pretraining, supervised fine-tuning, and reinforcement learning from human feedback. Designed as an interactive web-based platform grounded in educational psychology and AI principles, LLMimic enables users to actively engage with LLMs from the perspective of a model trainer. In a controlled experiment with 274 participants, LLMimic significantly improved AI literacy (p<0.001), reduced susceptibility to AI persuasion (p<0.05), and enhanced the authenticity and social responsibility of user responses in a hotel recommendation task (p<0.01).

Abstract

As large language models (LLMs) become increasingly persuasive, there is concern that people's opinions and decisions may be influenced across various contexts at scale. Prior mitigation (e.g., AI detectors and disclaimers) largely treats people as passive recipients of AI-generated information. To provide a more proactive intervention against persuasive AI, we introduce $\textbf{LLMimic}$, a role-play-based, interactive, gamified AI literacy tutorial, where participants assume the role of an LLM and progress through three key stages of the training pipeline (pretraining, SFT, and RLHF). We conducted a $2 \times 3$ between-subjects study ($N = 274$) where participants either (1) watched an AI history video (control) or (2) interacted with LLMimic (treatment), and then engaged in one of three realistic AI persuasion scenarios: (a) charity donation persuasion, (b) malicious money solicitation, or (c) hotel recommendation. Our results show that LLMimic significantly improved participants' AI literacy ($p < .001$), reduced persuasion success across scenarios ($p < .05$), and enhanced truthfulness and social responsibility levels ($p<0.01$) in the hotel scenario. These findings suggest that LLMimic offers a scalable, human-centered approach to improving AI literacy and supporting more informed interactions with persuasive AI.

Problem

• persuasive AI
• AI literacy
• large language models
• human-AI interaction
• AI influence

Innovation

• role-playing
• AI literacy
• persuasive AI
• gamified tutorial
• human-centered intervention