Understanding Learner-LLM Chatbot Interactions and the Impact of Prompting Guidelines

2025-04-10

AI Summary

Learners’ limited prompt engineering proficiency impedes effective LLM interaction in educational settings. Method: We propose the first learner-centered, structured prompt guidance framework and extend the Von NeuMidas pragmatic annotation scheme to enable fine-grained attribution of interactive errors. A controlled experiment (N = 642 authentic dialogues) compares three prompt guidance strategies. Contribution/Results: The task-oriented, self-constructed guidance framework significantly improves prompt precision (+42%) and strategy adherence (+57%), while enhancing AI response relevance (+31%) and practicality (+28%). This study provides the first empirical causal evidence linking prompt instruction to learners’ strategy adoption and output quality. It yields a transferable pedagogical intervention paradigm and assessment toolkit for cultivating AI literacy.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Large Language Models (LLMs) have transformed human-computer interaction by enabling natural language-based communication with AI-powered chatbots. These models are designed to be intuitive and user-friendly, allowing users to articulate requests with minimal effort. However, despite their accessibility, studies reveal that users often struggle with effective prompting, resulting in inefficient responses. Existing research has highlighted both the limitations of LLMs in interpreting vague or poorly structured prompts and the difficulties users face in crafting precise queries. This study investigates learner-AI interactions through an educational experiment in which participants receive structured guidance on effective prompting. We introduce and compare three types of prompting guidelines: a task-specific framework developed through a structured methodology and two baseline approaches. To assess user behavior and prompting efficacy, we analyze a dataset of 642 interactions from 107 users. Using Von NeuMidas, an extended pragmatic annotation schema for LLM interaction analysis, we categorize common prompting errors and identify recurring behavioral patterns. We then evaluate the impact of different guidelines by examining changes in user behavior, adherence to prompting strategies, and the overall quality of AI-generated responses. Our findings provide a deeper understanding of how users engage with LLMs and the role of structured prompting guidance in enhancing AI-assisted communication. By comparing different instructional frameworks, we offer insights into more effective approaches for improving user competency in AI interactions, with implications for AI literacy, chatbot usability, and the design of more responsive AI systems.

Problem

• Investigates learner-AI interactions and prompting efficacy
• Compares structured prompting guidelines for better AI responses
• Analyzes user behavior to improve AI communication effectiveness

Innovation

• Structured prompting guidelines for effective queries
• Von NeuMidas schema for interaction analysis
• Comparative evaluation of prompting frameworks