Exploring the Impact of Instruction-Tuning on LLM's Susceptibility to Misinformation

2025-07-24

AI Summary

This study investigates whether instruction tuning increases large language models’ (LLMs) susceptibility to misinformation. While prior work primarily examines conflicts within models’ internal knowledge, it overlooks how instruction tuning affects credibility assessment of user-provided information. To address this gap, we conduct controlled comparative experiments evaluating instruction-tuned models against their base counterparts when exposed to contradictory or false user inputs. Results demonstrate that instruction tuning significantly amplifies model reliance on user input, thereby increasing adoption of erroneous information; this effect is modulated by prompt structure, length of misleading content, and presence of system warnings. Our key contribution is the first empirical identification that instruction tuning shifts “misbelief risk” from the model’s parametric knowledge to user-supplied inputs. We further propose a reliability evaluation framework grounded in prompt robustness analysis, enabling systematic assessment of model trustworthiness under adversarial or ambiguous user prompts.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Application Domains: Misinformation & Fake News

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Instruction-tuning enhances the ability of large language models (LLMs) to follow user instructions more accurately, improving usability while reducing harmful outputs. However, this process may increase the model's dependence on user input, potentially leading to the unfiltered acceptance of misinformation and the generation of hallucinations. Existing studies primarily highlight that LLMs are receptive to external information that contradict their parametric knowledge, but little research has been conducted on the direct impact of instruction-tuning on this phenomenon. In our study, we investigate the impact of instruction-tuning on LLM's susceptibility to misinformation. Our analysis reveals that instruction-tuned LLMs are significantly more likely to accept misinformation when it is presented by the user. A comparison with base models shows that instruction-tuning increases reliance on user-provided information, shifting susceptibility from the assistant role to the user role. Furthermore, we explore additional factors influencing misinformation susceptibility, such as the role of the user in prompt structure, misinformation length, and the presence of warnings in the system prompt. Our findings underscore the need for systematic approaches to mitigate unintended consequences of instruction-tuning and enhance the reliability of LLMs in real-world applications.

Problem

• Investigates how instruction-tuning affects LLM's vulnerability to misinformation
• Compares susceptibility between instruction-tuned and base LLM models
• Explores factors influencing misinformation acceptance in instruction-tuned LLMs

Innovation

• Instruction-tuning increases LLM misinformation susceptibility
• User role shifts misinformation reliance in tuned LLMs
• Prompt structure and warnings influence misinformation acceptance