DyMT-ESB: Dynamic Multi-Turn Evaluation of Social Bias in User-LLM Interactions

2026-09-16

AI Summary

研究通过动态多轮对话评估用户与大型语言模型互动中的社会偏见，揭示了偏见的延迟出现、非单调模式及重现，强调了超越固定轮次预设协议评估的重要性。

Abstract

Warning: This paper contains examples of stereotypes and social bias. LLMs are increasingly used in interactive settings by the general public, making the evaluation of model behavior in multi-turn conversational scenarios important for safety, including stereotyping-related harms. However, existing multi-turn social bias evaluations often rely on pre-specified or template-based user inputs that do not adapt to model responses and typically assume a fixed dialogue length in advance. In this paper, we study social bias dynamics in response-conditioned multi-turn interactions using a controlled evaluation protocol that generates follow-up user queries from the evolving dialogue history and allows evaluation over variable numbers of turns. Experimental results show that LLMs exhibit social bias even in coherent, response-conditioned multi-turn interactions, revealing late-emerging bias, non-monotonic bias patterns, and bias re-emergence. These results motivate evaluations that extend beyond fixed-turn, pre-scripted protocols. Our findings highlight the importance of analyzing social bias as a turn-level dynamic phenomenon.

Problem

• social bias
• multi-turn interactions
• response-conditioned
• dynamic evaluation
• LLMs

Innovation

• Dynamic Multi-Turn Evaluation
• Social Bias
• Response-Conditioned Interactions
• Variable Dialogue Lengths