Cognitive World Models for Process-Level Social Influence Evaluation

2026-06-28

AI Summary

This work addresses the limitations of existing social influence dialogue evaluation methods, which often rely on surface-level text or single-point large model scores and fail to capture the dynamic evolution of users’ cognitive states—specifically beliefs, desires, intentions, and emotions (BDI/E). To overcome this, we propose CogWM, a cognitive world model grounded in large language models that jointly models user utterances and BDI/E states, shifting the evaluation focus from “what was said” to “how cognition evolves.” We introduce a three-tier evaluation framework encompassing turn-level fidelity, trajectory dynamics, and task-level composite scoring. Leveraging a novel Summarize-and-Allocate (SaA) annotation pipeline, we efficiently construct a dataset of 150,454 cognitively annotated dialogue turns to train CogWM, enabling multi-agent simulation and cognitive influence discrimination. Experiments show CogWM achieves 77.6% emotion recognition accuracy—2.1× higher than GPT-5.5—and significantly differentiates cognitive influence across six commercial agents in 3,600 multi-agent trials, with Llama-4-Scout achieving the highest Cognitive Trajectory Score (CTS +0.233).

Abstract

Social influence dialogue changes user behavior by altering internal cognitive states. The central evaluation question is whether the user's beliefs, desires, intentions, and emotions measurably change over the course of conversation, a process-oriented criterion that neither surface-level text metrics (BLEU/ROUGE) nor single-score LLM judgments can capture. We propose the \textbf{Cog}nitive \textbf{W}orld \textbf{M}odel \textbf{(CogWM)}, an LLM-based user model that reframes multi-turn dialogue evaluation from ``what did the user say'' to ``how did the user's internal cognitive state evolves.'' CogWM jointly predicts BDI/E cognitive states and user utterances and serves as both a user simulator and an evaluation platform, using a three-tier evaluation framework that covers turn-level fidelity, trajectory-level state dynamics, and task-level composite scoring. Trained via our \textbf{S}ummarize-\textbf{a}nd-\textbf{A}llocate \textbf{(SaA)} annotation pipeline on 150,454 user-turn samples across four social influence scenarios, CogWM achieves 77.6\% emotion accuracy (2.1$\times$ over GPT-5.5). In 3600 multi-agent discrimination trials, it distinguishes six commercial agents by their cognitive influence, with Llama-4-Scout ranking first (CTS +0.233). CogWM moves social influence dialogue evaluation from terminal judgment to process tracking. We have released our code\footnote{\scriptsize Code: https://github.com/lucianma05-create/CogWM} and models\footnote{Model: https://www.modelscope.cn/models/LucianMa/CogWM-14B}.

Problem

• social influence
• cognitive states
• dialogue evaluation
• process-level assessment
• BDI/E modeling

Innovation

• Cognitive World Model
• Process-Level Evaluation
• BDI/E States
• Social Influence Dialogue
• User Simulation