GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations

2026-05-14

AI Summary

This work addresses the critical gap in existing large language model agents’ memory systems, which lack explicit modeling of multi-user group chat scenarios and consequently struggle with essential capabilities such as group dynamics reasoning, speaker-aware belief tracking, and audience adaptation. The study formally defines three core attributes of group chat memory and introduces the first benchmark specifically designed for evaluating memory in multi-user dialogues. This benchmark leverages graph-structured synthetic data to generate controlled, multi-turn, multi-character conversations, incorporating role- and audience-conditioned generation alongside adversarial query mechanisms to assess six key memory challenges—including multi-hop reasoning and knowledge updating. Experimental results reveal that state-of-the-art memory systems achieve only 46.0% average accuracy on this benchmark, performing particularly poorly on knowledge updating (27.1%) and term ambiguity resolution (37.7%), even underperforming a simple BM25 baseline, thereby exposing significant deficiencies in their structural and lexical modeling for group chat memory.

Abstract

Large Language Model (LLM) agents increasingly serve as personal assistants and workplace collaborators, where their utility depends on memory systems that extract, retrieve, and apply information across long-running conversations. However, both existing memory systems and benchmarks are built around the dyadic, single-user setup, even though real deployments routinely span groups and channels with multiple users interacting with the agent and with each other. This mismatch leaves three properties of group memory unmeasured: (i) group dynamics that go beyond concatenated one-on-one chats, (ii) speaker-grounded belief tracking, where the per-user memory modeling is needed, and (iii) audience-adapted language, where Theory-of-Mind shifts produce role-specific vocabulary. We introduce GroupMemBench, a benchmark that exposes all three. A graph-grounded synthesis pipeline produces multi-party conversations with controllable reply structure and conditions each message on per-user personas and target audiences. An adversarial query pipeline then binds every question to a specific asker across six categories, spanning multi-hop reasoning, knowledge update, term ambiguity, user-implicit reasoning, temporal reasoning, and abstention, and iteratively searches challenging, realistic queries that reflect comprehensive memory capability. Benchmarking leading memory systems exposes a sharp collapse: the strongest one reaches only 46.0% average accuracy, with knowledge update at 27.1% and term ambiguity at 37.7%, while a simple BM25 baseline matches or exceeds most agent memory systems. This indicates current memory ingestion erases the structural and lexical features group memory depends on, leaving multi-user memory far from solved.

Problem

• group memory
• multi-party conversations
• LLM agent memory
• speaker-grounded belief tracking
• audience-adapted language

Innovation

• group memory
• multi-party conversation
• speaker-grounded belief tracking
• audience-adapted language
• adversarial query generation