Can AI Guess What You Know? Performance Comparison of Large Language Models for Human Domain Knowledge Estimation From Communication Logs

2026-05-21

AI Summary

This study addresses the challenge organizations face in accurately identifying employees’ expertise, which often leads to inefficiencies in collaboration. It presents the first systematic evaluation of seven large language models—including Gemini, Claude, and GPT variants—in their zero-shot capability to construct individual expertise graphs from real-world Slack communication logs, benchmarked against self-reported user skills. Results indicate that message volume has limited impact on inference accuracy, with Gemini 2.5 Flash achieving the best performance (MAE: 21.13%) and significantly outperforming all GPT-family models. The findings demonstrate the feasibility of leveraging large language models for automated expert identification while also highlighting current limitations of such approaches.

Abstract

Employees often struggle to identify ``who knows what,'' leading to organizational productivity losses. We investigate whether Large Language Models (LLMs) can infer individual domain knowledge directly from long-term Slack logs. Analyzing 27,188 messages from 43 users, we evaluated seven models (including Gemini, Claude, and GPT families) by comparing their zero-shot estimates against self-reported skill ratings from 27 participants. Gemini 2.5 Flash achieved the lowest error (MAE 21.13%), while GPT models showed significantly larger discrepancies. Notably, estimation accuracy depended only weakly on message volume, indicating that more text alone does not guarantee better inference. These findings demonstrate the feasibility and current limits of automated expertise mapping, highlighting the need for privacy-preserving deployments and richer, structure-aware representations of human knowledge.

Problem

• domain knowledge estimation
• expertise mapping
• communication logs
• Large Language Models
• organizational productivity

Innovation

• Large Language Models
• Zero-shot Estimation
• Expertise Mapping
• Communication Logs
• Human Knowledge Inference