Do LLM Embedding Spaces Recover Expert Structure?

2026-06-22

AI Summary

This study investigates whether the embedding spaces of large language models (LLMs) genuinely recover the expert-defined categorical structure of mental health symptoms, beyond merely achieving high classification separability. Leveraging text from Reddit mental health communities, we compare embeddings from pretrained and supervised fine-tuned variants of Qwen3 (0.6B/4B) by constructing category prototypes and evaluating their alignment with expert symptom structures via representational similarity analysis (RSA), while rigorously controlling for confounding factors such as linguistic style and topic distribution. To our knowledge, this is the first fine-grained validation of LLMs’ capacity to reconstruct expert taxonomies. Results demonstrate that pretrained embeddings already exhibit significant alignment with expert structures, which is further enhanced through fine-tuning; moreover, increasing model scale strengthens both zero-shot alignment and fine-tuning gains, with robustness maintained under stringent confound control.

Abstract

Pretrained text embeddings are increasingly used as representational maps, yet high category separability does not imply that their geometry recovers expert-defined structure. We study this problem in mental-health-related language, where symptom relations provide an external reference and online communities introduce strong domain, affective, stylistic, and discourse confounds. Using 28 Reddit communities, we compare pretrained and supervised fine-tuned Qwen3 embedding spaces at two scales (0.6B and 4B). We construct category prototypes, evaluate their representational dissimilarity matrices against an expert symptom matrix with representational similarity analysis, and complement this global test with prototype-based typicality and multi-baseline confound controls. Pretrained embeddings show measurable alignment with expert structure within the mental-health subset; fine-tuning strengthens this alignment most at the finest category level; and larger scale improves both zero-shot alignment and supervision-induced gains. Residual alignment remains substantial after controlling for VAD, LIWC, lexical style, and topic-distribution structure. These results suggest that LLM embeddings can recover expert-relevant category geometry, but this recovery is level-dependent and should be tested against explicit confounds rather than inferred from classification alone.

Problem

• LLM embeddings
• expert structure
• mental health
• representational geometry
• confounding factors

Innovation

• LLM embeddings
• expert structure recovery
• representational similarity analysis
• confound control
• mental health language