Analyzing Latent Concepts in Code Language Models

2025-09-30

AI Summary

Code large language models (Code LLMs) suffer from poor interpretability in trustworthy, transparent, and semantically robust settings due to the opacity of their internal mechanisms. Method: This paper proposes Code Concept Analysis (CoCoA), the first global post-hoc interpretability framework tailored for code models. CoCoA identifies stable, fine-tuning-evolving latent concept clusters across abstraction levels by clustering contextualized token embeddings, aligning them with static program analysis, and leveraging prompt-engineering-driven concept annotation. It further integrates Integrated Gradients for concept-level attribution. Results: Evaluated concepts exhibit strong robustness to semantics-preserving perturbations (Concept Stability Index = 0.288). A user study shows CoCoA improves explanation quality by 37 percentage points over token-level attribution, significantly enhancing model transparency and semantic robustness.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Computer Vision: Interpretability, Explainability, and Transparency
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Interpreting the internal behavior of large language models trained on code remains a critical challenge, particularly for applications demanding trust, transparency, and semantic robustness. We propose Code Concept Analysis (CoCoA): a global post-hoc interpretability framework that uncovers emergent lexical, syntactic, and semantic structures in a code language model's representation space by clustering contextualized token embeddings into human-interpretable concept groups. We propose a hybrid annotation pipeline that combines static analysis tool-based syntactic alignment with prompt-engineered large language models (LLMs), enabling scalable labeling of latent concepts across abstraction levels. We analyse the distribution of concepts across layers and across three finetuning tasks. Emergent concept clusters can help identify unexpected latent interactions and be used to identify trends and biases within the model's learned representations. We further integrate LCA with local attribution methods to produce concept-grounded explanations, improving the coherence and interpretability of token-level saliency. Empirical evaluations across multiple models and tasks show that LCA discovers concepts that remain stable under semantic-preserving perturbations (average Cluster Sensitivity Index, CSI = 0.288) and evolve predictably with fine-tuning. In a user study, concept-augmented explanations disambiguate token roles. In a user study on the programming-language classification task, concept-augmented explanations disambiguated token roles and improved human-centric explainability by 37 percentage points compared with token-level attributions using Integrated Gradients.

Problem

• Interpreting internal behavior of code language models for trust and transparency
• Uncovering emergent lexical, syntactic and semantic structures in representations
• Identifying latent interactions, trends and biases in learned code representations

Innovation

• Clustering token embeddings into interpretable concept groups
• Combining static analysis with LLMs for scalable labeling
• Integrating concept analysis with local attribution methods