Adversarial Concept Search: Predicting Compositional Errors From Feature Geometry

2026-06-11

AI Summary

Large language models frequently exhibit systematic errors on unseen compositional concepts, which are difficult to anticipate. This work establishes the first link between feature orthogonality in representation space and compositional generalization, introducing a proactive prediction method that requires no actual inference. By analyzing the geometry of linear representations and modeling feature interference, the approach identifies high-risk concept combinations without any input data. The method accurately forecasts model failures across diverse tasks—including program synthesis, multi-hop reasoning, and cross-lingual fact recall—providing a scalable foundation for targeted stress testing and active learning strategies.

Abstract

Humans cannot always intuit what scenarios are most challenging to LLMs. Hoping to capture challenging edge cases, developers either design problems to be difficult for humans or curate extensive benchmarks. What if we could instead anticipate which scenarios a model will fail on? In this paper, we use an LLM's representational geometry to predict which concept combinations it will fail on. We attribute this compositional failure to interference between salient features. In tasks that require systematic composition - toy programmatic settings, multihop reasoning, multilingual factual recall - we find that when a pair of concepts is encoded near-orthogonally, the model reliably composes them. When their linear encodings are close, producing interference, the model fails to compose them. Our method reliably anticipates failure modes across different compositional tasks, without evaluating specific inputs. These results lay the groundwork to use representational geometry to identify high-risk examples, construct targeted stress tests, and provide a scalable foundation for active learning in real-world deployment.

Problem

• compositional errors
• adversarial concept search
• feature geometry
• representational geometry
• LLM failure prediction

Innovation

• representational geometry
• compositional generalization
• feature interference
• adversarial concept search
• systematic composition