Characterizing Model-Native Skills

2026-04-19

AI Summary

This work addresses the limitation of existing skill representations, which rely on human-defined taxonomies and fail to align with the internal representations of language models, thereby constraining the efficacy of behavioral interventions. To overcome this, the paper introduces the concept of “model-native skills,” which are derived directly from sequence-level activations without requiring human ontological priors. These skills form a compact, orthogonal basis that yields a semantically interpretable skill space aligned with the model’s intrinsic representations. Leveraging this space, the authors integrate lightweight agent-based intervention, skill-guided data selection, and inference-time vector steering, achieving substantial performance gains on Llama3-8B and Qwen2.5-3B: Pass@1 accuracy improves by 20% on MATH and 41% on AMC, while Pass@8 at inference increases by up to 4.8%. The approach also demonstrates superior sample efficiency in safety alignment tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Skills are a natural unit for describing what a language model can do and how its behavior can be changed. However, existing characterizations rely on human-written taxonomies, textual descriptions, or manual profiling pipelines--all external hypotheses about what matters that need not align with the model's internal representations. We argue that when the goal is to intervene on model behavior, skill characterization should be *model-native*: grounded in the model's own representations rather than imposed through external ontologies. We instantiate this view by recovering a compact orthogonal basis from sequence-level activations. The resulting basis is semantically interpretable but need not correspond to any predefined human ontology; instead, it captures axes of behavioral variation that the model itself organizes around. We validate this characterization on reasoning post-training, using the recovered basis for both SFT data selection and inference-time steering. We develop lightweight proxy interventions to identify which directions are most useful for a given model. Across Llama3-8B and Qwen2.5-3B, selecting data along those directions improves Pass@1 by up to 20% on MATH and 41% on AMC, outperforming data selection based on human-characterized skills. Because the basis lives in activation space, the same directions also serve as steering vectors at inference time, improving Pass@8 by up to 4.8% on MATH--an intervention that human-characterized skills cannot support. We further validate the characterization on safety alignment, where selecting adversarial training data for model-native skill coverage rather than textual diversity yields more sample-efficient learning. These results suggest that recovering skills from the model's own representations, rather than imposing them externally, provides a more effective foundation for intervening on model behavior. Codes are open-sourced.

Problem

• model-native skills
• skill characterization
• language models
• behavioral intervention
• internal representations

Innovation

• model-native skills
• activation space basis
• orthogonal representation
• data selection
• inference-time steering