STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMs

2026-04-20

AI Summary

Current benchmarks struggle to precisely diagnose the specific skill gaps of large language models in compositional reasoning. This work proposes the Scaffolding-based Task Design (STaD) framework, which, for the first time, integrates educational scaffolding theory into model evaluation. By generating controlled task variants through structured and incremental support, STaD systematically decomposes and probes compositional reasoning capabilities. Operating under a black-box assumption, the approach enables scalable, fine-grained diagnosis of skill deficiencies. Experiments across six models and three reasoning benchmarks reveal distinct and concrete failure patterns unique to each model, demonstrating STaD’s effectiveness and novelty in pinpointing weaknesses in compositional reasoning.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Benchmarks are often used as a standard to understand LLM capabilities in different domains. However, aggregate benchmark scores provide limited insight into compositional skill gaps of LLMs and how to improve them. To make these weaknesses visible, we propose Scaffolded Task Design (STaD) framework. STaD generates controlled variations of benchmark tasks based on the concept of scaffolding, which introduces structured, incremental support in a step-by-step manner. Rather than inspecting failures individually, this approach enables systematic and scalable probing of model behavior by identifying the specific reasoning skill compositions they lack. Treating the LLM as a black box, our experiments on six models of varying sizes reveal multiple failure points in three reasoning benchmarks and highlight each model's unique and distinct skill gaps.

Problem

• compositional skill gaps
• large language models
• benchmarking
• reasoning skills
• scaffolded task design

Innovation

• Scaffolded Task Design
• compositional reasoning
• skill gap analysis
• controlled task variation
• black-box probing