Re-TASK: Revisiting LLM Tasks from Capability, Skill, and Knowledge Perspectives

2024-08-13

arXiv.org

AI Summary

Large language models (LLMs) frequently fail at domain-specific tasks due to misalignment among their underlying capabilities, reasoning skills, and domain knowledge—causing Chain-of-Thought (CoT) reasoning to break down. Method: This paper proposes Re-TASK, a novel theoretical framework featuring a three-dimensional task modeling paradigm (“capability–skill–knowledge”) and replacing CoT with Chain-of-Learning. It identifies structural capability gaps as the root cause of CoT failure and introduces a precise, augmentable capability alignment mechanism. Integrating Bloom’s taxonomy with knowledge space theory, Re-TASK designs prompt strategies enabling domain-knowledge injection and skill-adaptive fine-tuning. Contribution/Results: On legal benchmarks, Re-TASK boosts accuracy by 44.42% for Yi-1.5-9B and 33.08% for Llama3-Chinese-8B. It significantly enhances LLMs’ accuracy and cross-domain generalization in complex, specialized tasks—including law, finance, and mathematics.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The Chain-of-Thought (CoT) paradigm has become a pivotal method for solving complex problems. However, its application to intricate, domain-specific tasks remains challenging, as large language models (LLMs) often struggle to accurately decompose these tasks and, even when decomposition is correct, fail to execute the subtasks effectively. This paper introduces the Re-TASK framework, a novel theoretical model that revisits LLM tasks from the perspectives of capability, skill, and knowledge, drawing on the principles of Bloom's Taxonomy and Knowledge Space Theory. While CoT offers a workflow perspective on tasks, the Re-TASK framework introduces a Chain-of-Learning view, illustrating how tasks and their corresponding subtasks depend on various capability items. Each capability item is further dissected into its constituent aspects of knowledge and skills. Our framework reveals that many CoT failures in domain-specific tasks stem from insufficient knowledge or inadequate skill adaptation. In response, we combine CoT with the Re-TASK framework and implement a carefully designed Re-TASK prompting strategy to improve task performance. Specifically, we identify core capability items linked to tasks and subtasks, then strengthen these capabilities through targeted knowledge injection and skill adaptation. We validate the Re-TASK framework on three datasets across the law, finance, and mathematics domains, achieving significant improvements over the baseline models. Notably, our approach yields a remarkable 44.42% improvement with the Yi-1.5-9B model and a 33.08% improvement with the Llama3-Chinese-8b on the legal dataset. These experimental results confirm the effectiveness of the Re-TASK framework, demonstrating substantial enhancements in both the performance and applicability of LLMs.

Problem

• Improving LLM task decomposition for domain-specific applications
• Enhancing LLM capabilities via knowledge and skill adaptation
• Addressing CoT failures with targeted learning strategies

Innovation

• Introduces Chain-of-Learning (CoL) paradigm
• Proposes Re-TASK prompting strategy
• Enhances capabilities via knowledge and skill adaptation