Skill Weaving: Efficient LLM Improvement via Modular Skillpacks

2026-05-21

AI Summary

This work addresses the resource constraints—such as memory consumption and inference latency—that often hinder large language models in multi-domain specialization. To overcome these limitations, the authors propose SkillWeave, a framework that decomposes general-purpose capabilities into lightweight, domain-specific modular units called skillpacks, complemented by SkillZip compression to produce highly efficient inference formats. This approach enables knowledge recombination, fine-grained fine-tuning, and low-latency multitask inference. Experimental results demonstrate that a 9B-parameter SkillWeave model outperforms multiple baselines—including a 32B monolithic model—on multitask and agent-based benchmarks, achieving up to 4× faster inference while significantly enhancing deployment efficiency across multiple domains under a fixed memory budget.

Abstract

Large language models increasingly require specialization across diverse domains, yet existing approaches struggle to balance multi-domain capacities with strict memory and inference constraints. In this work, we introduce SkillWeave, a modular improvement framework that enables LLMs to specialize under fixed memory budgets. SkillWeave partitions full capabilities of a general-purpose model into skillpacks -- lightweight, domain-specific delta modules -- that reorganize and refine the model's internal knowledge. For efficient deployment, SkillWeave integrates SkillZip to compress skillpacks into compact and inference-ready format, enabling strong multi-domain performance with low-latency execution. On multi-task and agentic benchmarks, a 9B SkillWeave model outperforms several baselines and even surpasses a 32B monolithic LLM, while achieving up to 4x speedup.

Problem

• large language models
• multi-domain specialization
• memory constraints
• inference efficiency
• modular adaptation

Innovation

• modular skillpacks
• efficient LLM specialization
• delta modules
• SkillZip compression
• multi-domain adaptation