Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research

2025-09-19

AI Summary

Small-language-model (SLM) development (<100M parameters) has long relied on empirical intuition, lacking reproducible, attributable scientific validation paradigms. Method: We propose the first modular research framework for SLMs, enabling hypothesis-driven, controlled-variable experimentation by decoupling model architecture and training procedures into plug-and-play components for fine-grained behavioral analysis. We release pico-decoder—a standardized baseline model suite—with unified evaluation protocols and hyperparameter configurations, and conduct systematic ablation studies to validate key design choices. Contribution/Results: The framework significantly improves SLM iteration efficiency and experimental transparency, facilitating rapid validation of multiple lightweight improvements. It enables joint optimization of performance and interpretability under strict parameter constraints, establishing a rigorous, reproducible foundation for SLM research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Building language models (LMs), especially small and medium ones, remains more art than science. While large LMs often improve by sheer scale, it is still unclear why many design choices work. For small LMs, this uncertainty is more limiting: tight parameter budgets make each decision critical, yet researchers still lack systematic, scientific ways to test and refine new ideas. We introduce Pico, a lightweight, modular framework that enables systematic, hypothesis-driven research for small and medium-scale language model development. Pico consists of two libraries that together provide a practical sandbox where researchers can make targeted changes to a model's architecture or training procedures and directly observe their effects on the model's behavior. To support reproducible experimentation, we also release a suite of baseline models, pico-decoder, trained under standardized conditions and open-sourced for the community. Case studies highlight how Pico can support iterative small LM design and analysis.

Problem

• Developing small language models lacks systematic scientific methods
• Unclear design choices hinder efficient small LM development
• Limited parameter budgets require precise architectural decisions

Innovation

• Modular framework for hypothesis-driven small LM research
• Libraries enabling targeted architecture and training modifications
• Open-sourced baseline models supporting reproducible experimentation