Little Brains, Big Feats: Exploring Compact Language Models

2026-06-29

AI Summary

This study addresses the lack of systematic evaluation of small language models (SLMs) in retrieval-augmented generation (RAG) systems, particularly their potential for deployment on resource-constrained devices. It presents the first comprehensive assessment of SLMs in the RAG generation phase, benchmarking performance across diverse domains using both open-source and proprietary datasets. The work further demonstrates end-side inference entirely on CPU-based hardware without GPU acceleration. Experimental results show that SLMs can operate efficiently under such constraints, substantially reducing computational overhead while maintaining reasonable response times and generating high-quality outputs. These findings establish a viable pathway for deploying lightweight, edge-compatible AI systems leveraging RAG architectures.

Abstract

While large language models have been dominating the research landscape recently, small language models remain highly relevant across various domains; yet, they receive far less attention. In this study, we investigate how smaller language models perform during the generation stage within a Retrieval-Augmented Generation (RAG) system. To benchmark these models effectively, we utilised both open-source and proprietary datasets covering diverse subject areas and question types. Our findings demonstrate that a RAG system with small language models can be executed directly on-device without requiring any GPU hardware within a reasonable time. The experimental code and links to the supplementary materials can be accessed through the GitHub repository: https://github.com/SibNN/SLM-RAG-EVAL.

Problem

• small language models
• Retrieval-Augmented Generation
• on-device inference
• model evaluation
• compact models

Innovation

• Compact Language Models
• Retrieval-Augmented Generation
• On-device Inference
• Small Language Models
• Efficient NLP