🤖 AI Summary
"This study addresses the deployment of small language models (SLMs) on resource-constrained single-board computers, aiming to reduce reliance on cloud services, enhance local data processing capabilities, and accommodate intermittent connectivity. The research evaluates five open-weight models, each with fewer than 200 million parameters, across 100 application scenarios, using greedy decoding and nucleus sampling to test their ability to generate JSON format instructions. The work provides the first comprehensive evaluation benchmark for SLM tool invocation on edge devices and explores the trade-offs between model size and resource consumption. Results indicate that the Qwen2.5-1.5B model exhibits the best performance but requires the most resources, whereas the Qwen2.5-0.5B model, while more resource-efficient, shows slightly inferior performance. Additionally, the study reveals a low incidence of directly parseable JSON responses, underscoring the importance of output recovery mechanisms."
📝 Abstract
Resource constrained single-board computers including Raspberry Pi, NVIDIA Jetson Nano, Arduino UNO Q, Orange Pi, and LattePanda motivate on-device small language model (SLM) agents that reduce cloud dependence, improve data locality, and tolerate intermittent connectivity. Model Context Protocol (MCP)-style tool invocation demands more than fluent generation: an agent must emit machine-readable JSON, select the correct tool, supply all required arguments, and avoid unintended actions. We establish a platform-agnostic CPU baseline by evaluating five open-weight models below two billion parameters Phi-1.5, Pythia-1.4B, TinyLlama-1.1B-Chat, Qwen2.5-0.5B, and Qwen2.5-1.5B on 100 prompts spanning weather retrieval, web search, calculation, email composition, and task creation, under greedy decoding and nucleus sampling. A recovery parser strips Markdown fences, extracts brace-delimited substrings, and scores parseability, tool-name correctness, argument completeness, and value agreement. Under this criterion, Qwen2.5-1.5B achieves 75% (greedy) and 79% (sampling); Qwen2.5-0.5B achieves 72% (greedy) but drops to 32% under sampling. Phi-1.5 scores 0%; Pythia and TinyLlama reach at most 7%. A strict post-hoc audit finds only 5 of 1,000 raw responses directly parseable as JSON, exposing near-total dependence on output recovery. A CPU resource probe shows Qwen2.5-1.5B requires 7,960 MiB and 30.782 s mean latency; Qwen2.5-0.5B uses 3,637 MiB and 10.627 s, revealing a reliability-resource trade-off for edge deployment. These results do not cover the named boards directly or a full MCP implementation. Safe deployment requires schema validation, constrained generation, least-privilege execution, and human escalation for consequential actions.