STELLA: Efficient Sensor-to-LLM Translation for On-Device Human Activity Recognition

2026-07-03

AI Summary

This work addresses key challenges in deploying large language model (LLM)-based human activity recognition (HAR) on edge devices—namely data redundancy, high latency, privacy risks, and limited generalization. The authors propose a lightweight hierarchical sensor tokenizer that compresses multi-channel inertial signals into a fixed number of compact latent tokens, which are then mapped into the embedding space of a frozen pretrained LLM. Activity scoring is performed by combining these tokens with natural language prompts. By shifting the adaptation burden from LLM fine-tuning to the sensor side, the method enables on-device personalization using only minimal user data and incorporates local retrieval-augmented context. Evaluated on seven public HAR datasets, the approach achieves a new state of the art, with F1 scores improving by up to 11.83%; on-device personalization further boosts performance by up to 21.91%, meeting real-time inference requirements for mobile deployment.

Abstract

HAR is increasingly expected to run continuously on edge devices, yet recent LLM-based methods remain hard to deploy: raw sensor prompts are long, cloud inference adds latency and privacy risk, and fine-tuned LLM pipelines turn general-purpose models into task-specific classifiers. We present STELLA, an efficient sensor-to-LLM translation framework for on-device HAR that shifts the burden from LLM adaptation to sensor tokenization. A lightweight hierarchical tokenizer compresses an entire multi-channel inertial window into a fixed set of compact latent sensor tokens, which are projected into the embedding space of a frozen pretrained LLM and combined with a natural-language prompt for label scoring. This preserves activity-relevant temporal and cross-channel structure while keeping LLM-side computation predictable across sensor configurations. STELLA also supports on-device personalization, adapting only the lightweight tokenizer on small amounts of user-specific labelled data and augmenting inference with a local retrieval context, keeping the LLM, user data, and retrieval on device. Across seven public HAR datasets and eight benchmark settings, STELLA achieves new state-of-the-art performance, improving over prior methods by up to 11.83% F1; on-device personalization yields up to a further 21.91% F1 as user data accumulates after deployment. STELLA also outperforms representative time-series tokenizers under the same LLM pipeline and achieves real-time inference under practical mobile and edge budgets, showing that efficient sensor tokenization is a practical path toward accurate, private, and personalized LLM-based HAR on edge devices.

Problem

• Human Activity Recognition
• On-Device Inference
• Large Language Models
• Edge Computing
• Sensor Tokenization

Innovation

• sensor tokenization
• on-device LLM
• human activity recognition
• edge AI
• personalization