Harmony: A Home Agent for Responsive Management and Action Optimization with a Locally Deployed Large Language Model

2024-10-18

arXiv.org

AI Summary

To address privacy leakage and external dependency issues of cloud-based LLM-powered smart home assistants, as well as the reliability deficiencies—such as hallucination and instruction misinterpretation—of local lightweight models, this paper proposes the first localized, proactive intelligent assistant framework tailored for home environments. Our approach fine-tunes and compresses Llama3-8B to enable fully offline deployment on consumer-grade PCs. It integrates a multimodal state-aware interface, a hybrid rule-and-LLM action planning engine, and a localized intent understanding and task decomposition module, enabling context-aware reasoning, temporal inference, and instruction-free autonomous decision-making—all without uploading any user data. Evaluated on a custom-built smart home benchmark, our system achieves performance on par with GPT-4, with end-to-end latency under 800 ms, and successfully executes six representative autonomous scenarios—including wake-up orchestration and home-arrival presets.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: State Estimation
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Since the launch of GPT-3.5, intelligent home assistant technology based on large language models (LLMs) has made significant progress. These intelligent home assistant frameworks, such as those based on high-performance LLMs like GPT-4, have greatly expanded their functional range and application scenarios by computing on the cloud, enriching user experience and diversification. In order to optimize the privacy and economy of data processing while maintaining the powerful functions of LLMs, we propose Harmony, a smart home assistant framework that uses a locally deployable small-scale LLM. Based on Llama3-8b, an open LLM that can be easily deployed on a consumer-grade PC, Harmony does not send any data to the internet during operation, ensuring local computation and privacy secured. Harmony based on Llama3-8b achieved competitive performance on our benchmark tests with the framework used in related work with GPT-4. In addition to solving the issues mentioned above, Harmony can also take actions according to the user and home status, even if the user does not issue a command. For example, when the user wants to wake up later than normal on the weekend, Harmony would open the curtains only when the user gets up or prepare the room when the user comes home without requiring user commands.

Problem

• Privacy concerns with cloud-based smart home assistants
• Reliability issues in locally deployed smaller LLMs
• Offline personalized smart home interaction challenges

Innovation

• Locally deployed Llama3-8B model
• Structured prompting technique
• Modular agent design