Towards smart and adaptive agents for active sensing on edge devices

2025-01-09

AI Summary

To address the limited adaptability of TinyML in dynamic environments, this work introduces the first edge-device-oriented, active-inference-driven adaptive perception agent. Methodologically, it pioneers the integration of active inference—a neurobiologically grounded framework for decision-making under uncertainty—into the TinyML paradigm, combining a lightweight neural network (2.3 MB), an IoT camera interface, and the NVIDIA Jetson platform to enable on-device saccade policy generation and cloud-edge collaborative sensing. Key contributions include: (1) transcending conventional reactive approaches to data drift by modeling environmental dynamics and performing online uncertainty-aware planning; and (2) real-time deployment on Jetson hardware to drive pan-tilt-zoom cameras with human-inspired saccadic behavior, enabling low-power, closed-loop visual attention control, robust target tracking, and adaptive field-of-view modulation. Experimental results demonstrate significant improvements in perception efficiency and robustness for edge-based surveillance and robotic applications.

Technology Category

• Machine Learning: Active Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Video Understanding & Activity Analysis

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

TinyML has made deploying deep learning models on low-power edge devices feasible, creating new opportunities for real-time perception in constrained environments. However, the adaptability of such deep learning methods remains limited to data drift adaptation, lacking broader capabilities that account for the environment's underlying dynamics and inherent uncertainty. Deep learning's scaling laws, which counterbalance this limitation by massively up-scaling data and model size, cannot be applied when deploying on the Edge, where deep learning limitations are further amplified as models are scaled down for deployment on resource-constrained devices. This paper presents a smart agentic system capable of performing on-device perception and planning, enabling active sensing on the edge. By incorporating active inference into our solution, our approach extends beyond deep learning capabilities, allowing the system to plan in dynamic environments while operating in real time with a modest total model size of 2.3 MB. We showcase our proposed system by creating and deploying a saccade agent connected to an IoT camera with pan and tilt capabilities on an NVIDIA Jetson embedded device. The saccade agent controls the camera's field of view following optimal policies derived from the active inference principles, simulating human-like saccadic motion for surveillance and robotics applications.

Problem

• Adaptive Perception
• Resource-constrained Devices
• TinyML Limitations

Innovation

• Active Inference
• Adaptive Edge Computing
• TinyML for Resource-limited Devices