Selective Knowledge Control for Continual GUI Agent Learning over Application Streams

2026-09-06

AI Summary

本文提出了一种基于激活条件选择性知识控制的方法，通过神经元级梯度操作实现持续GUI代理学习中的选择性知识保留，有效缓解了灾难性遗忘问题。

Abstract

Continual learning is a crucial capability for Graphical User Interface (GUI) agents to adapt to evolving applications while retaining knowledge acquired from previous applications. Such application streams pose a challenging knowledge modeling problem: new applications often share underlying knowledge with past ones, yet also introduce specific knowledge that must not interfere with historical knowledge. In this paper, we propose activation-conditioned selective knowledge control, a lightweight method that achieves selective knowledge retention via neuron-level gradient manipulation. Our method maintains a compact historical knowledge state to protect highly activated MLP neurons that preserve previous knowledge. When a new application arrives, it performs real-time gradient surgery conditioned on forward activation. Concretely, the protected neurons are categorized into two types: unactivated neurons holding specific knowledge, whose gradients are truncated to prevent interference; and activated neurons holding shared knowledge, whose gradients are orthogonally projected to preserve stability while enabling adaptation. After each application stage, newly identified critical neurons are merged into the historical state for future learning. Empirical evaluations on multi-app sequential benchmark demonstrate that our method effectively mitigates catastrophic forgetting on prior applications while sustaining robust adaptation to new ones.

Problem

• Continual Learning
• Graphical User Interface (GUI) Agents
• Knowledge Retention
• Catastrophic Forgetting
• Application Streams

Innovation

• activation-conditioned selective knowledge control
• neuron-level gradient manipulation
• gradient surgery
• catastrophic forgetting