BOWL: A Deceptively Simple Open World Learner

2024-02-07

AI Summary

Traditional machine learning excels on static benchmarks but struggles in dynamic open-world settings—facing challenges including poor out-of-distribution (OOD) input detection, weak adaptation to emerging classes, and severe catastrophic forgetting during long-term deployment; existing works typically address these issues in isolation. This paper identifies, for the first time, that the running statistics of Batch Normalization (BN) layers serve as a unified representation foundation for open-world learning. Leveraging this insight, we propose an integrated framework comprising BN-statistics-based OOD detection, uncertainty-driven active sampling, and lightweight parameter incremental updating. Our method requires no architectural modification. Experiments demonstrate substantial improvements: open-set recognition accuracy is significantly enhanced; catastrophic forgetting is reduced by over 35%; and comparable performance is achieved using only one-third of the labeled data. The approach establishes an efficient, low-overhead paradigm for open-world continual learning.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Traditional machine learning excels on static benchmarks, but the real world is dynamic and seldom as carefully curated as test sets. Practical applications may generally encounter undesired inputs, are required to deal with novel information, and need to ensure operation through their full lifetime - aspects where standard deep models struggle. These three elements may have been researched individually, but their practical conjunction, i.e., open world learning, is much less consolidated. In this paper, we posit that neural networks already contain a powerful catalyst to turn them into open world learners: the batch normalization layer. Leveraging its tracked statistics, we derive effective strategies to detect in- and out-of-distribution samples, select informative data points, and update the model continuously. This, in turn, allows us to demonstrate that existing batch-normalized models can be made more robust, less prone to forgetting over time, and be trained efficiently with less data.

Problem

• Detect in- and out-of-distribution samples effectively.
• Select informative data points for continuous model updates.
• Enhance robustness and reduce forgetting in neural networks.

Innovation

• Utilizes batch normalization for open world learning
• Detects in- and out-of-distribution samples effectively
• Enables continuous model updates with less data