NFQ2.0: The CartPole Benchmark Revisited

2025-11-16

AI Summary

Original Neural Fitted Q-Iteration (NFQ) suffers from poor reproducibility, intricate hyperparameter tuning, and limited robustness in real-world control tasks. To address these limitations, this paper proposes NFQ2.0—a refined variant incorporating batch learning to enable seamless transition between online and offline training, a deep multilayer neural network architecture, systematic hyperparameter optimization, and ablation studies to isolate the impact of key design choices. On the CartPole benchmark, NFQ2.0 significantly improves training stability and consistency of policy performance. Furthermore, its engineering viability is validated on a physical testbed built from real industrial components. This work delivers a lightweight, robust, and easily tunable algorithmic framework for reproducible deployment of deep reinforcement learning in industrial control applications.

Technology Category

• Intelligent Robots: Behavior Learning & Control
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

This article revisits the 20-year-old neural fitted Q-iteration (NFQ) algorithm on its classical CartPole benchmark. NFQ was a pioneering approach towards modern Deep Reinforcement Learning (Deep RL) in applying multi-layer neural networks to reinforcement learning for real-world control problems. We explore the algorithm's conceptual simplicity and its transition from online to batch learning, which contributed to its stability. Despite its initial success, NFQ required extensive tuning and was not easily reproducible on real-world control problems. We propose a modernized variant NFQ2.0 and apply it to the CartPole task, concentrating on a real-world system build from standard industrial components, to investigate and improve the learning process's repeatability and robustness. Through ablation studies, we highlight key design decisions and hyperparameters that enhance performance and stability of NFQ2.0 over the original variant. Finally, we demonstrate how our findings can assist practitioners in reproducing and improving results and applying deep reinforcement learning more effectively in industrial contexts.

Problem

• Improving reproducibility and robustness of neural fitted Q-iteration algorithm
• Enhancing learning process stability for real-world control problems
• Identifying key design decisions for better industrial RL applications

Innovation

• Modernized NFQ2.0 variant for enhanced repeatability
• Ablation studies identifying key hyperparameters for stability
• Applied deep reinforcement learning to industrial systems