🤖 AI Summary
This study systematically investigates the impact of optimizer selection on predictive performance for heart disease diagnosis. Within a unified training framework, we evaluate ten optimization algorithms—including SGD, Adam, and RMSProp—using a multilayer perceptron model, comparing their convergence speed, training stability, and classification performance (AUC, precision, recall). We propose a multi-criteria trade-off-based optimizer selection methodology to enhance interpretability and practicality in deep learning training. Experimental results demonstrate that RMSProp achieves superior overall performance: AUC = 0.841, recall = 0.827, accuracy = 0.765, faster convergence, and greater training stability—significantly outperforming all other optimizers. This work provides empirical evidence and a principled methodology for scientifically selecting optimizers in clinical prediction tasks.
📝 Abstract
Optimization has been an important factor and topic of interest in training deep learning models, yet less attention has been given to how we select the optimizers we use to train these models. Hence, there is a need to dive deeper into how we select the optimizers we use for training and the metrics that determine this selection. In this work, we compare the performance of 10 different optimizers in training a simple Multi-layer Perceptron model using a heart disease dataset from Kaggle. We set up a consistent training paradigm and evaluate the optimizers based on metrics such as convergence speed and stability. We also include some other Machine Learning Evaluation metrics such as AUC, Precision, and Recall, which are central metrics to classification problems. Our results show that there are trade-offs between convergence speed and stability, as optimizers like Adagrad and Adadelta, which are more stable, took longer time to converge. Across all our metrics, we chose RMSProp to be the most effective optimizer for this heart disease prediction task because it offered a balanced performance across key metrics. It achieved a precision of 0.765, a recall of 0.827, and an AUC of 0.841, along with faster training time. However, it was not the most stable. We recommend that, in less compute-constrained environments, this method of choosing optimizers through a thorough evaluation should be adopted to increase the scientific nature and performance in training deep learning models.