Calibrated Hybrid CNN-Transformer for Retinal OCT Classification

2026-07-09

AI Summary

This study addresses the critical issue of unreliable model confidence in retinal optical coherence tomography (OCT) classification, which can lead to high-confidence erroneous predictions and delayed clinical diagnosis. To enhance diagnostic safety, the authors propose a hybrid encoder architecture combining CNN and Transformer modules, paired with an XGBoost classification head. Notably, this work is the first to integrate three complementary clinical safety mechanisms—confidence calibration, out-of-distribution (OOD) rejection, and predictive uncertainty quantification—within a unified OCT classification framework. Evaluated on a four-class OCT dataset, the method achieves 95.4% accuracy while reducing the expected calibration error (ECE) to 0.0024, a twelvefold improvement over baseline models, thereby substantially aligning predicted confidence with true accuracy. The trained model weights and reproducible evaluation code have been publicly released.

Abstract

Deep models for retinal optical coherence tomography (OCT) classification report high accuracy but rarely report whether their confidence can be trusted -- a gap that matters when a wrong-but-confident reading delays sight-saving treatment. We pair a hybrid convolutional-Transformer encoder with a gradient-boosting (XGBoost) classification head and a three-part clinical safety layer: confidence calibration, out-of-distribution (OOD) rejection, and per-prediction uncertainty flagging. On four-class OCT (84,495 scans) the model reaches 95.4% accuracy while cutting calibration error twelve-fold (expected calibration error, ECE = 0.0024), so the confidence it reports tracks its true accuracy. To our knowledge this is the first OCT classifier to validate all three safety mechanisms jointly, with public weights and reproducible multi-seed evaluation.

Problem

• retinal OCT classification
• confidence calibration
• out-of-distribution rejection
• prediction uncertainty
• model trustworthiness

Innovation

• confidence calibration
• hybrid CNN-Transformer
• out-of-distribution rejection
• uncertainty quantification
• retinal OCT classification