🤖 AI Summary
This work systematically evaluates whether the Predictor-Corrector (PC) sampler outperforms Stochastic Gradient Langevin Dynamics (SGLD) in standard Joint Energy-Based Model (JEM) training. Using the WideResNet-28-10 architecture, we conduct a fair comparison between the two samplers across image generation, classification, and out-of-distribution (OOD) detection tasks, employing a replay buffer, Fréchet Inception Distance (FID), AUROC, and statistical equivalence testing. Under a fixed noise schedule, PC and SGLD exhibit no statistically significant differences in test accuracy (≈92.9%), FID (difference < 0.5), or AUROC (difference < 0.007), and both share similar failure modes, including late-stage training collapse. To our knowledge, this is the first study to reproduce and directly compare the PC sampler within the standard JEM framework, demonstrating that its practical performance is indistinguishable from SGLD and thereby challenging claims of its superiority.
📝 Abstract
Joint Energy-Based Models (JEM) unify classification and generation within a single network and support out-of-distribution (OOD) detection. Canonical JEM training relies on stochastic gradient Langevin dynamics (SGLD); a theoretically motivated alternative, the Predictor-Corrector (PC) sampler, has not previously undergone a systematic replication test on the canonical model. We reproduce canonical JEM on WideResNet-28-10 without normalisation layers on two independent runs and test whether PC retains its theoretical advantage without an annealed noise schedule, across three protocols: PC replacing SGLD throughout the roughly 130 training epochs; cold-start generation (FID); and refinement-style multi-OOD detection (AUROC). The reconstruction reaches 92.88% test accuracy and buffer-FID 44.46 (canonical: 92.9% and 38.40). We document two failure modes: catastrophic late-training divergence via the canonical outlier-buffer mechanism (both SGLD runs and, with the same signature, both PC runs), and run-dependent SVHN OOD-discrimination dynamics. No method-level advantage of PC over SGLD is observed on any protocol: at inference the absolute AUROC difference stays below 0.007 across all ten checkpoint-OOD pairs and the FID difference below 0.5; on the training protocol a hierarchical seed-by-image bootstrap gives a 95% confidence interval on the macro-averaged AUROC difference that contains zero, while a seed-level equivalence test with two runs per method cannot establish formal equivalence. The data are consistent both with equivalence and with a small directional effect. This practical indistinguishability is theoretically expected: under fixed noise the PC predictor step degenerates by construction, so its guarantees do not transfer to canonical JEM.