CARB: A Characterization-Guided Framework for CNN Inference Cost Prediction and Deployment Screening

2026-08-11

AI Summary

This work addresses the challenge of accurately predicting inference energy consumption, latency, and peak memory usage of convolutional neural networks (CNNs) on resource-constrained GPUs—a task existing methods struggle with due to their neglect of nonlinear interactions between model architecture and hardware load. Leveraging GPU telemetry data from 13,419 CNN configurations evaluated on RTX 5090 and RTX 3080 platforms, the study reveals fundamentally distinct scaling behaviors and cross-platform transferability patterns among these three metrics. To exploit these insights, the authors propose a cascaded hybrid ensemble model that achieves joint high-accuracy prediction (R² ≈ 0.99). Furthermore, they introduce a two-stage Pareto-aware filtering mechanism that eliminates over 90% of nonviable candidates within seconds, drastically shrinking the deployment design space.

Abstract

Accurate pre-deployment estimation of CNN inference cost--energy, latency, and peak memory--is increasingly critical as models are deployed on resource-constrained GPU platforms. Existing approaches rely on FLOPs, latency measurements, or single-device profiling as energy proxies, overlooking the non-linear interactions between architectural design and hardware load. We present a workload characterization study of 13 419 CNN configurations on two GPU platforms (RTX 5090 and RTX 3080) under GPU telemetry, revealing that energy, latency, and memory exhibit fundamentally distinct scaling behaviors: energy and latency diverge by 3x under high computational demand, and cross-GPU transferability differs by target--energy and latency require platform-specific models while memory transfers well across the two tested platforms. Building on these characterization findings, we develop CARB, a cascade-blended ensemble that jointly predicts all three targets with R2 ~0.99, and a two-stage deployment screening workflow that eliminates over 90% of candidates in seconds, reducing large design spaces to a Pareto-prioritized shortlist validated against real hardware.

Problem

• CNN inference cost
• energy prediction
• latency estimation
• memory footprint
• GPU deployment

Innovation

• workload characterization
• CNN inference cost prediction
• cross-platform transferability
• cascade-blended ensemble
• deployment screening