Vision-centric generative AI models: A software-hardware perspective

2026-08-27

AI Summary

本文探讨了视觉生成AI模型在硬件受限边缘设备上的应用问题，通过量化参数成本和能效，并倡导软硬件协同设计方法来解决。

Abstract

Vision generative artificial intelligence (AI) has emerged as one of the most rapidly advancing areas of deep learning. The explosion of multimodal models has made them widely associated with text-to-image applications running on large datacentres. However, vision generative models are equally needed in applications that operate under strict hardware constraints at the edge, including autonomous vehicles, agricultural sensors, and mobile devices. In this Perspective, we argue that progress in vision generative AI has been driven by output quality, with hardware evolving reactively to accommodate growing model demands. We quantify the parameter cost and energy efficiency of these models across a range of accelerator platforms, and map four generative model families against seven real-world application domains. Finally, we advocate a software-hardware co-design approach, where deployment constraints are considered from the start of the design process, ensuring that the "right model" runs on the "right hardware" to serve the "right application", making generative AI deployment sustainable and accessible across a much broader range of platforms.

Problem

• Vision generative AI
• hardware constraints
• edge devices
• software-hardware co-design
• deployment

Innovation

• software-hardware co-design
• deployment constraints
• parameter cost
• energy efficiency