DCO: Dynamic Cache Orchestration for LLM Accelerators through Predictive Management

2025-12-08

AI Summary

Deep hierarchical scratchpad memories (SPMs) in LLM accelerators introduce programming complexity and inefficient cache management. Method: This paper proposes a shared system-level cache architecture for many-core AI accelerators, coupled with an application-aware dynamic orchestration mechanism. Innovatively integrating dataflow analysis and cache behavior modeling, it enables predictive replacement, dead-block preemption, bypass decisions, and jitter-resistant scheduling—unified for both shared and private data scenarios. Contribution/Results: Evaluated via cycle-accurate simulation, analytical modeling, and a 15 nm RTL implementation (0.064 mm², 2 GHz), the design preserves programming simplicity while improving cache efficiency over baseline. It achieves up to 1.80× end-to-end speedup, significantly enhancing scalability and many-core collaboration performance.

Technology Category

• Machine Learning: Hardware-aware ML
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Mechanism Design

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The rapid adoption of large language models (LLMs) is pushing AI accelerators toward increasingly powerful and specialized designs. Instead of further complicating software development with deeply hierarchical scratchpad memories (SPMs) and their asynchronous management, we investigate the opposite point of the design spectrum: a multi-core AI accelerator equipped with a shared system-level cache and application-aware management policies, which keeps the programming effort modest. Our approach exploits dataflow information available in the software stack to guide cache replacement (including dead-block prediction), in concert with bypass decisions and mechanisms that alleviate cache thrashing. We assess the proposal using a cycle-accurate simulator and observe substantial performance gains (up to 1.80x speedup) compared with conventional cache architectures. In addition, we build and validate an analytical model that takes into account the actual overlapping behaviors to extend the measurement results of our policies to real-world larger-scale workloads. Experiment results show that when functioning together, our bypassing and thrashing mitigation strategies can handle scenarios both with and without inter-core data sharing and achieve remarkable speedups. Finally, we implement the design in RTL and the area of our design is $mathbf{0.064mm^2}$ with 15nm process, which can run at 2 GHz clock frequency. Our findings explore the potential of the shared cache design to assist the development of future AI accelerator systems.

Problem

• Optimizes cache management for AI accelerators using predictive policies
• Reduces programming complexity with shared cache and dataflow guidance
• Mitigates cache thrashing and improves performance in multi-core systems

Innovation

• Shared system-level cache with application-aware management
• Predictive cache replacement using dataflow information
• Bypass and thrashing mitigation strategies for performance