ANEForge: Python for direct computation on the Apple Neural Engine

2026-06-12

AI Summary

This work addresses the limitations of the Apple Neural Engine (ANE), which in production is accessible only indirectly through CoreML and lacks native support for training, sparse weights, and low-precision computation. The authors propose ANEForge, the first Python framework enabling direct ANE programming without relying on CoreML. It constructs a lazy tensor graph comprising 58 fused and 19 bridge operators, compiled into a single ANE program for execution. ANEForge enables, for the first time, end-to-end training on the ANE—including forward pass, backward pass, and optimizer updates—and supports native fused attention, int8/int4 quantization, and streaming loading of sparse weights. Leveraging macOS 14+ kernel drivers, it maintains strided state residency across invocations. Experiments demonstrate a minimal dispatch latency of 90 µs for small fused programs—close to the ANE’s scheduling lower bound of 70 µs—and achieve 0.33 ms end-to-end inference latency on ResNet-18, confirming both correctness and efficiency.

Abstract

ANEForge is a Python package that programs the Apple Neural Engine (ANE), the fixed-function neural accelerator on every recent Apple device, directly and without CoreML. In production the engine is reachable only through CoreML, which treats it as a scheduling option: no configuration requires the ANE, and a model can silently run on the CPU or GPU instead. ANEForge compiles a lazy tensor graph, built from 58 fused operators and 19 native bridge operators, into a single ANE program. The program is dispatched through the same ANE daemon and kernel-driver stack as Apple's internal framework. Beyond inference, the package reaches the engine's native fused attention, streams int8, int4, and sparse weights, keeps decoder and optimizer state resident across steps, and runs the forward pass, backward pass, and optimizer update of training on the engine. A small fused program completes a call in about 90us, near the engine's 70us per-program dispatch floor, and a pretrained ResNet-18 forward runs end-to-end in 0.33ms. ResNet-18, a sentence encoder, and a Vision Transformer run end-to-end against framework references, and a Stable Diffusion U-Net validates its forward pass. ANEForge targets Apple Silicon under macOS 14 and later. Each release is verified against a recorded macOS and ANE-compiler version.

Problem

• Apple Neural Engine
• CoreML
• direct computation
• neural accelerator
• hardware access

Innovation

• Apple Neural Engine
• direct hardware access
• fused operators
• on-device training
• sparse quantization