Apple Intelligence Foundation Language Models: Tech Report 2025

2025-07-17

AI Summary

Addressing Apple’s ecosystem demand for efficient, secure, multilingual, and multimodal AI models capable of seamless on-device and cloud deployment. Method: We propose a dual-track foundational model architecture co-optimized for edge and cloud execution, incorporating KV cache sharing, 2-bit quantization-aware training, PT-MoE (parallel token-mixture-of-experts) routing, and global-local interleaved attention. The framework integrates asynchronous reinforcement learning with a private-cloud training infrastructure and natively supports Swift LoRA fine-tuning and constrained tool calling. Contribution/Results: This work delivers the first full-stack optimized end-to-end multimodal large language model for device-cloud collaboration. It achieves state-of-the-art performance across image understanding, multilingual text generation, and tool-augmented reasoning—outperforming same-scale open-source models. The model supports >40 languages and cross-modal inference while rigorously preserving data privacy and minimizing computational cost. It is production-deployed across iOS/macOS devices and Apple’s Private Cloud Compute platform.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Security and Privacy: Security and privacy of machine learning and AI applications
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

We introduce two multilingual, multimodal foundation language models that power Apple Intelligence features across Apple devices and services: i a 3B-parameter on-device model optimized for Apple silicon through architectural innovations such as KV-cache sharing and 2-bit quantization-aware training; and ii a scalable server model built on a novel Parallel-Track Mixture-of-Experts PT-MoE transformer that combines track parallelism, mixture-of-experts sparse computation, and interleaved global-local attention to deliver high quality with competitive cost on Apple's Private Cloud Compute platform. Both models are trained on large-scale multilingual and multimodal datasets sourced via responsible web crawling, licensed corpora, and high-quality synthetic data, then further refined with supervised fine-tuning and reinforcement learning on a new asynchronous platform. The resulting models support several additional languages while understanding images and executing tool calls. In public benchmarks and human evaluations, both the server model and the on-device model match or surpass comparably sized open baselines. A new Swift-centric Foundation Models framework exposes guided generation, constrained tool calling, and LoRA adapter fine-tuning, allowing developers to integrate these capabilities with a few lines of code. The latest advancements in Apple Intelligence models are grounded in our Responsible AI approach with safeguards like content filtering and locale-specific evaluation, as well as our commitment to protecting our users' privacy with innovations like Private Cloud Compute.

Problem

• Develop efficient multilingual multimodal foundation language models for Apple devices
• Optimize model performance with architectural innovations and scalable server solutions
• Ensure responsible AI with privacy safeguards and developer-friendly integration tools

Innovation

• 3B-parameter on-device model with KV-cache sharing
• Parallel-Track Mixture-of-Experts transformer model
• Swift-centric Foundation Models framework for developers