Gemma 4 Technical Report

2026-07-02

AI Summary

This work addresses key limitations of current language models in multimodal understanding, reasoning, computational efficiency, and long-context processing by introducing the Gemma 4 series—an open-source family of natively multimodal language models spanning 2.3B to 31B parameters. The architecture integrates dense and mixture-of-experts (MoE) components, with a unified encoder-free design for the 12B variant that directly processes raw audio and video patches. The approach incorporates mode-of-thought generation to produce explicit reasoning traces, coupled with efficient attention mechanisms and long-context optimization techniques. Evaluated across STEM, multimodal, and long-context benchmarks, Gemma 4 substantially outperforms existing open-source models, matches or approaches the performance of significantly larger systems in human evaluations, and achieves notable improvements in computational efficiency and memory footprint.

Abstract

We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemma 4 model suite features dense and Mixture-of-Experts architectures, ranging from 2.3B to 31B parameters. Alongside improved vision and audio encoders for all model sizes, we propose a unified, encoder-free architecture for our 12B model, which ingests raw audio and image patches. Furthermore, we integrate a thinking mode, enabling Gemma models to generate reasoning traces prior to responding. We improve inference speed, memory, and compute efficiency, as well as long-context abilities through critical design choices. Gemma 4 establishes a leap in performance across STEM, multimodal, and long-context benchmarks, and rivals larger, frontier open models in human-rated tasks.

Problem

• multimodal language models
• compute efficiency
• reasoning
• long-context processing
• open-weight models

Innovation

• multimodal
• Mixture-of-Experts
• thinking mode
• encoder-free architecture
• long-context