LLM360 K2: Scaling Up 360-Open-Source Large Language Models
To address the opacity and irreproducibility of large language model (LLM) training due to prohibitive costs, this work open-sources K2 DIAMOND—a 65B-parameter LLM—and achieves, for the first time, full transparency across the entire training lifecycle, including data, code, logs, and hardware configurations. We propose a “360-degree full-stack open-source paradigm,” introducing a low-FLOP/low-token training pathway and establishing critical best practices such as loss stability. Key technical innovations include mixed-precision training, dynamic sequence-length scheduling, fine-grained logging, a custom distributed optimizer, and an end-to-end data deduplication pipeline. Experiments show that K2 DIAMOND outperforms LLaMA-65B and matches LLaMA2-70B in performance, while reducing training FLOPs by 18% and token consumption by 22%. Complementing this, we launch the TXT360 initiative to advance open, reproducible AI research.