FlexOlmo: Open Language Models for Flexible Data Use

2025-07-09

AI Summary

This work addresses the challenge of collaborative inference under strict data isolation—where distributed data cannot be shared and fine-grained, runtime-access control is required. To this end, we propose FlexOlmo, a Mixture-of-Experts (MoE) framework featuring a domain-aware routing mechanism that integrates independently trained expert models without joint training or raw-data sharing. FlexOlmo supports dynamic expert activation/deactivation during inference and enforces fine-grained, policy-driven data access control. Leveraging the FlexMix dataset and a decentralized training protocol, it enables model collaboration while preserving strict data sovereignty. Evaluated across 31 benchmark tasks, FlexOlmo achieves an average 41% performance gain over baseline models and outperforms existing model merging approaches by 10.1%. Moreover, it significantly surpasses conventional MoE models under equivalent computational overhead, demonstrating superior efficiency and adaptability in privacy-sensitive distributed settings.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Security and Privacy: Data transparency and provenance
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We introduce FlexOlmo, a new class of language models (LMs) that supports (1) distributed training without data sharing, where different model parameters are independently trained on closed datasets, and (2) data-flexible inference, where these parameters along with their associated data can be flexibly included or excluded from model inferences with no further training. FlexOlmo employs a mixture-of-experts (MoE) architecture where each expert is trained independently on closed datasets and later integrated through a new domain-informed routing without any joint training. FlexOlmo is trained on FlexMix, a corpus we curate comprising publicly available datasets alongside seven domain-specific sets, representing realistic approximations of closed sets. We evaluate models with up to 37 billion parameters (20 billion active) on 31 diverse downstream tasks. We show that a general expert trained on public data can be effectively combined with independently trained experts from other data owners, leading to an average 41% relative improvement while allowing users to opt out of certain data based on data licensing or permission requirements. Our approach also outperforms prior model merging methods by 10.1% on average and surpasses the standard MoE trained without data restrictions using the same training FLOPs. Altogether, this research presents a solution for both data owners and researchers in regulated industries with sensitive or protected data. FlexOlmo enables benefiting from closed data while respecting data owners' preferences by keeping their data local and supporting fine-grained control of data access during inference.

Problem

• Enables distributed training without sharing closed datasets
• Supports flexible data inclusion/exclusion during inference
• Improves performance while respecting data licensing constraints

Innovation

• Distributed training without data sharing
• Data-flexible inference with no retraining
• Domain-informed routing in MoE architecture