MojoFrame: Dataframe Library in Mojo Language

2025-05-07

AI Summary

Mojo excels in tensor computation but lacks native DataFrame support; its performance for relational operations—such as filtering, joins, and grouping—remains unvalidated, and critical ecosystem infrastructure is missing. To address this, we introduce MojoFrame, the first native DataFrame library for Mojo. It innovatively integrates tensor acceleration with a cardinality-aware heterogeneous columnar memory layout and implements a hardware-transparent relational algebra execution engine supporting user-defined functions (UDFs) and the full TPC-H query suite. Built atop the MLIR/JIT compilation framework, MojoFrame deeply leverages Mojo’s tensor primitives and custom relational operators. Experimental evaluation demonstrates that MojoFrame achieves up to 2.97× speedup over state-of-the-art DataFrame libraries on representative relational workloads, thereby closing a fundamental gap in Mojo’s data science ecosystem.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Search and Optimization: Distributed Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Mojo is an emerging programming language built on MLIR (Multi-Level Intermediate Representation) and JIT compilation. It enables transparent optimizations with respect to the underlying hardware (e.g., CPUs, GPUs), while allowing users to express their logic using Python-like user-friendly syntax. Mojo has been shown to offer great performance in tensor operations; however, its performance has not been tested for relational operations (e.g., filtering, join, and group-by), which are common in data science workflows. To date, no dataframe implementation exists in the Mojo ecosystem. In this paper, we introduce the first Mojo-native dataframe library, called MojoFrame, that supports core relational operations and user-defined functions (UDFs). MojoFrame is built on top of Mojo's tensor to achieve fast operations on numeric columns, while utilizing a cardinality-aware approach to effectively integrate non-numeric columns for flexible data representation. To achieve high efficiency, MojoFrame takes significantly different approaches than existing libraries. MojoFrame supports all operations for TPC-H queries, and achieves up to 2.97x speedup versus existing dataframe libraries in other programming languages. Nevertheless, there remain optimization opportunities for MojoFrame (and the Mojo language), particularly in data loading and dictionary operations.

Problem

• Evaluating Mojo's performance for relational operations in data science
• Introducing the first Mojo-native dataframe library (MojoFrame)
• Achieving speedup versus existing dataframe libraries

Innovation

• First Mojo-native dataframe library
• Uses Mojo's tensor for numeric columns
• Cardinality-aware approach for non-numeric columns