TrEnv: Transparently Share Serverless Execution Environments Across Different Functions and Nodes

2025-09-11

AI Summary

Serverless platforms face high infrastructure overhead when executing emerging workloads such as LLM agents, where frequent cold starts and volatile resource demands inflate platform costs up to 70% of LLM API call expenses. To address this, we propose a high-density serverless execution framework enabling shared execution environments across functions and nodes. Our approach introduces (1) reusable sandboxes and memory templates for rapid environment restoration and sharing, and (2) in VM-based deployments, integrates browser sharing with page-cache bypassing to reduce system-level overhead. The framework unifies container and VM abstractions. Evaluation shows that, in containerized environments, it reduces P99 latency by 7× and cuts memory usage by 48%; in VM environments, it achieves 58% lower P99 latency and 61% memory savings—significantly outperforming state-of-the-art systems like E2B.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Serverless computing provides dynamic scalability, but its infrastructure overhead becomes a bottleneck for emerging workloads such as LLM agents, which exhibit unpredictable invocation patterns and variable resource demands. Our analysis shows that for these agents, the cost of running on serverless platforms can reach up to 70% of the cost of LLM API calls. This finding motivates the need for a more efficient, high-density serverless platform. We present TrEnv, a co-designed serverless platform that supports both container- and VM-based environments, optimized for the unique demands of LLM agents. TrEnv reduces startup latency and memory usage through repurposable sandboxes and memory templates, which enable fast reuse and restoration of execution environments. To further reduce overhead in VM-based agent workloads, TrEnv leverages browser sharing and a page cache bypassing mechanism. Evaluations show that TrEnv reduces P99 latency by up to 7X and memory usage by 48% in container-based settings, and achieves up to 58% lower P99 latency and 61% memory savings for VM-based agents compared to state-of-the-art systems like E2B.

Problem

• Reducing infrastructure overhead in serverless computing for LLM agents
• Optimizing execution environments for unpredictable invocation patterns
• Minimizing startup latency and memory usage in serverless platforms

Innovation

• Repurposable sandboxes for fast environment reuse
• Memory templates reducing startup latency usage
• Browser sharing and page cache bypassing