The Hitchhikers Guide to Production-ready Trustworthy Foundation Model powered Software (FMware)

2025-05-15

AI Summary

This paper systematically addresses core challenges in industrializing foundation model–driven software (FMware), including model–task mismatch, insufficient domain data alignment, brittle prompting, uncontrolled multi-agent orchestration, lack of rigorous testing and validation, and difficult legacy system integration. To tackle these, we propose the first end-to-end, production-oriented FMware technology roadmap: a trustworthiness assurance framework integrating model evaluation, data flywheel construction, robust prompt engineering, multi-agent workflow orchestration, A/B testing–driven deployment, and API gateway–level model integration—complemented by a phased evolution strategy. Our contributions include a comprehensive KDD 2025 tutorial series featuring methodological guidelines, actionable checklists, and decision trees. Empirical adoption demonstrates significantly reduced FMware time-to-production, alongside improved system reliability and maintainability.

Technology Category

• Machine Learning: Hardware-aware ML
• Multiagent Systems: Other Foundations of Multi Agent Systems
• Application Domains: Software Engineering

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Foundation Models (FMs) such as Large Language Models (LLMs) are reshaping the software industry by enabling FMware, systems that integrate these FMs as core components. In this KDD 2025 tutorial, we present a comprehensive exploration of FMware that combines a curated catalogue of challenges with real-world production concerns. We first discuss the state of research and practice in building FMware. We further examine the difficulties in selecting suitable models, aligning high-quality domain-specific data, engineering robust prompts, and orchestrating autonomous agents. We then address the complex journey from impressive demos to production-ready systems by outlining issues in system testing, optimization, deployment, and integration with legacy software. Drawing on our industrial experience and recent research in the area, we provide actionable insights and a technology roadmap for overcoming these challenges. Attendees will gain practical strategies to enable the creation of trustworthy FMware in the evolving technology landscape.

Problem

• Challenges in selecting suitable foundation models for FMware
• Difficulties in aligning domain-specific data and engineering prompts
• Issues in testing, optimizing, and deploying production-ready FMware

Innovation

• Integrates Foundation Models as core components
• Addresses model selection and data alignment
• Provides system testing and deployment solutions