FactoryLLM: A Safe and Open-Source AI Playground for Evaluating LLMs in Smart Factories

2026-06-12

AI Summary

This study addresses the challenge of fault diagnosis in smart factories, where critical information is fragmented across numerous equipment manuals. To tackle this issue, the authors develop a secure, open-source, localized AI experimentation platform tailored for intelligent manufacturing, introducing for the first time an LLM evaluation sandbox in this domain. The platform integrates Retrieval-Augmented Generation (RAG) with open-source large language models and employs a dual evaluation framework combining RAGAS and LLM-as-a-Judge to enable safe, controllable assessment of cross-document reasoning capabilities. Evaluated on a test set comprising 600 pages of documentation and 30 maintenance-related queries, all models achieved groundedness scores exceeding 0.88, demonstrating the method’s effectiveness and practicality for multi-source industrial document reasoning.

Abstract

Fault diagnostics and recovery in smart factories is challenging because critical information is dispersed across manuals of multiple machines which are interconnected through the manufacturing process. Large Language Models (LLMs) can provide a promising approach. In this paper, we propose FactoryLLM, a safe and open-source AI playground designed for evaluating different LLM-based retrieval-augmented generation (RAG) models by analysing documents from multiple machines across the manufacturing process. FactoryLLM enables the user to configure the LLM, and assess performance when reasoning over multiple documents, through a dual evaluation setup using both RAGAS and NVIDIA's LLM-as-a-Judge metrics. FactoryLLM is safe because it allows users to run local or open-source LLMs without sharing sensitive industrial data, providing a controlled environment for experimentation. We demonstrate the efficacy of FactoryLLM through a case study which involves an Autonomous Intelligent Vehicle and its Mobile Planner software, evaluating three LLMs across 30 maintenance queries derived from approximately 600 pages of cross-machine documentation. The results suggest that FactoryLLM is effective in cross-machine document reasoning: every model achieved a groundedness score above 0.88. The full code and documentation for community to test FactoryLLM with their manufacturing specific scenarios are publicly available.

Problem

• fault diagnostics
• smart factories
• cross-machine documentation
• information dispersion
• manufacturing process

Innovation

• FactoryLLM
• Retrieval-Augmented Generation
• Smart Factory
• LLM Evaluation
• Cross-Machine Reasoning