On the Effectiveness of Proposed Techniques to Reduce Energy Consumption in RAG Systems: A Controlled Experiment

2026-01-05

arXiv.org

AI Summary

This study addresses the high energy consumption of Retrieval-Augmented Generation (RAG) systems, which poses a significant barrier to their environmental sustainability, and the lack of systematic evaluation of existing energy-saving techniques in real-world settings. The authors present the first controlled empirical investigation of multiple energy-efficient strategies—including retrieval threshold tuning, embedding dimensionality reduction, vector index optimization, and BM25-based re-ranking—within a production-grade RAG system. Conducting over 200 hours of multi-configuration experiments on the CRAG benchmark, they comprehensively assess the trade-offs among energy consumption, latency, and accuracy. Their findings reveal that certain approaches can reduce energy usage by up to 60% without compromising accuracy, with retrieval threshold optimization and smaller embedding dimensions proving particularly effective, thereby offering practical pathways toward building efficient and environmentally sustainable RAG systems.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Efficient ML / Green AI
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments

Abstract

The rising energy demands of machine learning (ML), e.g., implemented in popular variants like retrieval-augmented generation (RAG) systems, have raised significant concerns about their environmental sustainability. While previous research has proposed green tactics for ML-enabled systems, their empirical evaluation within RAG systems remains largely unexplored. This study presents a controlled experiment investigating five practical techniques aimed at reducing energy consumption in RAG systems. Using a production-like RAG system developed at our collaboration partner, the Software Improvement Group, we evaluated the impact of these techniques on energy consumption, latency, and accuracy. Through a total of 9 configurations spanning over 200 hours of trials using the CRAG dataset, we reveal that techniques such as increasing similarity retrieval thresholds, reducing embedding sizes, applying vector indexing, and using a BM25S reranker can significantly reduce energy usage, up to 60% in some cases. However, several techniques also led to unacceptable accuracy decreases, e.g., by up to 30% for the indexing strategies. Notably, finding an optimal retrieval threshold and reducing embedding size substantially reduced energy consumption and latency with no loss in accuracy, making these two techniques truly energy-efficient. We present the first comprehensive, empirical study on energy-efficient design techniques for RAG systems, providing guidance for developers and researchers aiming to build sustainable RAG applications.

Problem

• energy consumption
• retrieval-augmented generation
• sustainability
• machine learning
• RAG systems

Innovation

• energy efficiency
• retrieval-augmented generation
• controlled experiment
• embedding size reduction
• similarity threshold tuning