Evaluating Fault Tolerance and Scalability in Distributed File Systems: A Case Study of GFS, HDFS, and MinIO

2025-02-04

AI Summary

Distributed file systems (DFS) exhibit divergent fault tolerance and horizontal scalability characteristics, yet systematic, cross-architecture evaluation under realistic hybrid workloads remains scarce. Method: We conduct a comprehensive empirical study of Google File System (GFS), Hadoop Distributed File System (HDFS), and MinIO—representing legacy, big-data, and cloud-native paradigms—using unified benchmarks on physical clusters. We apply stress testing, controlled fault injection, and protocol-level log analysis to quantify throughput, recovery latency, and consistency guarantees across data redundancy, node failure recovery, and high-concurrency client access. Contribution/Results: This is the first study to comparatively evaluate these three architecturally distinct DFS under concurrent cloud-native and big-data workloads. We identify MinIO’s low-latency advantage for small files, HDFS’s stability limits in batch processing, and GFS’s enduring influence on lightweight DFS design. Based on these findings, we propose a scenario-driven selection framework—distinguishing high-availability storage from analytical workloads—to guide industrial storage system deployment with empirical evidence.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Distributed File Systems (DFS) are essential for managing vast datasets across multiple servers, offering benefits in scalability, fault tolerance, and data accessibility. This paper presents a comprehensive evaluation of three prominent DFSs - Google File System (GFS), Hadoop Distributed File System (HDFS), and MinIO - focusing on their fault tolerance mechanisms and scalability under varying data loads and client demands. Through detailed analysis, how these systems handle data redundancy, server failures, and client access protocols, ensuring reliability in dynamic, large-scale environments is assessed. In addition, the impact of system design on performance, particularly in distributed cloud and computing architectures is assessed. By comparing the strengths and limitations of each DFS, the paper provides practical insights for selecting the most appropriate system for different enterprise needs, from high availability storage to big data analytics.

Problem

• Evaluates fault tolerance in distributed file systems
• Assesses scalability under varying data loads
• Compares GFS, HDFS, MinIO for enterprise needs

Innovation

• Evaluates fault tolerance in DFS
• Compares scalability of GFS, HDFS, MinIO
• Assesses data redundancy and server failures