HDFS

HDFS, or the Hadoop Distributed File System, is a distributed file system designed to store and manage large data sets across clusters of commodity hardware. It is a core component of the Apache Hadoop ecosystem and was inspired by Google's GFS. HDFS provides scalable, fault-tolerant storage with high-throughput access to data, optimized for streaming reads and writes of large files.

Architecture and components: HDFS follows a master/slave model. A single NameNode manages the filesystem namespace and metadata, while a set of DataNodes store the actual file blocks on local disks. Files are divided into blocks (the default size is 128 MB in many releases) and each block is replicated across multiple DataNodes (the default replication factor is 3) to tolerate failures.

Data management: Clients interact with the NameNode to determine where blocks are stored and then read or write block data directly from DataNodes. The NameNode keeps metadata in memory with a disk-persisted edit log; DataNodes periodically report block information. The system uses heartbeats and block reports to monitor health and replica status.

Reliability and availability: HDFS is designed to handle DataNode failures by re-replicating blocks on healthy nodes. Historically a single NameNode was a single point of failure, leading to high availability options using a Quorum Journal Manager and a standby NameNode in an Active/Standby configuration.

Performance and usage: HDFS is optimized for large, sequential reads and writes rather than random access or many small files. It is widely used as a storage layer for data processing frameworks such as MapReduce, Apache Spark, and Apache Hive. In newer releases, features like erasure coding and federation improve storage efficiency and scalability.