DataNode

A DataNode is a fundamental component within distributed storage systems, most notably Hadoop's Hadoop Distributed File System (HDFS). Its primary role is to store and manage actual data blocks on disk. When a file is stored in HDFS, it is broken down into smaller, fixed-size blocks, typically 128MB or 256MB. Each DataNode is responsible for holding a subset of these blocks.

DataNodes communicate with a NameNode, which acts as the master server in HDFS. The NameNode maintains the file system's metadata, such as the names of files and directories, and crucially, the locations of each block across all DataNodes. DataNodes are continuously sending heartbeats and block reports to the NameNode to inform it of their status and the blocks they store. This allows the NameNode to manage block replication and detect DataNode failures.

When a client requests to read a file, the NameNode tells the client which DataNodes hold the necessary blocks. The client then directly communicates with those DataNodes to retrieve the data. Similarly, when writing data, the client writes to a pipeline of DataNodes, with the NameNode orchestrating the process. DataNodes are designed to be commodity hardware, and their distributed nature provides fault tolerance through block replication, ensuring data availability even if some DataNodes fail.