S3N

S3N (S3 Native FileSystem) is a Hadoop FileSystem implementation that provides a Hadoop-compatible interface to data stored in Amazon S3. It allows Hadoop MapReduce, Hive, and Pig to read and write data directly in S3 using the s3n:// URL scheme. S3N maps Hadoop filesystem calls to Amazon S3 operations, with files stored as objects in S3 buckets and directory structures represented by key prefixes.

S3N was introduced to enable scalable, cost-effective storage for Hadoop workloads without provisioning a traditional HDFS cluster. It relies on the AWS SDK for Java to sign requests and handle data transfer, and supports multi-part uploads for large files and parallel reads to improve throughput. However, S3N does not fully emulate POSIX filesystem semantics; some operations such as true random writes or in-place appends are not natively supported and may require workarounds, and directory listing can incur higher latency due to the object-store nature of S3.

Configuring S3N typically involves setting credentials (for example, awsAccessKeyId and awsSecretAccessKey) and optional endpoint and region parameters in Hadoop's core-site.xml. Performance depends on network bandwidth, S3 latency, and workload; S3N performance is generally lower than HDFS for small, random I/O but scales well for large, sequential transfers.

With the advent of newer Hadoop-compatible S3 connectors, S3N has become largely superseded by S3A and is now considered legacy in many deployments. S3A offers improved performance, larger object support, and better integration with Hadoop's IO abstractions.