InputFormat

InputFormat is an abstraction used in data processing frameworks to describe how input data is split into manageable chunks and how those chunks are read. In the Hadoop MapReduce ecosystem, InputFormat is a core interface that determines two complementary responsibilities: how the input data is partitioned into InputSplit objects and how a RecordReader is created to produce key/value pairs from each split. By delegating these concerns to the InputFormat, the framework can efficiently schedule work and support diverse data sources, from text files to binary formats and specialized data stores.

The splits represent portions of input data that can be processed independently. The framework queries the InputFormat for the list of splits for a job, and for each split requests a RecordReader that iterates over the records in that split. The RecordReader yields successive key/value pairs which are supplied to mappers. Different InputFormat implementations tailor the splitting and reading strategy to the data type. For example, a text-based format may split by line boundaries and emit the line content as value with a numeric offset as key; a key-value text format may treat a separator to split into key and value.

Common implementations include TextInputFormat (reads lines of text), KeyValueTextInputFormat (parses lines into a key and value using a separator), and SequenceFileInputFormat (reads Hadoop sequence files). Some formats cannot be split, such as certain compressed files, which affects parallelism. InputFormat interacts with compression codecs and data locality to optimize distribution of work. Although originated in Hadoop MapReduce, the InputFormat concept has influenced data-source APIs in other frameworks as well.

See also: InputSplit, RecordReader, Hadoop MapReduce.