Newlinedelimited

Newline-delimited, sometimes written as newlinedelimited, is a simple text data format in which records are separated by newline characters. Each line represents a distinct record, and the newline serves as the record delimiter. There is no universal single standard for the exact per-record structure, making the approach flexible across contexts. A common variant is newline-delimited JSON (NDJSON), where each line is a complete JSON object.

Common variant: In NDJSON, every line is a valid JSON object, and the file is a sequence of such objects. This allows easy line-by-line processing and streaming, while preserving the full expressiveness of JSON for each record.

Use cases: Newline-delimited data is widely used for logs, event streams, data ingestion pipelines, inter-process communication, and exporting or importing large datasets in a streaming-friendly format. It is particularly suited to environments that emphasize incremental processing and memory efficiency.

Processing and tooling: Readers typically parse input line by line, enabling streaming analytics and piping data between tools in UNIX-like environments. Writers append records as new lines, which simplifies appending and appending-only workflows.

Advantages: Simplicity, low memory footprint, good for streaming and incremental loading, and strong compatibility with line-oriented tools and languages. It is easy to append to existing datasets and to transport over text-based interfaces.

Limitations: Records containing embedded newlines require escaping or alternative representations; there is no universal, strict standard for all-newline-delimited forms beyond variant-specific rules; cross-platform line ending differences can complicate processing; for NDJSON, each line must be valid JSON and encoded consistently (commonly UTF-8).

Examples: A two-line NDJSON file might contain:

{"time":"2023-01-01T00:00:00Z","level":"INFO","msg":"start"}

{"time":"2023-01-01T00:00:01Z","level":"ERROR","msg":"failure","code":42}