WARCFormat

WARCFormat refers to the standardized structure used for storing web archive data in WARC files. It is the widely adopted format for representing captured web content in a portable, machine-readable form and is formalized in ISO 28500:2017 as the Web ARChive file format. The format is maintained by web-archiving communities and organizations such as the International Internet Preservation Consortium, and is used by major archives and preservation projects.

A WARC file is a sequence of records. Each record begins with a header block, followed by an optional content payload. The header starts with the line WARC/1.0 or WARC/1.1 and includes metadata such as WARC-Type, WARC-Target-URI, WARC-Date, WARC-Record-ID, Content-Length, and Content-Type. A blank line separates the header from the content. Records may be stored uncompressed or compressed (commonly with gzip), which gives the familiar .warc.gz file extension in practice.

Record types describe the nature of the captured data. Common types include warcinfo (file-level information), response (HTTP response data), request (HTTP request data), metadata (supplementary metadata about the capture), resource (non-HTTP resources), and revisit (indicating content that is considered a repeat capture). The Content-Type field and the payload conventionally reflect the record type, with HTTP payloads embedded for response and request records.

WARCFormat supports identification and provenance through fields such as WARC-Record-ID, allowing records to be linked and referenced reliably. It also supports date and metadata for reproducibility, provenance tracking, and interoperability across tools. Tools such as web crawlers, ingest pipelines, and preservation repositories rely on WARCFormat to enable long-term access to web content. See also WARC, ISO 28500, IIPC.