WARC

WARC, short for Web ARChive, is a file format designed for storing web content captured during web crawling and preservation activities. A WARC file contains a sequence of records that can hold both the HTTP transaction data and the actual content that was captured, along with metadata. The format is widely used by web archives, libraries, and research institutions to preserve snapshots of websites for long-term access and study. It evolved from the older ARC format and was standardized by the International Organization for Standardization as ISO 28500, consolidating archival practices for web content.

Each record in a WARC file begins with a header that includes fields such as WARC-Type, WARC-Target-URI, WARC-Date, Content-Length, and Content-Type, followed by the record’s content. Common record types include: request (the HTTP request), response (the HTTP response), metadata (supplemental information about the capture), resource (non-HTTP data), revisit (to indicate multiple captures of the same content with reduced storage), and warcinfo (archive-wide information). This structure allows archives to preserve both the data and the provenance of a capture, enabling reconstruction and analysis of the original web experience.

WARC files are frequently compressed for storage efficiency and are produced by many web crawlers and preservation tools, such as Heritrix. They are designed to be machine-readable and interoperable, with a range of software libraries and tools available to read, write, and process WARC data. WARC has become a central standard in modern web archiving, facilitating long-term access to historical web content. See also ARC, Memento, and web archiving practices.