appendonly

Appendonly is a data storage approach in which changes to a dataset are recorded only by appending new data to a log, rather than updating existing data in place. The resulting append-only log can be replayed to reconstruct the current state by applying the recorded operations in sequence. This pattern is used to provide durability, crash recovery, and auditability in databases, message brokers, and file systems.

In practice, operations are serialized as discrete entries and appended to a log file or stream. The log grows monotonically and can be used to rebuild state after a restart by replaying the entries. To manage log size and performance, systems often combine the log with periodic checkpoints or snapshots, and may perform log compaction or rewrite to produce a smaller, non-redundant log.

A well-known example in modern systems is Redis with its Append Only File (AOF) persistence. When appendonly is enabled, every write command is appended to an AOF file. Recovery consists of re-reading the AOF and reapplying commands in order. AOF persistence offers configurable durability via appendfsync settings, such as always, everysec, or no, trading safety for throughput. The log may be rewritten in the background to reduce size and remove redundant commands, a process known as AOF rewrite.

Benefits of append-only logs include strong durability guarantees and straightforward recovery, while downsides can include higher storage requirements and potential write latency due to disk syncing. Trade-offs are managed through sync policies, snapshot strategies, and background compaction or rewrite processes.