vlen

Vlen, short for variable-length, is a data type concept used in computing and data storage to represent elements whose size can vary from one instance to another. It contrasts with fixed-length types, where every element occupies the same amount of memory. Variable-length types enable compact and flexible representations of data such as lists, strings, or sub-arrays that do not have a uniform length.

In practical data formats, the most common use of vlen is within hierarchical data formats like HDF5. In HDF5, a vlen datatype allows a single dataset element to store a sequence of elements or a string whose length is not known a priori. Each element can contain zero or more items, and the payload is typically stored in a separate memory region managed by the format’s library, with an offset or length descriptor guiding access. Languages and libraries that work with these formats often present vlen data as native lists or arrays: for example, a dataset of variable-length strings might appear as a list of Python strings in h5py, while in C it may appear as a pointer to data with an accompanying length.

Typical use cases include ragged or irregular data structures, such as sequences of tokens per record, varying-length text fields, or per-row lists in tabular data. Vlen can reduce wasted space when data sizes vary greatly, but it introduces complexity in memory management, I/O, and downstream processing. Access patterns may require additional indirection, and user code must handle allocation and deallocation of the variable payloads when not managed automatically by the data format or language bindings.

See also: variable-length data types, ragged arrays, HDF5, and data serialization concepts.