fsspec

fsspec is a Python library that provides a unified, Pythonic interface for interacting with a wide range of storage backends, from local disks to cloud services and HTTP endpoints. It defines a common filesystem API that allows code to read and write files without worrying about the underlying backend.

At the core is the AbstractFileSystem class and a registry that maps protocol strings (such as 'file', 's3', 'gs', 'http') to concrete implementations like LocalFileSystem or S3FileSystem. Users obtain a filesystem object via fsspec.filesystem(protocol, ...) or by calling fsspec.open. The interface includes methods for opening files, listing directories, checking existence, deleting, creating directories, and more.

fsspec emphasizes lazy opening of files through file-like objects and streaming; it supports opening both local and remote files with a uniform API. It also includes optional caching wrappers to store remote data locally and to speed repeated reads. A notable feature is compatibility with other PyData projects such as Dask, Xarray, and pandas, which rely on fsspec.open and fsspec.filesystem to abstract I/O.

Backends cover a wide range: local filesystem, cloud providers (S3, Google Cloud Storage, Azure Data Lake), HTTP/HTTPS, FTP, WebDAV, and HDFS. The project focuses on a small, extensible API so new backends can be added as separate implementations.

fsspec is open source, actively maintained, and widely used in the Python data ecosystem. It is released under an open license and available on PyPI and the project's repository, with documentation detailing the protocol registry, configuration options, and example patterns for reading and writing data across storage systems.