Icechunk

A transactional storage engine for Zarr designed to handle petabyte-scale multidimensional arrays with Git-like versioning.

Icechunk solves the consistency challenges of cloud-native scientific data by bringing ACID transactions and branching to the Zarr ecosystem. Built in Rust with Python bindings, it eliminates the risk of corrupted datasets during concurrent writes and enables zero-copy snapshots for reproducible research. It integrates directly with S3-compatible object storage (using the Icechunk format specification) to provide a high-performance alternative to traditional NetCDF or HDF5 files in distributed computing environments.

https://icechunk.io