ZarrConvert

Hard-link conversion between FilesDaf and ZarrDaf directories.

The two on-disk formats differ in their per-property metadata encoding ( FilesDaf uses one JSON sidecar per array, ZarrDaf uses one zarr.json per array), but both store every numeric blob byte-identically:

• Flat (unpacked) dense array chunks and the colptr / rowval / nzind / nzval components of flat sparse arrays are raw little-endian bytes without headers — bit-for-bit identical between FilesDaf 's <name>.data / <name>.<component> and ZarrDaf 's single-chunk file at <name>/c/0[/0] .
• Packed (chunked + compressed) dense properties and packed sparse components are stored in the v3 sharded-array binary format (ZEP-0002): one shard file per property ( <name>.zip in FilesDaf , <name>/c/0[/0] in ZarrDaf ). For the same input data, same chunk_shape , and same codec, the writer emits byte-identical shard bytes regardless of which backend hosts it — so the shard files hard-link cleanly across formats.

The functions in this module exploit these equivalences to convert one tree into the other by hard-linking every numeric blob (flat or packed) and re-serializing only the metadata and the string-valued properties, so the on-disk cost of a conversion is close to zero.

The consolidated metadata files of the two formats — FilesDaf 's metadata.json (a single-line JSON object mapping relative path to per-property descriptor) and ZarrDaf 's root zarr.json#consolidated_metadata.metadata (the same mapping in the on-disk shape zarr-python 3.x writes) — carry the same information about the same set of properties; the per-property descriptor schemas are bijective. The translation rules between the two descriptor schemas are formally defined by this module: every dense / sparse / packed / flat / scalar / axis case has a documented mapping, and the source's consolidated metadata is read once at the start of a conversion to drive the per-property iteration. The destination's consolidated metadata is rebuilt at the end of the conversion (via refresh_consolidated_metadata! ) so the destination is immediately usable, including for HTTP serving.

Hard-linking requires the source and destination to live on the same filesystem; each conversion verifies this up front with an actual hard-link probe and refuses otherwise. Each conversion also refuses if the destination path already exists — it never overwrites pre-existing data. On any error the partially-populated destination is removed, so the destination is always either fully-populated or fully-absent.

Only the directory-tree ZarrDaf backend is supported (paths ending with .daf.zarr ). The ZIP archive backend ( .daf.zarr.zip , .dafs.zarr.zip#/group ) and the HTTP backend ( http(s)://… ) are rejected — neither provides the per-chunk on-disk file that hard-linking needs.

zarr_to_files(;
    zarr_path::AbstractString,
    files_path::AbstractString,
)::Nothing

Convert a ZarrDaf directory at zarr_path into an equivalent FilesDaf directory at files_path . Every numeric blob (dense chunks and the colptr / rowval / nzind / nzval components of sparse arrays) is hard-linked from the source into the destination; scalars, axes, string vectors and string matrices are re-serialized through the respective readers/writers. zarr_path must be a directory whose name ends with .daf.zarr (the ZIP and HTTP backends are rejected); files_path must not already exist, and the two paths must live on the same filesystem.

files_to_zarr(;
    files_path::AbstractString,
    zarr_path::AbstractString,
)::Nothing

Convert a FilesDaf directory at files_path into an equivalent ZarrDaf directory at zarr_path . Every numeric blob (dense chunks and the colptr / rowval / nzind / nzval components of sparse arrays) is hard-linked from the source into the destination; scalars, axes, string vectors and string matrices are re-serialized through the respective readers/writers. files_path must be an existing FilesDaf directory; zarr_path must not already exist, its name must end with .daf.zarr (the ZIP and HTTP backends are rejected), and the two paths must live on the same filesystem.