Zarr Format

DataAxesFormats.ZarrFormat

— Module

A Daf storage format in a Zarr directory tree or ZIP archive. Like FilesDaf , the data can live in a directory of files on the filesystem (so standard filesystem tools work, and deleting a property immediately frees its storage), and offers a different trade-off compared to FilesDaf and H5df .

FilesDaf uses its own Daf-specific layout, but the individual files are in deliberately simple formats ( JSON for metadata, one-line-per-entry text for axis entries, raw little-endian binary for numeric data), so they are easy to inspect or produce with standard command-line tools even without any Daf-aware library. ZarrDaf instead lays the files out according to the Zarr v3 specification: the per-node zarr.json metadata and the chunk files are more opaque than FilesDaf's plain text/JSON, but in exchange the directory can be read directly by any Zarr library (e.g. the Python zarr package) without that library having to know anything about Daf.

A Zarr directory is still a directory rather than a single file, so for convenient publication or transport we also support storing a Daf data set inside a single ZIP archive; zipping a Zarr directory (or, actually, a tree containing several such directories) would give a valid Zarr ZIP archive. An advantage of this is that a single ZIP file can hold several Daf repositories, while ZipDaf is restricted to a single repository per zip file.

ZIP archives written by this package hold every flat chunk uncompressed (ZIP method 0) so it can be memory-mapped for direct access just like the directory backend (you should also force this if manually zipping a directory yourself); packed chunks live inside one dual-format shard file per property (see the packed-property notes below), still in the same archive. On the ZIP backend the archive is append-only: properties cannot be deleted and axes cannot be reordered. For read access, any Zarr v2 ZIP archive that matches the internal structure described below is accepted (including ones produced by foreign tools such as Python's zarr package, even if the chunks are chunked and/or compressed, subject to Zarr.jl's support for data types, filters, and compressors). Remote object stores (S3, GCS, …) are not supported.

We use the following internal structure under some root Zarr group (which is not compatible with any specific existing Zarr-based convention such as OME-NGFF ):

The directory will contain 4 sub-groups: scalars, axes, vectors, and matrices, and a daf group attribute.
The daf group attribute signifies that the group contains Daf data. It is a two-element array of integers, the first being the major version number and the second the minor version number, using semantic versioning . This makes it easy to test whether some Zarr group does/n't contain Daf data, and which version of the internal structure it is using. The defined version is [1,0]. The underlying Zarr store is Zarr v3.
The scalars group contains scalar properties, each as a single-element Zarr array. The only supported scalar data types are these included in StorageScalar . If you really need something else, serialize it to JSON and store the result as a string scalar. This should be extremely rare.
The axes group contains a Zarr array per axis, which contains a vector of strings (the names of the axis entries).
The vectors group contains a sub-group for each axis. Each such sub-group contains vector properties. If the vector is dense, it is stored directly as a Zarr array. Otherwise, it is stored as a sub-group containing two child Zarr arrays: nzind containing the indices of the non-zero values, and nzval containing the actual values. See Julia's SparseVector implementation for details. The only supported vector element types are these included in StorageScalar , same as StorageVector .

If the data type is Bool then the data vector is typically all- true values; in this case we simply skip storing the nzval child array.
The matrices group contains a sub-group for each rows axis, which contains a sub-group for each columns axis. Each such sub-sub-group contains matrix properties. If the matrix is dense, it is stored directly as a Zarr array (in column-major layout). Otherwise, it is stored as a sub-group containing three child Zarr arrays: colptr containing the indices of the rows of each column in rowval, rowval containing the indices of the non-zero rows of the columns, and nzval containing the non-zero matrix entry values. See Julia's SparseMatrixCSC implementation for details. The only supported matrix element types are these included in StorageReal - this explicitly excludes matrices of strings, same as StorageMatrix .

If the data type is Bool then the data matrix is typically all- true values; in this case we simply skip storing the nzval child array.
Flat properties (the default) are stored as a single Zarr chunk covering the full array, without compression, so the chunk file on disk is a raw binary image that we can memory-map. Packed properties ( packed = true, uncompressed size at or above DAF_PACKED_TARGET_CHUNK_KB ) are stored as v3 sharded arrays (ZEP-0002) with one shard file per property — directory backend at <name>/c/0[/0], ZIP backend inside one archive entry per property — and chunked + compressed by the codec resolved from DAF_PACKED_COMPRESSION . Both encodings coexist within a single Daf data set without a version bump ( [1,0] covers both).

Daf-written packed shards carry a daf_packed_format attribute on each sharded ZArray. Its value drives the dispatch in zarr_convert (hard-link vs. re-encode) and in the read paths:
- "indexed+zipped" — produced by this package's writer: the shard bytes are simultaneously a valid Zarr v3 sharded array (with a shard index at offset 0) and a valid ZIP archive (with a central directory at the tail). The same shard bytes can hard-link between ZarrDaf and FilesDaf directories, and ZipDaf / FilesDaf / HttpDaf read them via the ZIP central directory while ZarrDaf reads them via the Zarr shard index.
- attribute absent (or any other value) — produced by a foreign Zarr writer: the shard has only the index, no ZIP framing. ZarrDaf still reads via the index; conversion to FilesDaf falls back to rewrite_index_only_as_dual_format_shard.
The root group's zarr.json carries a consolidated_metadata field — an inline index of every per-node zarr.json under the root, so an open / HTTP-served reader does not have to issue one GET per node. Its content is bijective with the consolidated metadata that FilesDaf writes in its metadata.json (see the FilesDaf documentation for the formal mapping; zarr_to_files and files_to_zarr translate between them). The on-disk shape is the one zarr-python 3.x writes, which informally tracks zarr-specs PR #309 (still open as of writing, so the spec PR is not the authoritative reference — zarr-python's ConsolidatedMetadata class is). The field lives at the top level of the root group's zarr.json, sibling to attributes:
```
{
  "zarr_format": 3,
  "node_type": "group",
  "attributes": {"daf": [1, 0]},
  "consolidated_metadata": {
    "kind": "inline",
    "must_understand": false,
    "metadata": {
      "<relative_path>": <full v3 metadata blob>,
      ...
    }
  }
}
```
where <relative_path> is each node's path relative to the root group (no leading slash, no trailing /zarr.json) and the value is the verbatim parsed content of that node's zarr.json — i.e. the same MetadataV3 that the per-node file holds, including its own attributes. kind is always "inline"; must_understand is always false. Order of keys inside metadata is not significant for interop; zarr-python's writer happens to sort by (depth, casefolded NFKC name) but its reader does not require it. On every property set! / delete! we update the field in the root zarr.json (full file rewrite per operation, which is unavoidable since zarr.json is one document; we cache the serialized bytes of the metadata sub-dict and append in place rather than re-serializing every existing entry, so per- set! CPU work is O(size-of-one-descriptor)). On every open (read or write), if the field is missing we attempt to rebuild it by walking the per-node zarr.json files; the rebuild is best-effort with the same swallow-on-read-only-frozen-filesystem semantics as FilesDaf's metadata.json.

Example Zarr v3 directory structure (every group and every array has its own zarr.json; an array's chunk data lives under its c/ directory — c/0 for a 1D array, c/0/0 for a 2D array; the root group's zarr.json holds the daf attribute and the consolidated metadata):

example-daf-dataset-root-directory.daf.zarr/
├─ zarr.json                     # root group (attributes.daf = [1, 0], consolidated_metadata)
├─ scalars/
│  ├─ zarr.json
│  └─ version/
│     ├─ zarr.json
│     └─ c/0
├─ axes/
│  ├─ zarr.json
│  ├─ cell/
│  │  ├─ zarr.json
│  │  └─ c/0
│  └─ gene/
│     ├─ zarr.json
│     └─ c/0
├─ vectors/
│  ├─ zarr.json
│  ├─ cell/
│  │  ├─ zarr.json
│  │  └─ batch/
│  │     ├─ zarr.json
│  │     └─ c/0
│  └─ gene/
│     ├─ zarr.json
│     └─ is_marker/
│        ├─ zarr.json
│        └─ c/0
└─ matrices/
   ├─ zarr.json
   ├─ cell/
   │  ├─ zarr.json
   │  └─ gene/
   │     ├─ zarr.json
   │     ├─ UMIs/                # sparse → sub-group with one array per component
   │     │  ├─ zarr.json
   │     │  ├─ colptr/
   │     │  │  ├─ zarr.json
   │     │  │  └─ c/0
   │     │  ├─ rowval/
   │     │  │  ├─ zarr.json
   │     │  │  └─ c/0
   │     │  └─ nzval/
   │     │     ├─ zarr.json
   │     │     └─ c/0
   │     └─ fractions/           # dense, packed → single v3 sharded array
   │        ├─ zarr.json
   │        └─ c/0/0
   └─ gene/
      ├─ zarr.json
      ├─ cell/
      └─ gene/

Note

Zarr.jl maps Julia's column-major arrays onto Zarr v3's row-major model by listing the zarr.json shape in the reverse of the Daf (Julia) matrix shape, so the raw chunk bytes match Julia's native column-major layout. A Daf matrix whose (rows_axis, columns_axis) are (cell, gene) (a Julia (n_cells, n_genes) matrix) is therefore written with zarr.json containing "shape": [n_genes, n_cells]. A client using a different Zarr implementation — most notably Python's zarr package — reads this as a C-contiguous NumPy array of shape (n_genes, n_cells), which is the transpose of the Daf (Julia) view. The bytes on disk are identical; only the shape labels are swapped. To obtain the Daf-canonical (cell, gene) orientation in Python, apply .T (a zero-copy view) to the loaded array. This affects only dense matrices (the colptr/ rowval/ nzval child arrays of sparse matrices are 1D vectors, unaffected); 1D axis-entry arrays and vector properties have the same shape in both languages.

Note

The code here assumes the Zarr data obeys all the above conventions and restrictions. As long as you only create and access Daf data in Zarr directories using ZarrDaf , then the code will work as expected (assuming no bugs). However, if you do this in some other way (e.g., a Zarr library in another language producing compressed or multi-chunk arrays), and the result is invalid, then the code here may fail with "less than friendly" error messages.

DataAxesFormats.ZarrFormat.MAJOR_VERSION

— Constant

The major version of the ZarrDaf on-disk format supported by this code.

DataAxesFormats.ZarrFormat.MINOR_VERSION

— Constant

The highest minor version of the ZarrDaf on-disk format supported by this code.

DataAxesFormats.ZarrFormat.ZarrDaf

— Type

ZarrDaf(
    path::AbstractString,
    mode::AbstractString = "r";
    [name::Maybe{AbstractString} = nothing,
    packed::Bool = false]
)

Storage in a Zarr directory tree, Zarr ZIP archive, or remote HTTP(S) Zarr group.

The path is a filesystem path that follows one of these conventions:

something.daf.zarr — a Zarr directory containing a single Daf data set at its root.
something.daf.zarr.zip — a Zarr ZIP archive containing a single Daf data set at its root.
something.dafs.zarr.zip#/group — a Zarr ZIP archive containing Daf data sets in sub-groups, addressed by group.
http://… or https://… — a URL pointing at a remote Zarr directory that contains a Daf data set, served over HTTP (e.g. via a static file server, HTTP.serve(store, path, …), or xpublish). Only mode = "r" is supported; the HTTP backend is strictly read-only and returns a DafReadOnly . The remote root zarr.json must carry an inline consolidated_metadata field (the on-disk shape zarr-python 3.x writes — see the module docstring), and the served content must be stable for the lifetime of the open handle: per-chunk GETs happen lazily, so if the underlying data set is rewritten or relocated while the handle is open, subsequent reads may see inconsistent bytes.

The backend (directory, ZIP, or HTTP) is selected from the path prefix / file-name suffix. The ZIP backend is append-only: properties cannot be deleted and axes cannot be reordered (attempts to do so raise an error).

Note

If you create a directory whose name is something.dafs.zarr.zip# and place Daf ZIP archives in it, this scheme will fail. So don't.

When opening an existing data set, if name is not specified, and there exists a "name" scalar property, it is used as the name. Otherwise, the path (including any #/group suffix) will be used as the name.

If packed is true, subsequent writes through this handle default to the packed (chunked + compressed) on-disk encoding for properties whose uncompressed size is at or above DAF_PACKED_TARGET_CHUNK_KB . Per-call packed kwargs on set_*! / empty_*! / copy_*! override this default. The default is false (flat single-chunk uncompressed encoding). HTTP-mode opens are read-only and ignore the packed kwarg.

The valid mode values are as follows (the default mode is r):

Mode	Allow modifications?	Create if does not exist?	Truncate if exists?	Returned type
`r`	No	No	No	`DafReadOnly`
`r+`	Yes	No	No	`ZarrDaf`
`w+`	Yes	Yes	No	`ZarrDaf`
`w`	Yes	Yes	Yes	`ZarrDaf`

Truncating a sub-daf inside a ZIP archive is not supported (because the ZIP backend is append-only) and raises an error; use r+ or w+ to open a sub-daf for writing without truncation.

Note

When several ZarrDaf instances in the same process share a ZIP archive path (typically different #/group sub-dafs of the same .dafs.zarr.zip file, or repeated opens of the same single-daf .daf.zarr.zip), they share a single underlying MmapZipStore and a single data_lock, so that concurrent calls serialize correctly and the archive is never mmap-ed twice. The first such open determines the store's writability: a later open of the same archive that requests write access will raise an error if the first open was read-only. Release the read-only handle first, or open the writable instance first. The directory backend does not share a store — each open creates its own independent DirectoryStore over the same filesystem tree.

Note

.daf.zarr.zip archives written by this code do not carry the inline consolidated_metadata field in their root zarr.json, because the ZIP central directory plays the same enumeration role. Consequently, an unzip foo.daf.zarr.zip -d foo.daf.zarr/ produces a directory whose root zarr.json lacks consolidated_metadata. Before exposing such a directory over HTTP, open it once locally with ZarrDaf("foo.daf.zarr") (any mode) so ensure_consolidated_metadata! builds it. Symmetrically, when ZarrDaf appends to a .daf.zarr.zip whose root zarr.json does have an inline consolidated_metadata (e.g. from a zip -r of a directory daf), the field is stripped from the rewritten root zarr.json so subsequent unzips can't see a stale snapshot.

Warning

The byte-surgery append on consolidated_metadata.metadata and the atomic stage-rename rewrite of root zarr.json assume a single writer at a time. The in-process Daf write lock serializes writers within one Julia process; opening the same .daf.zarr directory from multiple processes (or multiple machines, e.g. NFS) and writing concurrently can interleave rewrites and corrupt the consolidated metadata. Open writable from one process at a time.

DataAxesFormats.ZarrFormat.DAF_ZARR_ZIP_MAX_FILE_SIZE

— Constant

The virtual address reservation size used for writable MmapZipStore opens of a ZarrDaf (modes r+, w+, w). Each such open reserves this much virtual address space via a single anonymous PROT_NONE mapping and overlays the real file onto its first filesize bytes; subsequent ftruncate + re-overlay calls extend the accessible portion as the archive grows. The physical file stays at its real size — only VA is reserved. Defaults to 128 GiB, leaving plenty of room for concurrent live stores on platforms with ~128 TiB of user VA (Apple Silicon). Set to a larger value before opening a ZarrDaf whose ZIP archive might grow past this bound. An append that would cross the bound fails with an explicit error pointing back here.

Zarr Format

Index