Zip Files Format

A Daf storage format that packs one or more FilesDaf on-disk layout verbatim into a single ZIP archive. This is a convenient single-file form for publication and transport: copy one *.daf[s].zip instead of recursively copying a directory tree, without giving up the zero-copy memory-mapped access that FilesDaf provides.

The encoding is the FilesDaf encoding — every entry name inside the archive is the relative path of the matching FilesDaf file ( daf.json , scalars/<name>.json , axes/<axis>.txt , vectors/<axis>/<name>.{json,txt,data,nzind,nzval,nztxt} , matrices/<rows>/<cols>/<name>.{json,data,txt,colptr,rowval,nzval,nztxt} ), and every blob is the same little-endian byte stream. As a consequence:

• Unzipping a *.daf[s].zip produced by this format yields directories that FilesDaf opens directly. The unzipped tree might not contain the metadata.json consolidated index FilesDaf uses for HttpDaf enumeration — it is rebuilt automatically by FilesDaf.ensure_metadata_json! on the first local open on a writable filesystem (of each separate files repository, if there is more than one).
• Conversely, zip -r foo.daf.zip foo.daf/ over an existing FilesDaf directory produces a valid ZipDaf . Since the source directory may hold a (possible stale, ignored) metadata.json , that entry can get bundled inside the archive too — ZipDaf strips it from the central directory on every writable open to ensure stale metadata is not retained. After unzipping such a stripped archive, the next writable FilesDaf open rebuilds metadata.json from scratch, so the consolidated index is always coherent with the directory tree.

Dense numeric blobs (and colptr / rowval / nzind / nzval for sparse ones) are served zero-copy via memory-mapped views of the archive file when possible; foreign archives whose entries are compressed (DEFLATE, DEFLATE64, BZIP2, or ZSTD) or unaligned fall back to in-memory decoded copies. The only modification allowed to writable ZIP repositories is appending new data: no entry deletes or overwrites, and no axis reorder.

Path conventions, mirroring ZarrDaf :

• something.daf.zip — single Daf at the archive root. A #/group fragment in this form is rejected.
• something.dafs.zip#/group — a multi-Daf archive holding several Daf data sets under sub-paths; group selects one and must be non-empty. A bare something.dafs.zip without a #/group fragment is rejected.

Example archive entry layout (single-Daf form; the multi-Daf form prefixes every entry with <group>/ ):

daf.json
scalars/version.json
axes/cell.txt
axes/gene.txt
vectors/cell/batch.json
vectors/cell/batch.txt
vectors/gene/is_marker.json
vectors/gene/is_marker.data
matrices/cell/gene/UMIs.json
matrices/cell/gene/UMIs.colptr
matrices/cell/gene/UMIs.rowval
matrices/cell/gene/UMIs.nzval
matrices/cell/gene/fractions.json   # dense, packed
matrices/cell/gene/fractions.zip

Packed (chunked + compressed) properties are stored as one <name>.zip archive entry per property (or <name>.<component>.zip per packed sparse component) — a nested ZIP archive of the inner chunks. ZipDaf reads the inner chunks through that shard's own central directory, the same code path as FilesDaf and HttpDaf ; the "packed_format" descriptor key and the shard layout are documented in FilesFormat and PackedFormat .

The valid mode values are as follows (the default mode is r ):

Mode	Allow modifications?	Create if does not exist?	Truncate if exists?	Returned type
r	No	No	No	DafReadOnly
r+	Yes	No	No	ZipDaf
w+	Yes	Yes	No	ZipDaf
w	Yes	Yes	Yes	ZipDaf

Truncating a sub-daf inside a .dafs.zip is not supported (the ZIP backend is append-only) and raises an error; use r+ or w+ to open a sub-daf for writing without truncation.

The virtual address reservation size used for writable MmapZipStore opens of a ZipDaf (modes r+ , w+ , w ). Each such open reserves this much virtual address space via a single anonymous PROT_NONE mapping and overlays the real file onto its first filesize bytes; subsequent ftruncate + re-overlay calls extend the accessible portion as the archive grows. The physical file stays at its real size — only VA is reserved. Defaults to 128 GiB, leaving plenty of room for concurrent live stores on platforms with ~128 TiB of user VA (Apple Silicon). Set to a larger value before opening a ZipDaf whose ZIP archive might grow past this bound. An append that would cross the bound fails with an explicit error pointing back here.

ZipDaf(
    path::AbstractString,
    mode::AbstractString = "r";
    [name::Maybe{AbstractString} = nothing,
    packed::Bool = false]
)

Storage in a single ZIP archive of the FilesDaf on-disk layout. See the module documentation for the path conventions, the mode table, the on-disk entry layout, the sub-daf sharing semantics, and the relationship to unzip / zip -r round-trips.

When opening an existing data set, if name is not specified, and there exists a name scalar property, it is used as the name. Otherwise, the path (including any #/group fragment) is used as the name.

If packed is true , subsequent writes through this handle default to the packed (chunked + compressed) on-disk encoding for properties whose uncompressed size is at or above DAF_PACKED_TARGET_CHUNK_KB . Per-call packed kwargs on set_*! / empty_*! / copy_*! override this default. The default is false (today's flat encoding).

A weak-cache entry pairing an open MmapZipStore with the data_lock shared by every sub-daf of that archive in the same process. MmapZipStore is stateful (single io_stream , archive-wide mmap, appended-entry mmap table), so multiple sub-dafs of the same archive must share one instance to avoid corrupting the archive with two independent writers. is_writable records the mode the store was opened in, so that a later sub-daf opening the same archive in an incompatible mode can be rejected cleanly.

acquire_shared_mmap_zip_store!(;
    container_path::AbstractString,
    is_read_only::Bool,
    create_if_missing::Bool,
    truncate::Bool,
    max_file_size::Integer,
)::SharedMmapZipStoreHandle

Acquire (or open, if not already open in this process) the SharedMmapZipStoreHandle for the ZIP archive at container_path . The handle is registered in a process-wide weak cache keyed on container_path , so concurrent opens of the same archive in the same process share one underlying MmapZipStore and one data_lock regardless of which Daf format ( ZipDaf or ZarrDaf ) opened it. This is deliberate: MmapZipStore is stateful (single io_stream , archive-wide mmap, appended-entry mmap table), so two independent stores over the same file would corrupt each other on writes.

If the archive is already cached as a read-only store and the new open requests write access, this raises an explicit error pointing at the conflict; release the read-only handle first, or open the writable instance first.

parse_zip_archive_path(
    path::AbstractString;
    single_daf_suffix::AbstractString,
    multi_dafs_suffix::AbstractString,
    multi_dafs_marker::AbstractString,
    format_name::AbstractString,
)::Maybe{Tuple{String, Maybe{String}}}

Parse path as either a single-Daf archive ( *<single_daf_suffix> , no #/group fragment) or a multi-Daf archive ( *<multi_dafs_suffix><#/group> , group required and non-empty). Return (container_path, group_or_nothing) for either valid form. Return nothing if path matches neither form, leaving the caller free to try further format-specific suffix recognition (e.g. ZarrFormat 's .daf.zarr directory case) before issuing its own catch-all error.

Raises an explicit error for the following ill-formed near-miss cases:

• *<single_daf_suffix>#/group — a singular path with a sub-daf fragment; can't address a sub-daf in a singular <format_name> path … .
• *<multi_dafs_suffix> (no #/group ) — a plural path missing its sub-daf; missing '#/<group>' in plural <format_name> path … .
• *<multi_dafs_marker> (empty group) — empty group name after '#/' in <format_name> path … .

format_name is the human-readable label used in error messages ( ZipDaf , ZarrDaf ).