Packed Format

Configuration globals and the shared implementation of the packed (chunked + compressed) on-disk encoding used by Daf storage formats.

Packed encoding stores large dense matrices and sparse-matrix components as chunked, compressed arrays on disk and over the wire, in exchange for paying decompression CPU at read time. It is most useful when the data lives on a slow tier (NFS, HTTP, archival storage) where the bandwidth saving outweighs the read overhead; for compute-intensive work on a fast local SSD, prefer staging an unpacked copy via copy_all! .

The exported globals in this module are the only knobs tuning the packed encoding. They apply process-wide; there is no per-daf override.

Dual-format shard layout

A packed property is stored as a single shard — one file ( <name>.zip in FilesFormat , one archive entry per property in ZipFormat ) holding all of the property's inner chunks. A shard written by this package is dual-format : the same bytes are simultaneously

• a valid Zarr v3 sharded array (ZEP-0002) — a shard index at file offset 0 maps each inner chunk to a (offset, n_bytes) byte range, so a Zarr reader decodes chunks by index lookup; and
• a valid ZIP archive — a central directory at the file tail lists one entry per inner chunk, so a ZIP reader (and this package's FilesDaf / ZipDaf / HttpDaf backends) decodes chunks through the central directory.

The two indices point into the same chunk bytes; the dead bytes each format ignores (the Zarr index for ZIP readers, the ZIP local file headers for Zarr readers) are tolerated by both. Which indices are present is recorded by the "packed_format" key in a FilesDaf / ZipDaf JSON descriptor and, in parallel, by the daf_packed_format attribute on a ZarrDaf sharded array. This package writes "indexed+zipped" for the dual-format shard above in both places. A foreign source carries one of:

• a FilesDaf / ZipDaf descriptor with "packed_format" => "zipped" — a ZIP-only archive of chunks with no leading Zarr index; or
• a ZarrDaf array with no daf_packed_format attribute — a Zarr-only sharded array with no ZIP framing (conceptually "indexed"; there is no literal "indexed" value, the absence of the attribute is the signal).

Random-access central directory

The ZIP central-directory entries are fixed-length and written in chunk order , so a chunk's entry is located by index arithmetic ( cd_offset + chunk_index * cd_entry_size ) without scanning the directory. This is a hard requirement of the format — readers assert it and a 3rd-party producer must honor it. The fixed length implies a fixed inner-chunk entry-name width: for STORED / ZSTD shards the name is c/<i_N>/.../<i_1> with each dimension zero-padded to a fixed width; for DEFLATE shards the name field is repurposed (see below).

Per-codec ZIP method and self-description

Each chunk entry is stamped, where possible, with the ZIP compression method that matches its codec, so that a generic ZIP tool which recognises the method decompresses the chunk to its uncompressed bytes automatically:

• zstd → ZIP method 93 . The entry data is the raw zstd frame, which is exactly what the Zarr zstd codec consumes, so both readers share it directly.
• gzip → ZIP method 8 (DEFLATE). A ZIP method-8 entry needs a raw DEFLATE payload, but a Zarr gzip codec needs a full gzip stream ( [header][DEFLATE][trailer] ). Both are satisfied at once: the 10-byte gzip header is placed in the entry's name field (binary but NUL-free), the entry data is the raw DEFLATE payload, and the 8-byte gzip trailer follows as dead bytes. A single contiguous Zarr range from the name start spans a valid gzip stream, while the ZIP entry points only at the DEFLATE payload.
• blosc, bitshuffle variants, or any codec with no matching ZIP method → method 0 (STORED). The entry data is the codec output verbatim, which a generic ZIP tool cannot decode. For these shards a final codec.json entry (after the last chunk, before the central directory) records the inner codec pipeline as a Zarr v3 codec list, so an external tool with a Zarr v3 codec implementation can decode the otherwise-opaque STORED bytes. This package's own readers ignore codec.json — they recover the codec from the descriptor — and the Zarr shard index has slots only for the inner chunks, so it is invisible to that read path too.

On read, the chunk decoder validates that a chunk's ZIP method matches what its descriptor codec implies and errors otherwise — a STORED entry under a zstd descriptor, for example, would be contradictory metadata.

The target uncompressed size in kilobytes for a single chunk (page) of a packed property. Doubles as the threshold below which a property is stored or fetched flat instead of being chunked. Properties whose uncompressed size is below this value get the flat (mmap-friendly single-chunk uncompressed) on-disk path on every backend regardless of packed=true . (The HTTP backend uses the same threshold to gate stripe-synthesized fetches versus single- GET fetches; see the HTTP backend documentation.)

Default 8 (8 KB) — page-sized, enabling sub-column slice access (the common K-marker query pattern fetches the first page of each of K columns rather than the full column). Small enough to keep network slice fetches cheap, but large enough that compression codecs still produce reasonable ratios.

Kilobytes are binary (1 KB = 1024 bytes), matching OS page size and the conventions of DAF_PACKED_LOCAL_CACHE_KB and DAF_PACKED_HTTP_CACHE_KB .

To tune: DataAxesFormats.PackedFormat.DAF_PACKED_TARGET_CHUNK_KB = 16 at the top of your script or REPL session.

The compression codec used for packed properties. The default :blosc_zstd_bitshuffle combines zstd compression with a bitshuffle pre-filter, giving good ratios for integer-typical scientific data (e.g. UMI counts, gene indices) where bitshuffle isolates high-zero bytes; for floats with clustered exponents it does similarly well on the exponent bytes.

Supported codecs:

Symbol	Zarr backend	HDF5 backend	Plug-in needed for non-Julia consumers
:blosc_zstd_bitshuffle (default)	BloscCompressor(cname="zstd", shuffle=BITSHUFFLE)	H5Zblosc filter	HDF5 readers only
:blosc_lz4_bitshuffle	BloscCompressor(cname="lz4", shuffle=BITSHUFFLE)	H5Zblosc filter	HDF5 readers only
:zstd_bitshuffle	bitshuffle filter + ZstdCompressor	H5Zbitshuffle + H5Zzstd filters	HDF5 readers; Zarr readers for bitshuffle
:zstd	ZstdCompressor	H5Zzstd filter	HDF5 readers only
:gzip	ZlibCompressor	built-in deflate	None
:gzip_shuffle	ZlibCompressor + byte-shuffle filter	built-in deflate + built-in shuffle	None

The "plug-in needed" column reflects what a consumer of the produced files needs to add beyond a stock install of their HDF5 / Zarr library to be able to read the data. :gzip and :gzip_shuffle use only HDF5 / Zarr built-in filters and require no plug-ins anywhere. All other codecs require the consumer to load filter libraries; the exact recipes per language are listed below.

Plug-in installation by codec and language , for tools that need to read the produced files:

Codec	Python	R
:gzip , :gzip_shuffle	none	none
:blosc_zstd_bitshuffle , :blosc_lz4_bitshuffle	hdf5plugin for HDF5	rhdf5filters for HDF5
:zstd	hdf5plugin for HDF5	rhdf5filters for HDF5
:zstd_bitshuffle	hdf5plugin for HDF5; bitshuffle for Zarr	rhdf5filters for HDF5

In Julia, no extra installation step is needed: DataAxesFormats declares H5Zblosc , H5Zzstd , and (where available) H5Zbitshuffle as direct dependencies and loads them at module init, so using DataAxesFormats is enough to register all required filters with HDF5.jl's filter registry. Zarr.jl already ships with blosc, zstd, and zlib built in; the bitshuffle Zarr filter is registered on the same module init when its adapter is available.

Install commands for non-Julia consumers:

• Python : pip install hdf5plugin (one library covers blosc / zstd / lz4 / bitshuffle for HDF5); pip install zarr brings numcodecs which already includes blosc, zstd, and zlib; pip install bitshuffle adds bitshuffle support for Zarr ( :zstd_bitshuffle ).
• R : BiocManager::install("rhdf5filters") covers blosc, zstd, lz4, and bitshuffle for rhdf5 .

Setting DAF_PACKED_COMPRESSION to any value outside the table causes a runtime error listing the supported codecs at the first write that needs to resolve the codec.

To tune: DataAxesFormats.PackedFormat.DAF_PACKED_COMPRESSION = :gzip_shuffle for plug-in-free interop with vanilla HDF5 / Zarr tooling, at the cost of weaker compression ratios.

The compression level passed to the inner codec ( zstd / lz4 / zlib ). Default 5 , the standard Blosc clevel default — a balanced speed-vs-ratio choice for the default :blosc_zstd_bitshuffle codec.

Higher levels (e.g. 9 ) produce smaller files at the cost of slower writes; reads are roughly unaffected by level. Lower levels ( 1 ) are faster to write at the cost of larger files.

The numeric meaning of the level varies per codec — Blosc and zlib use a 1:9 scale, zstd uses a 1:22 scale. The codec-specific valid range is enforced when the codec is resolved. Level 0 (which means "no compression" in Blosc and zlib , and "library default" in zstd ) is excluded for all codecs because picking a packed codec implies you actually want compression applied.

The cache size in kilobytes used by the per-property DiskArrays.cache LRU when reading packed properties on local-disk backends. Larger caches reduce repeat decompression cost when scattered scalar access patterns revisit the same chunks.

Default 65536 (64 MiB). Each get_matrix / get_vector call on a packed local-disk property returns a DiskArrays.CachedDiskArray wrapper sized at this value (in bytes, after multiplying by 1024); the cache is held alive by the daf's internal cache ( MemoryData cache group) and released by empty_cache! .

Kilobytes are binary (1 KB = 1024 bytes), consistent with DAF_PACKED_TARGET_CHUNK_KB .

This is independent of the HTTP cache (see DAF_PACKED_HTTP_CACHE_KB ) because re-fetching over the network is much more expensive than re-decompressing on local disk.

The cache size in kilobytes used by the per-property DiskArrays.cache LRU when reading packed properties (or stripe-synthesised unpacked properties) over HTTP.

Default 262144 (256 MiB), meaningfully larger than DAF_PACKED_LOCAL_CACHE_KB because re-fetches over HTTP are far more expensive (network round-trip + bandwidth) than local re-decompressions.

Kilobytes are binary (1 KB = 1024 bytes), consistent with DAF_PACKED_TARGET_CHUNK_KB .

The maximum gap (in kilobytes) between two needed byte ranges that the HTTP read path coalesces into a single Range GET. When materialising a slice that needs chunks c1 and c2 with an unneeded gap of g bytes between them, the path issues one Range GET covering [c1_start, c2_end) if g ≤ DAF_HTTP_MAX_COALESCE_GAP_KB * 1024 , otherwise two separate Range GETs.

Default matches DAF_PACKED_TARGET_CHUNK_KB — i.e., coalesce across at most one chunk's worth of unneeded bytes per gap.

Kilobytes are binary (1 KB = 1024 bytes).

StripedVector(::Type{T}, n_elements::Integer, stripe_n_elements::Integer, byte_fetcher::Function)::ChunkedArray{T, 1}

Factory for a lazy 1-D DiskArrays.AbstractDiskArray that fetches a flat dense numeric vector served over HTTP in stripes (Range GETs of stripe_n_elements elements at a time, coalesced when adjacent).

StripedMatrix(::Type{T}, n_rows::Integer, n_columns::Integer, stripe_n_rows::Integer, byte_fetcher::Function)::ChunkedArray{T, 2}

Factory for a lazy 2-D DiskArrays.AbstractDiskArray that fetches a flat dense numeric column-major matrix served over HTTP in column tiles of shape (stripe_n_rows, 1) (coalesced when adjacent).

PackedDenseArray(
    ::Type{T},
    shape::NTuple{N, Int},
    chunk_shape::NTuple{N, Int},
    codec::PackedCodec,
    index_location::Symbol,
    byte_fetcher::Function,
    suffix_byte_fetcher::Function,
)::ChunkedArray{T, N}

Factory for a lazy N-dimensional DiskArrays.AbstractDiskArray that fetches a v3-sharded packed dense array served over HTTP. The shard index footer is fetched once on first read (one Range GET via suffix_byte_fetcher when index_location == :end , otherwise via byte_fetcher(0, index_size) ); subsequent reads look up per-chunk byte ranges in the cached index, coalesce adjacent ones, and decode each fetched chunk through the codec pipeline.

ZipShardArray(
    ::Type{T},
    shape::NTuple{N, Int},
    chunk_shape::NTuple{N, Int},
    codec::PackedCodec,
    byte_fetcher::Function,
    suffix_byte_fetcher::Function,
)::ChunkedArray{T, N}

Factory for a lazy N-dimensional DiskArrays.AbstractDiskArray that reads a ZIP-archive shard file via its central directory. byte_fetcher and suffix_byte_fetcher abstract over the source (HTTP range GETs, an mmap of a local file, the in-memory bytes of an outer-ZIP entry).

The EOCD region (a fixed-size tail block) is fetched once on first read to learn the CD's absolute offset; each chunk's CD entry is then random-accessed by cd_offset + (chunk_index - 1) * cd_entry_size (fixed-size entries, see inner_chunk_name_bytes ) and parsed on demand. No code walks the whole CD. Per-chunk reads coalesce adjacent byte ranges and decode through decode_zip_entry .