Files Format

A Daf storage format in disk files. This is an efficient way to persist Daf data in a filesystem, and offers a different trade-off compared to storing the data in an HDF5 file.

On the upside, the format of the files is so simple that it is trivial to access them from any programming environment, without requiring a complex library like Zarr or HDF5. In addition, since each scalar, vector or matrix property is stored in a separate file, deleting data automatically frees the storage (unlike in an HDF5 file, where you must manually repack the file to actually release the storage). Also, you can use standard tools to look at the data (e.g. use ls or the Windows file explorer to view the list of properties, how much space each one uses, when it was created, etc.). Most importantly, this allows using standard tools like make to create automatic repeatable processing workflows.

For packed (chunked + compressed) vectors and matrices, FilesDaf stores the entire property as one shard file ( <name>.zip for dense, plus the per-component packed .zip s under sparse properties). FilesDaf reads the inner chunks of such a shard through its ZIP central directory, decoding each chunk on demand and caching the decoded chunks. The shard layout, codec catalogue, and on-disk write protocol are documented in PackedFormat . It is still relatively simple and would be accessible to any tool which can look inside ZIP files.

On the downside, this being a directory, you need to create a zip archive file if you want to publish it. This will give you a valid ZipDaf file which you can access directly. However such files aren't easy to modify - you can append new data to them but that's it. Such a zip file would still be relatively easily accessible to tools that can look inside a zip file (and you can unzip such a file to get a valid repository directory (but see the section about metadata.json below). If you want a truly modifiable single-file format, you should use the H5DF format (which has its own trade-offs).

This format is very close but not identical to the Zarr DirectoryStore format. Specifically, the binary blob files (numeric matrices and vectors) are byte-identical between the formats (we make a special effort to make it so even for packed numeric data). The format here has the advantage that the rest of the meta/data is easily accessed by standard tools - all metadata are simple JSON files, text vector/matrix data is stored in one-entry-per-line files, etc. The Zarr format is more opaque - one can't really access it other than through a Zarr library. So things like wc repo.daf/axes/gene.txt or grep -in Fox repo.daf/axes/gene.txt will work here but not in Zarr.

We use multiple files to store Daf data, under some root directory, as follows:

• The directory will contain 4 sub-directories: scalars , axes , vectors , and matrices , and two files at the root: daf.json (always) and metadata.json (a consolidated index, regenerated on demand — see below).

• The daf.json signifies that the directory contains Daf data. In this file, there should be a mapping with a version key whose value is an array of two integers. The first is the major version number and the second is the minor version number, using semantic versioning . This makes it easy to test whether a directory does/n't contain Daf data, and which version of the internal structure it is using. Defined versions are [1,0] and [1,1] . New code emits [1,1] ; the reader accepts both. The on-disk difference is the JSON descriptor for sparse properties (see below) — the binary data files are unchanged across versions.

• The metadata.json is a consolidated index of every property's descriptor. After it has been seeded once (by walking the tree on a writable open), subsequent opens consume it directly instead of walking the tree, and an HTTP-served FilesDaf can be browsed without per-property readdir round-trips. Its content is bijective with the per-property descriptors documented below, and with the same consolidated metadata that ZarrDaf embeds in its root zarr.json (see the ZarrDaf documentation for the formal mapping; zarr_to_files and files_to_zarr translate between them). The file is a single-line JSON object mapping each property's relative path to its descriptor: {"<relative_path>":<descriptor>,...} , where <relative_path> is the property's location relative to the root (e.g. vectors/cell/batch , matrices/cell/gene/UMIs , axes/cell , scalars/version ) and <descriptor> is byte-identical to the per-property sidecar JSON content described below (for axes, the descriptor is {"format":"axis","n_entries":<N>} ). On set! the file is appended in place via byte-level surgery: truncate the trailing } and write ,"<new_path>":<descriptor>} (or "<new_path>":<descriptor>} if the file was the empty object {} ). On delete! the file is rebuilt from scratch by walking the tree.

  This file does NOT exist (or, if it does, is ignored) in a single-file ZipDaf repository, because we use the central directory to efficiently access the metadata there, and we want to allow efficient append to the zipped repository. To compensate for this, on every open (read or write), if the file is missing or fails to parse we attempt to rebuild it by walking the tree. If it fails because the underlying filesystem is read-only, the error is swallowed for read-only opens. So the workflow unzip foo.daf.zip; open foo.daf works as long as the filesystem is writable. This flow is required if you want to serve the data over HTTP as this relies on the metadata file existing.

• The scalars directory contains scalar properties, each as in its own name.json file, containing a mapping with a type key whose value is the data type of the scalar (one of the StorageScalar types, with String for a string scalar) and a value key whose value is the actual scalar value.

• The axes directory contains a name.txt file per axis, where each line contains a name of an axis entry.

• The vectors directory contains a directory per axis, containing the vectors. For every vector, a name.json file will contain a mapping with an eltype key specifying the type of the vector element, and a format key specifying how the data is stored on disk, one of dense and sparse .

  If the format is dense , then there will be a file containing the vector entries, either name.txt for strings (with a value per line), name.data for flat binary data (which we can memory-map for direct access), or name.zip for a packed (chunked + compressed) binary payload (see the packed-property note below).

  If the format is sparse , then in v1.1 the JSON contains a per-property descriptor for each component: nzind and nzval , each shaped like a stand-alone vector descriptor. The component bytes live in name.nzind (indices of the non-zero entries) and name.nzval (values of the non-zero entries) for flat components, or name.nzind.zip / name.nzval.zip for packed components (each component is independently packed). Flat components are memory-mappable. See Julia's SparseVector implementation for details. The legacy v1.0 schema instead writes top-level eltype and indtype keys; the reader accepts both shapes.

  If the data type is Bool then the data vector is typically all- true values; in this case we simply skip storing it.

  We switch to using this sparse format for sufficiently sparse string data (where the zero value is the empty string). This isn't supported by SparseVector because "reasons" so we load it into a dense vector. In this case we name the values file name.nztxt .

• The matrices directly contains a directory per rows axis, which contains a directory per columns axis, which contains the matrices. For each matrix, a name.json file will contain a mapping with an eltype key specifying the type of the matrix element, and a format key specifying how the data is stored on disk, one of dense and sparse .

  If the format is dense , then there will be a name.data binary file in column-major layout (which we can memory-map for direct access), or a name.zip packed shard (see the packed-property note below).

  If the format is sparse , then in v1.1 the JSON contains a per-property descriptor for each component: colptr , rowval , and nzval , each shaped like a stand-alone vector descriptor. The component bytes live in name.colptr , name.rowval (indices of the non-zero values) and name.nzval (values of the non-zero entries) for flat components, or name.<component>.zip for packed components. Flat components are memory-mappable. See Julia's SparseMatrixCSC implementation for details. The legacy v1.0 schema instead writes top-level eltype and indtype keys; the reader accepts both shapes.

  If the data type is Bool then the data vector is typically all- true values; in this case we simply skip storing it.

  We switch to using this sparse format for sufficiently sparse string data (where the zero value is the empty string). This isn't supported by SparseMatrixCSC because "reasons" so we load it into a dense matrix. In this case we name the values file name.nztxt .

• Packed (chunked + compressed) properties carry an extra "packed_format" key in their JSON descriptor:

  • "indexed+zipped" — produced by this package's writer. The .zip payload is a dual-format shard: it is simultaneously a valid ZIP archive (central directory at the tail) and a valid Zarr v3 sharded array (shard index at offset 0), so the very same bytes are readable both ways.
  • "zipped" — a ZIP archive of inner chunks with no leading shard index (e.g. written by a tool that only produced the ZIP framing).

  FilesDaf reads packed properties through the ZIP central directory in both cases. We require the central directory entries to be fixed-length and in chunk order, so a chunk's entry is located by index arithmetic rather than by scanning the directory. We also try to stamp each chunk with its codec-specific ZIP method ( 93 for zstd, 8 for gzip) so generic ZIP tools that recognise those method codes decompress the chunk to its uncompressed bytes automatically. Codecs with no matching ZIP method (blosc and the bitshuffle variants) are stored with method 0 (STORED); for those shards a final codec.json entry records the codec pipeline so an external tool can decode the otherwise-opaque STORED bytes. The packed-shard layout, codec catalogue, and on-disk write protocol are documented in PackedFormat .

Example directory structure:

example-daf-dataset-root-directory/
├─ daf.json
├─ metadata.json
├─ scalars/
│  └─ version.json
├─ axes/
│  ├─ cell.txt
│  └─ gene.txt
├─ vectors/
│  ├─ cell/
│  │  ├─ batch.json
│  │  └─ batch.txt
│  └─ gene/
│     ├─ is_marker.json
│     └─ is_marker.data
└─ matrices/
   ├─ cell/
   │  ├─ cell/
   │  └─ gene/
   │     ├─ UMIs.json            # sparse, flat components
   │     ├─ UMIs.colptr
   │     ├─ UMIs.rowval
   │     ├─ UMIs.nzval
   │     ├─ fractions.json       # dense, packed
   │     └─ fractions.zip
   └─ gene/
      ├─ cell/
      └─ gene/

character (regardless of the OS used). Therefore, you can't have a line break inside an axis entry name or in a vector property value, at least not when storing it in FilesDaf`.

That's all there is to it. The format is intentionally simple and transparent to maximize its accessibility by other (standard) tools. Still, it is easiest to create the data using the Julia Daf package.

The specific major version of the FilesDaf format that is supported by this code ( 1 ). The code will refuse to access data that is stored in a different major format.

The maximal minor version of the FilesDaf format that is supported by this code ( 1 ). The code will refuse to access data that is stored with the expected major version ( 1 ), but that uses a higher minor version.

FilesDaf(
    path::AbstractString,
    mode::AbstractString = "r";
    [name::Maybe{AbstractString} = nothing,
    packed::Bool = false]
)

Storage in disk files in some directory.

By convention the root directory name carries the .daf suffix (e.g. cells.daf/ ), but this isn't enforced — any directory containing a daf.json is a valid FilesDaf . The matching single-file ZIP form lives under ZipDaf , with the .daf.zip and .dafs.zip#/group path conventions.

When opening an existing data set, if name is not specified, and there exists a "name" scalar property, it is used as the name. Otherwise, the path will be used as the name.

If packed is true , subsequent writes through this handle default to the packed (chunked + compressed) on-disk encoding for properties whose uncompressed size is at or above DAF_PACKED_TARGET_CHUNK_KB . Per-call packed kwargs on set_*! / empty_*! / copy_*! override this default. The default is false (today's flat encoding).

The valid mode values are as follows (the default mode is r ):

Mode	Allow modifications?	Create if does not exist?	Truncate if exists?	Returned type
r	No	No	No	DafReadOnly
r+	Yes	No	No	FilesDaf
w+	Yes	Yes	No	FilesDaf
w	Yes	Yes	Yes	FilesDaf