Complete Repositories

The common idiom in Daf is to have multiple repositories in a chain; actually, they typically form a tree where different leaf repositories are based on common ancestor repositories. For example, a base cells repository can be used by multiple alternative metacells repositories.

Tracking this tree manually is possible, by using naming conventions (which is always a good idea). However, this gets tedious. The code here automates this by using an additional convention - each repository contains a scalar property called base_daf_repository which identifies the repositories it is immediately based on (if any). Each path is relative to the directory containing the child repository. See open_daf for details.

The property holds only the immediate bases; what each of them in turn is based on is recorded in it, so the shape of the whole is found by following the records. It is one of:

• A path, for the common case of resting on the whole of a single repository.
• A JSON object {"path": ..., "axes": ..., "data": ...} , where the optional axes and data are the parameters of a DafView to apply, so that the child rests on a subset of the base data, and/or on renamed base data.
• A JSON array of either of the above, for a repository resting on several. Later bases override earlier ones, as in any chain.

Since the same base repository can be reached through more than one of these, it is opened once (using the GlobalWeakCache ) and appears once in the chain, at its earliest position - a base must not override what rests on it.

Since the same base repository can be used by multiple other repositories, we use the GlobalWeakCache to avoid needlessly re-opening the same repository more than once.

complete_daf(
    leaf::AbstractString,
    mode::AbstractString = "r";
    [name::Maybe{AbstractString} = nothing,
    packed::Bool = false]
)::Union{DafReader, DafWriter}

Open a complete chain of Daf repositories by tracing back through the base_daf_repository of each. Valid modes are only "r" and "r+"; if the latter, only the leaf repository is opened in write mode.

If packed is true , the leaf repository (the only one opened in write mode under "r+") gets packed = true as its per-daf default. Base repositories are always opened in "r" mode and the packed value is irrelevant for them.

A convenient way to create persistent complete chains is using complete_chain! .

TODO: Properly indent the log messages of the created leaf repositories. Generic mechanism for indenting all hierarchical log messages?

open_daf(
    path::AbstractString,
    mode::AbstractString = "r";
    [name::Maybe{AbstractString} = nothing,
    packed::Bool = false]
)::Union{DafReader, DafWriter}

Open a Daf data set, dispatching to the appropriate backend based on path :

• If path ends with .daf.zarr , ends with .daf.zarr.zip , or contains .dafs.zarr.zip# (followed by a sub-daf group path), open a ZarrDaf .
• Otherwise, if path ends with .daf.zip or contains .dafs.zip# (followed by a sub-daf group path), open a ZipDaf .
• Otherwise, if path starts with http:// or https:// , open an HttpDaf . Only mode = "r" is supported for the HTTP backend; any other mode raises an error.
• Otherwise, if path ends with .h5df or contains .h5dfs# (followed by a group path), open an H5df file (or a group in one).
• Otherwise, open a FilesDaf .

The packed kwarg is forwarded to the chosen backend; see each backend's constructor for what it controls.