Reconstruction

Reconstruct implicit axes. Due to AnnData two-axes limitations, other axes are often represented by storing their expanded data (e.g., a type for each cell, and a color for each cell, where the color is actually per type). When converting such data to Daf , it is useful to reconstruct such axes (e.g., create a type axis, assign a color for each type, and delete the per-cell color property).

reconstruct_axis!(
    daf::DafWriter;
    existing_axis::AbstractString,
    implicit_axis::AbstractString,
    [rename_axis::Maybe{AbstractString} = nothing,
    implicit_properties::Maybe{AbstractSet{<:AbstractString}} = nothing,
    skipped_properties::Maybe{AbstractSet{<:AbstractString}} = nothing,
    properties_defaults::Maybe{AbstractDict} = nothing]
)::AbstractDict{<:AbstractString, Maybe{StorageScalar}}

Given an existing_axis in daf , which has a property implicit_axis , create a new axis with the same name as the property (or, if specified, call it rename_axis ). An empty string means there is no value associated with that existing_axis entry; data spelling that some other way - NA , Outliers , a sentinel number - should be passed through unify_empty_vector_values! first, which is where that concept lives. For each of the implicit_properties , we collect the mapping between the implicit_axis and the property values, and store it as a property of the newly created axis.

exist as names of entries in the implicit_axis . This allows manually creating the implicit_axis with additional entries that are not currently in use.

If implicit_properties are explicitly specified, then we require the mapping from implicit_axis to be consistent for them. Otherwise, we look at all the properties of the existing_axis , and check for each one whether the mapping is consistent; if it is, we migrate the property to the new axis. For example, when importing AnnData containing per-cell data, it isn't always clear which property is actually per-batch (e.g., cell age) and which is actually per cell (e.g., doublet score). Not specifying the implicit_properties allows the function to figure it out on its own. If skipped_properties are specified, they are skipped, then these properties are skipped even if they happen (accidentally) to have a consistent mapping with the type.

If the implicit_axis already exists, we verify that all the values provided for it by the existing_axis do, in fact,

If the reconstructed implicit_axis axis already exists, it may contain values that don't exist in the property of the existing_axis . In this case, for each reconstructed property, you should specify an entry in the properties_defaults to use for these values.

connect_axes!(
    daf::DafWriter;
    base_axis::AbstractString,
    from_axis::AbstractString,
    [from_property::Maybe{AbstractString} = nothing,]
    to_axis::AbstractString,
    [to_property::Maybe{AbstractString} = nothing,
    connect_property::Maybe{AbstractString} = nothing,
    overwrite::Bool = false]
)::Nothing

Given a base_axis with two vector properties, one holding a reference to from_axis and one to to_axis , create a property of from_axis that references to_axis . This is only possible if every entry of from_axis is always associated with a single entry of to_axis .

This can happen when one axis (say, "batch") references two other axes (say, "plate" and "tray"). If every batch was placed in one plate and every plate was in a tray, then we'd have batch refers to plate, plate refers to run; reconstruct_axis! would have been enough to deal with it, and batch simply wouldn't have a "tray" property. This is the more common and more sensible case.

However, if for some reason some batches do have a tray reference, but (for whatever reason) do not have a plate reference, we still want to record that "each plate is in a tray", while not giving up on "each batch is in a tray". So we must duplicate data. We record for each plate which tray it is in using connect_axes! - creating a new "tray" property for the plate axis - while keeping the original tray property per batch.

This is in contrast to reconstruct_axis! which does not duplicate data - it moves the data to its proper place, removing the original which became redundant.

By default the properties of base_axis holding the references are named after the axes they refer to, and the created connect_property of from_axis is named after to_axis . Specify from_property , to_property and connect_property when they are not; a base axis may refer to the same axis twice (a "sorted by" and a "sequenced by" run, say), in which case the name of the property is the only thing telling them apart.

An entry of base_axis with no from_axis reference is skipped, since there is nothing to record it against; its to_axis reference is therefore not examined at all. An entry of from_axis which no entry of base_axis refers to is given an empty value.

unify_empty_vector_values!(
    daf::DafWriter;
    axis::AbstractString,
    property::AbstractString,
    empty_values::EmptyImplicit,
    [dtype::Maybe{Type{<:StorageScalarBase}} = nothing,
    empty_value::Maybe{StorageScalar} = nothing]
)::Nothing

Replace every one of the empty_values of a property of an axis with a single empty_value , so that "there is no value here" is spelled one way, converting the property to a dtype on the way if one is given.

Data arrives spelling it several ways, often several ways in the same property: an empty string in some entries and NA in others, (Missing) elsewhere, and for numbers a sentinel such as the smallest integer, which is not obviously a sentinel at all - it is a number, so a mean or a plot of that property is silently wrong rather than visibly absent.

This matters before reconstruct_axis! and connect_axes! , which decide what to do with an entry by asking whether its value is empty. A property still saying NA would have NA reconstructed into an entry of the new axis, sitting among the real ones.

Numbers often arrive as text for exactly this reason - a column of measurements is a column of strings because a few of its entries say NA . Giving a dtype converts the values which are not empty, which is an error unless all of them are values of that type; the ones which are empty become the empty_value , which is why this is one operation and not two.

By default the empty_value is the empty string for strings, NaN for floats, and 0 for unsigned integers, which is the same convention Daf uses for module indices - they are 1-based, so 0 is free to mean "none". A signed integer or a Boolean has no such value, so one must be given, or a dtype which has one.

A property none of whose values is empty is left as it is, rather than being an error: which markers a property carries is a fact about the file, and the same cleanup has to keep working on a file which happens to be clean. What is an error is asking for nothing at all - no empty_values and no dtype - since that cannot do anything whatever the data says, and so is a mistake in the call rather than a fact about the file.

The result is bestify d, so a property which turns out to be mostly empty is stored sparsely rather than densely.

Map property names to a default value. This can be specified as a dictionary, a vector of pairs, or a named tuple.

The value(s) of a property which mean "there is no value". This can be specified as a single value, or as a vector, set or tuple of them, for data which spells "no value" in more than one way (e.g., both Outliers and Doublet ).