GitHub

Copies

DataAxesFormats.Copies — Module

Copy data between Daf data sets.

Note

Copying into an in-memory data set does not duplicate the data; instead it just shares a reference to it. This is fast. In contrast, copying into a disk-based data set (e.g. using HDF5 or simple files) will create a duplicate of the data on disk. This is slow. However, both directions will not significantly increase the amount of memory allocated by the application.

DataAxesFormats.Copies.copy_scalar! — Function 
copy_scalar(;
    destination::DafWriter,
    source::DafReader,
    name::AbstractString,
    [rename::Maybe{AbstractString} = nothing,
    type::Maybe{Type{<:StorageScalarBase}} = nothing,
    default::Union{StorageScalar, Nothing, UndefInitializer} = undef,
    overwrite::Bool = false,
    insist::Bool = true]
)::Nothing

Copy a scalar with some name from some source DafReader into some destination DafWriter .

The scalar is fetched using the name and the default . If rename is specified, store the scalar using this new name. If type is specified, the data is converted to this type. If the scalar already exists in the target, if overwrite , it will be replaced; otherwise, if not insist , skip the copy; otherwise, fail.

DataAxesFormats.Copies.copy_axis! — Function 
copy_axis(;
    destination::DafWriter,
    source::DafReader,
    axis::AbstractString,
    [rename::Maybe{AbstractString} = nothing,
    default::Union{Nothing, UndefInitializer} = undef,
    overwrite::Bool = false,
    insist::Bool = true]
)::Nothing

Copy an axis from some source DafReader into some destination DafWriter .

The axis is fetched using the name and the default . If rename is specified, store the axis using this name.

If the axis already exists in the target, if overwrite , it will be replaced (erasing all data for that axis); otherwise, if not insist , skip the copy; otherwise, fail.

DataAxesFormats.Copies.copy_vector! — Function 
copy_vector!(;
    destination::DafWriter,
    source::DafReader,
    axis::AbstractString,
    name::AbstractString,
    [reaxis::Maybe{AbstractString} = nothing,
    rename::Maybe{AbstractString} = nothing,
    type::Maybe{Type{<:StorageScalarBase}} = nothing,
    default::Union{StorageScalar, StorageVector, Nothing, UndefInitializer} = undef,
    empty::Maybe{StorageScalar} = nothing,
    bestify::Bool = false,
    min_sparse_saving_fraction::AbstractFloat = ```0.25```,
    overwrite::Bool = false,
    insist::Bool = true,
    packed::Maybe{Bool} = nothing]
)::Nothing

Copy a vector from some source DafReader into some destination DafWriter .

The vector is fetched using the axis , name and the default . If reaxis is specified, store the vector using this axis. If rename is specified, store the vector using this name. If type is specified, the data is converted to this type. If the vector already exists in the target, if overwrite , it will be replaced; otherwise, if not insist , skip the copy; otherwise, fail.

If bestify is set, then bestify the data before writing it, using min_sparse_saving_fraction .

This requires the axis of one data set is the same, or is a superset of, or a subset of, the other. If the target axis contains entries that do not exist in the source, then empty must be specified to fill the missing values. If the source axis contains entries that do not exist in the target, they are discarded (not copied).

DataAxesFormats.Copies.copy_matrix! — Function 
copy_matrix(;
    destination::DafWriter,
    source::DafReader,
    rows_axis::AbstractString,
    columns_axis::AbstractString,
    name::AbstractString,
    [rows_reaxis::Maybe{AbstractString} = nothing,
    columns_reaxis::Maybe{AbstractString} = nothing,
    rename::Maybe{AbstractString} = nothing,
    eltype::Maybe{Type{<:StorageScalarBase}} = nothing,
    default::Union{StorageScalar, StorageVector, Nothing, UndefInitializer} = undef,
    empty::Maybe{StorageScalar} = nothing,
    bestify::Bool = false,
    min_sparse_saving_fraction::AbstractFloat = ```0.25```,
    relayout::Bool = true,
    overwrite::Bool = false,
    insist::Bool = true,
    packed::Maybe{Bool} = nothing]
)::Nothing

Copy a matrix from some source DafReader into some destination DafWriter .

The matrix is fetched using the rows_axis , columns_axis , name , relayout and the default . If rows_reaxis and/or columns_reaxis are specified, store the vector using these axes. If rename is specified, store the matrix using this name. If eltype is specified, the data is converted to this type. If the matrix already exists in the target, if overwrite , it will be replaced; otherwise, if not insist , skip the copy; otherwise, fail.

If bestify is set, then bestify the data before writing it, using min_sparse_saving_fraction .

This requires each axis of one data set is the same, or is a superset of, or a subset of, the other. If a target axis contains entries that do not exist in the source, then empty must be specified to fill the missing values. If a source axis contains entries that do not exist in the target, they are discarded (not copied).

Note

When copying a matrix from a subset to a superset, if the empty value is zero, then we create a sparse matrix in the destination. However, currently we create a temporary dense matrix for this; this is inefficient and should be replaced by a more efficient method.

DataAxesFormats.Copies.copy_tensor! — Function 
copy_tensor(;
    destination::DafWriter,
    source::DafReader,
    main_axis::AbstractString,
    rows_axis::AbstractString,
    columns_axis::AbstractString,
    name::AbstractString,
    [rows_reaxis::Maybe{AbstractString} = nothing,
    columns_reaxis::Maybe{AbstractString} = nothing,
    rename::Maybe{AbstractString} = nothing,
    eltype::Maybe{Type{<:StorageScalarBase}} = nothing,
    empty::Maybe{StorageScalar} = nothing,
    bestify::Bool = false,
    min_sparse_saving_fraction::AbstractFloat = ```0.25```,
    relayout::Bool = true,
    overwrite::Bool = false,
    insist::Bool = true,
    packed::Maybe{Bool} = nothing]
)::Nothing

Copy a tensor from some source DafReader into some destination DafWriter .

If bestify is set, then bestify the data before writing it, using min_sparse_saving_fraction .

This is basically a loop that calls copy_matrix! for each of the tensor matrices, based on the entries of the main_axis in the destination . This will create an matrix full of the empty value for any entries of the main axis which exist in the destination but do not exist in the source. If a tensor matrix already exists in the target, if overwrite , it will be replaced; otherwise, if not insist , skip the copy; otherwise, fail.

DataAxesFormats.Copies.copy_all! — Function 
copy_all!(;
    destination::DafWriter,
    source::DafReader
    [empty::Maybe{EmptyData} = nothing,
    types::Maybe{DataTypes} = nothing,
    overwrite::Bool = false,
    insist::Bool = true,
    relayout::Bool = true,
    packed::Maybe{Bool} = nothing]
)::Nothing

Copy all the content of a source DafReader into a destination DafWriter . If some data already exists in the target, if overwrite , it will be replaced; otherwise, if not insist , skip the copy; otherwise, fail.

This will create target axes that exist in only in the source, but will not overwrite existing target axes, regardless of the value of overwrite . An axis that exists in the target must be identical to, or be a subset of, the same axis in the source.

If the source has axes which are a subset of the same axes in the target, then you must specify a dictionary of values for the empty entries that will be created in the target when copying any vector and/or matrix properties. This is specified using a (axis, property) => value entry for specifying an empty value for a vector property and a (rows_axis, columns_axis, property) => entry for specifying an empty value for a matrix property. The order of the axes for matrix properties doesn't matter (the same empty value is automatically used for both axes orders).

If types are specified, the copied data of the matching property is converted to the specified data type.

If a TensorKey is specified, this will create an matrix full of the empty value for any entries of the main axis which exist in the destination but do not exist in the source.

DataAxesFormats.Copies.EmptyData — Type

Specify the data to use for missing properties in a Daf data set. This is specified using a DataKey for which property we specify a value for, and the value to use. This can be specified as a dictionary, a vector of pairs, or a named tuple.

Note

A TensorKey is interpreted as if it as the set of MatrixKey s that are included in the tensor. These are expanded in an internal copy of the dictionary and will override any other specified MatrixKey .

DataAxesFormats.Copies.DataTypes — Type

Specify the data type to use for overriding properties types in a Daf data set. This is specified using a DataKey for which property we specify a type for, and the data type to use. This can be specified as a dictionary, a vector of pairs, or a named tuple.

Note

A TensorKey is interpreted as if it as the set of MatrixKey s that are included in the tensor. These are expanded in an internal copy of the dictionary and will override any other specified MatrixKey .

Picking the right format for the workload

Packed and unpacked-on-local-SSD layouts optimise for different bottlenecks; which one wins for a given workload depends on where the data actually lives and the storage tier's bandwidth relative to a single core's decompression speed. The single rule: pick the format that matches where the data actually lives. Packed for slow tiers, unpacked on fast local SSD if you can stage it. Neither side is the default.

When packed wins for compute

Packed format isn't only for transport and archival — for compute against data living on a slow tier, packed often beats the equivalent unpacked layout because the bandwidth saving outweighs the decompression CPU and the chunk-cache lock overhead:

  • Data on an NFS mount. NFS mmap is page-by-page; random scattered scalar access can produce one round-trip per page, and aggregate bandwidth from a network file system is typically tens of MB/s. Packed format compresses the wire transfer 2–10× and serves chunks via DiskArrays.cache , so a per-block iteration touches only the chunks it needs. Net throughput is usually higher than reading the equivalent unpacked layout.
  • Data accessed via HTTP ( HttpDaf , ZarrDaf over HTTP). Same story more so: round-trip latency is the dominant cost; striped / chunked access amortises it across the data the user actually touches, and compression cuts wire bytes. Reading an unpacked HTTP-served property doesn't even avoid the chunk-cache machinery — the unpacked HTTP striped path also wraps through DiskArrays.cache — so the only thing un-packing the source buys you is wire bandwidth, which is precisely the thing you don't want to give up here.
  • Data on a slow local disk (mechanical HDD, slow USB storage, some cloud-attached block volumes). Disk bandwidth is on the same order as a single core's decompression speed (~150 MB/s at the low end), so the trade is roughly even on raw bytes — but packed reads fewer bytes, so the disk-bound side wins.

When staging to local SSD wins

For data that's already on (or can fit on) a fast local SSD, an unpacked staged copy is the throughput-optimal choice for compute-intensive work:

  • mmap-Strided fast path: zero per-access overhead, full @turbo / LoopVectorization / BLAS support.
  • No DiskArrays.cache SpinLock in the access path; on many-core boxes (32+ threads) the cache-line bouncing on the lock atomic is meaningful, and the unpacked path avoids it entirely.
  • NVMe SSD bandwidth (~3–7 GB/s for modern drives) exceeds zstd decompression speed (~500 MB/s/core), so even when bytes-on-disk are similar, the unpacked side reads them faster.

Staging idiom

Open the source packed, copy once to a local unpacked daf via copy_all! , then run compute against the staged copy: 

using DataAxesFormats

remote = open_daf("https://example.com/dataset")           # packed remote source
local_dir = tempname() * ".daf"
staged = open_daf(local_dir, "w+"; packed = false)
copy_all!(; destination = staged, source = remote)

# ... compute against `staged` here, with full mmap-Strided fast path,
# no SpinLock contention, full @turbo / BLAS support ...

The same pattern applies for compressed local archives you want to run heavy compute against: open packed, copy_all! to a fresh unpacked daf on local SSD, compute against that. copy_all! resolves its packed kwarg against the destination's per-daf default and routes every property through the same writer the user would call directly — so the staged copy ends up byte-identical to one written from scratch.

Index