Lazy Sparse

Lazy slice-then-materialize wrappers around sparse properties whose components are chunked sources.

A LazySparseMatrix holds the colptr of a sparse matrix in memory plus the rowval and nzval as lazy sources — packed on disk ( Zarr.ZArray , HDF5.Dataset , FilesDaf shard) or chunked over HTTP ( ChunkedArray ) — so a packed sparse property opened from disk does not pay the decompression cost up front. A LazySparseVector is the 1-D counterpart, holding nzind and nzval as the same kind of lazy sources. Slicing either wrapper accumulates SparseSelection s without touching the chunked storage; materialisation runs only when downstream code asks for concrete rowval / nzval / nzind data, and only on the selected slice.

mutable struct LazySparseMatrix{Tv, Ti <: Integer} <: AbstractSparseMatrixCSC{Tv, Ti}
    full_n_rows::Int
    full_n_columns::Int
    full_colptr::Vector{Ti}
    rowval_source::AbstractVector{Ti}
    nzval_source::AbstractVector{Tv}
    row_select::SparseSelection
    column_select::SparseSelection
    materialized::Maybe{SparseMatrixCSC{Tv, Ti}}
end

Lazy AbstractSparseMatrixCSC{Tv, Ti} over packed rowval / nzval sources.

Holds the original colptr in memory (small — sizeof(Ti) × (full_n_columns + 1) bytes) and the rowval / nzval as one-dimensional indexable sources (typically Zarr.ZArray{T, 1} or a DiskArrays.cache -wrapped ZArray , with Vector{T} accepted for flat mmap'd components in mixed flat/packed properties) that decompress per-chunk on access. row_select and column_select describe the slice of the original axes exposed through this wrapper. Materialisation copies the selected slice into a plain SparseMatrixCSC the first time it is required and caches it in materialized for subsequent use.

The full_n_rows / full_n_columns / full_colptr fields describe the original (unsliced) matrix. The corresponding AbstractSparseMatrixCSC accessors ( size , SparseArrays.getcolptr ) report the sliced shape — getcolptr triggers materialisation so it can return the per-slice colptr. Use the full_* fields when internal Daf code needs the original layout without materialising.

Materialisation triggers

The following operations materialise the current slice into the cached materialized matrix on first call (and reuse the cache on subsequent calls):

• SparseArrays.rowvals / SparseArrays.nonzeros / SparseArrays.getcolptr / SparseArrays.nnz .
• SparseMatrixCSC(lazy) and convert(SparseMatrixCSC{Tv, Ti}, lazy) .
• Any generic AbstractSparseMatrixCSC / AbstractMatrix algorithm that calls those primitives (e.g. Matrix(lazy) , multiplication, the LinearAlgebra operations).

The following operations do not materialise:

• Base.size , Base.eltype , Base.length .
• The four slicing forms lazy[:, range] / lazy[range, :] / lazy[:, indices] / lazy[mask, :] (each rebinds selections and clears the cache, returning a fresh wrapper).
• Base.getindex(lazy, ::Int, ::Int) — the only access form that decompresses on the fly without populating the cache: each call reads the column's rowval chunk and one element of nzval . Repeated calls on the same column are amortised by the DiskArrays.cache wrapper that the read paths place over the packed sources.

User code obtains a LazySparseMatrix only as the result of reading a sparse property whose rowval / nzval components are lazy sources — packed on disk, or chunked over HTTP — through DataAxesFormats.Formats.format_get_matrix ; construction lives behind the read paths and is not part of the public API.

mutable struct LazySparseVector{Tv, Ti <: Integer} <: AbstractSparseVector{Tv, Ti}
    full_n_elements::Int
    nzind_source::AbstractVector{Ti}
    nzval_source::AbstractVector{Tv}
    select::SparseSelection
    materialized::Maybe{SparseVector{Tv, Ti}}
end

Lazy AbstractSparseVector{Tv, Ti} over packed nzind / nzval sources.

nzind_source and nzval_source are one-dimensional indexable sources (typically Zarr.ZArray{T, 1} or a DiskArrays.cache -wrapped ZArray for on-disk packed storage, or an ChunkedArray for HTTP-served packed storage) that decompress per-chunk on access. select describes the slice of the original axis exposed through this wrapper. Materialisation copies the selected slice into a plain SparseVector the first time it is required and caches it in materialized for subsequent use.

The full_n_elements field is the original (unsliced) axis length. The corresponding AbstractSparseVector accessors ( size , SparseArrays.nonzeroinds ) report the sliced shape; nonzeroinds triggers materialisation so it can return the per-slice indices. Use full_n_elements when internal Daf code needs the original layout without materialising.

Materialisation triggers

The following operations materialise the current slice into the cached materialized vector on first call (and reuse the cache on subsequent calls):

• SparseArrays.nonzeroinds / SparseArrays.nonzeros / SparseArrays.nnz .
• SparseVector(lazy) and convert(SparseVector{Tv, Ti}, lazy) .
• Any generic AbstractSparseVector / AbstractVector algorithm that calls those primitives.

The following operations do not materialise the slice into vector.materialized :

• Base.size , Base.eltype , Base.length .
• The slicing forms lazy[range] / lazy[indices] / lazy[mask] (each rebinds the selection and clears the cache, returning a fresh wrapper).
• Base.getindex(lazy, ::Int) — binary-searches nzind_source ( O(log nnz) chunk reads) and reads one element of nzval_source . The underlying chunked source's per-chunk LRU (set up by the read paths) amortises repeated accesses, so subsequent scalar lookups within the same chunk are cheap.

User code obtains a LazySparseVector only as the result of reading a sparse property whose nzind / nzval components are lazy sources — packed on disk, or chunked over HTTP — through DataAxesFormats.Formats.format_get_vector ; construction lives behind the read paths and is not part of the public API.

abstract type SparseSelection end

The kind of slice exposed through a LazySparseMatrix along one of the original sparse matrix axes. Concrete subtypes:

• AllOf — every entry of the original axis.
• RangeOf — a contiguous range of original indices.
• IndicesOf — an explicit ordered vector of original indices.
• MaskOf — entries selected by a boolean mask.

Each subtype implements Base.length .

struct AllOf <: SparseSelection
    n_entries::Int
end

A SparseSelection that exposes every entry of the original axis (the axis has n_entries entries).

struct RangeOf <: SparseSelection
    range::UnitRange{Int}
end

A SparseSelection that exposes a contiguous range of entries from the original axis.

struct IndicesOf <: SparseSelection
    indices::Vector{Int}
end

A SparseSelection that exposes an explicit ordered vector of original-axis indices.

struct MaskOf <: SparseSelection
    mask::BitVector
end

A SparseSelection that exposes the entries selected by a boolean mask over the original axis.