Skip to content

Improve OMX loading performance with OpenMatrix 0.4 - #92

Open
jpn-- wants to merge 9 commits into
ActivitySim:mainfrom
driftlesslabs:codex/openmatrix-0.4-performance
Open

jpn-- wants to merge 9 commits into
ActivitySim:mainfrom
driftlesslabs:codex/openmatrix-0.4-performance

Conversation

@jpn--

@jpn-- jpn-- commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

OMX loading can allocate large intermediate arrays while decoding, stacking time periods, and converting dtypes. This change retains OpenMatrix as the OMX interface, requires its h5py-backed 0.4 release, and loads matrices through Sharrow's bounded parallel decoder into their destination arrays.

  • Add eager, shared-memory, and memmap loading, variable-level lazy tasks, and parallel reloads with bounded in-flight chunks.
  • Convert dtypes during decoding, preserve source precedence, and improve shared-memory/memmap cleanup.
  • Support paths, caller-owned OpenMatrix and h5py handles (including in-memory files), and filename-bearing legacy handles. Use absolute HDF5 group paths to accommodate OpenMatrix's matrix-name indexing.
  • Make optional HDF5 filter packages an extra, fix OMX splitting and Zarr conversion, and add reproducible benchmark tools and documentation.
  • Preserve upstream's Python 3.14 support and Zarr 3 compatibility, including format-2 output for OMX conversion.

Validation:

  • 183 unit tests passed, 1 skipped after rebasing onto upstream main.
  • Loading-skims notebook and all pre-commit checks passed.

Addresses both #90 and #91

Port the performance and portability work from
codex/replace-openmatrix-with-h5py (b024fb3) onto main (fc175b2).
Retain bounded parallel chunk decoding, dtype conversion into destination
arrays, variable-level lazy tasks, direct shared-memory/memmap loading,
parallel reloads, cleanup fixes, OMX utilities, and benchmark tools.

Require OpenMatrix >=0.4.0 and lock 0.4.0. Open OMX paths through OpenMatrix
and use absolute HDF5 group paths to respect its matrix-name indexing.
Support caller-owned OpenMatrix and h5py handles, including in-memory files,
and retain filename-based compatibility with legacy wrappers. Update the
notebook, environments, CI, and benchmark dependencies accordingly.

Validation: 128 unit tests passed, 1 skipped; loading-skims notebook passed;
all pre-commit hooks passed. Synthetic gzip/shuffle comparison against main
and the original branch produced byte-identical arrays, retained the old
branch's timings, and reduced peak memory versus main.

Original implementation: c816dc7 through b024fb3.

Authored by codex/astra-6-medium
@jpn--

jpn-- commented Nov 12, 2026 •

Copy link
Copy Markdown
Member Author

This PR improves how Sharrow loads OMX matrices into Xarray datasets while adopting OpenMatrix 0.4, which uses h5py instead of PyTables. It preserves the existing default of returning a lazy, Dask-backed dataset and adds explicit loading modes for ordinary NumPy arrays, shared memory, and memory-mapped files.

The main performance improvement comes from reducing intermediate arrays. Wherever possible, Sharrow now allocates the destination in its final shape and dtype, then decodes the source data directly into that destination. The PR also coordinates parallel decoding across matrices and allows multiple source files to populate shared storage concurrently.

Why loading OMX data can require much more memory than the finished dataset

OMX files store two-dimensional matrices. Sharrow often combines several of these into one three-dimensional variable. For example:

TIME__AM  ─┐
TIME__MD  ─┼─> TIME[origin, destination, time_period]
TIME__PM  ─┘

Loading this variable involves more than reading bytes from disk. Sharrow must decompress the stored matrices, arrange them into the requested dimensions, and sometimes convert their numeric type.

On main, the lazy loader represents each source matrix as a separate Dask array, stacks those arrays along the time-period dimension, and then applies any requested dtype conversion. Dask defers this work until the caller requests the values, but deferring work does not eliminate the allocations needed to execute it. During materialization, source matrices, assembled results, and converted arrays can overlap in memory.

For scale, a single 5,000 × 5,000 matrix contains 25 million values. It occupies about 191 MiB as float64 or 95 MiB as float32. Five time periods require approximately 954 MiB or 477 MiB, respectively. Additional matrix-sized or variable-sized temporary arrays therefore become expensive quickly.

These numbers describe array storage, not a prediction of total process memory. Actual peak memory also depends on task scheduling, compression, concurrent operations, and library overhead.

Decode into the destination dtype

Sharrow already has a parallel HDF5 chunk decoder on main; this PR extends that implementation rather than introducing parallel decoding from scratch.

An HDF5 chunk is a rectangular block of a larger matrix. For supported compression filters, Sharrow reads the compressed chunk, decompresses it, and writes its values into the corresponding region of the output array.

Previously, the decoder required the output array to have the same dtype as the source. When a file contained float64 data but the destination used float32, the reload path read a complete source-dtype matrix and then copied it into the destination, converting values along the way.

The decoder now distinguishes between:

  • The source dtype, which describes the bytes stored in each chunk.
  • The destination dtype, which describes the array Sharrow is filling.

It decodes each chunk using the source dtype and converts the values as it assigns them into the destination. This removes the need for a complete float64 temporary matrix when filling a float32 result.

The existing max_float_precision setting still controls precision reduction. This PR changes when conversion happens; it does not introduce a new default reduction in precision.

Some temporary storage remains necessary for compressed and decompressed chunks. The improvement is that conversion can operate chunk by chunk instead of requiring a second full matrix.

Arrange memory to match how OMX stores the data

A three-dimensional array can have the same logical dimensions while using different physical layouts in memory.

For a conventional C-contiguous array shaped as:

(origin, destination, time_period)

the time-period values for each origin–destination pair sit next to each other. Consequently, selecting one entire period with array[:, :, period] produces a view whose elements are spread across the buffer.

OMX presents the opposite arrangement: each period is already a separate, contiguous two-dimensional matrix.

The new loader uses a layout that stores each complete time-period matrix together. NumPy strides describe how to interpret that storage as the same logical (origin, destination, time_period) array. Callers still index the dimensions in the same order, but each array[:, :, period] page is contiguous.

This lets Sharrow fill the final three-dimensional result one source matrix at a time. It also allows h5py’s ordinary direct-read fallback to write into those pages without first creating a complete matrix-sized temporary.

The shared-memory metadata now records these strides, so another process attaching to the same buffer reconstructs the correct array layout.

There is a tradeoff: the complete three-dimensional array is not necessarily C-contiguous. Downstream code that explicitly requires a C-contiguous array may still need to make a copy.

Group lazy work by logical variable

The default lazy loader now creates one Dask loading task per logical variable, rather than one per physical time-period matrix followed by a stacking operation.

For TIME, that task:

  1. Allocates the three-dimensional destination using the final dtype.
  2. Opens the required source files.
  3. Reads each time-period matrix into its destination page.
  4. Fills missing periods with zeros.
  5. Closes the files and returns the assembled variable.

This reduces task-graph overhead, reuses file handles within a variable, and avoids a separate stack of fully materialized period arrays inside the loading task.

The result is still lazy: constructing the dataset generally reads metadata and builds tasks; the matrix values are loaded when requested. Sources that cannot be reopened, such as some in-memory handles, need an eager snapshot so the returned dataset remains usable after the caller closes the source.

Variable-level loading is especially useful when the caller will consume all periods. If only selected periods are needed, grouping everything into one task may read more data than necessary. The PR therefore also supports:

task_granularity="matrix"

This retains separate tasks for individual matrices and can be a better fit for selective time-period loading.

Offer explicit choices for where and when data is loaded

from_omx_3d now supports four loading modes:

Mode Returned storage Intended use
lazy Dask arrays Defer loading until values are requested; remains the default.
eager Ordinary NumPy arrays Load immediately without Dask task execution or result assembly.
shared Shared-memory arrays Load directly into storage that multiple processes can access.
memmap File-backed arrays Use the operating system’s memory mapping to back the destination with files.

For example, a caller that needs all of the data immediately can use:

skims = sharrow.dataset.from_omx_3d(
    skim_files,
    time_periods=["EA", "AM", "MD", "PM", "EV"],
    load="eager",
)

The eager loader allocates the output arrays once and fills them directly. It processes source files in the calling process while still using the parallel chunk decoder.

The shared and memmap modes first construct a lightweight description of the output shapes, coordinates, and dtypes. They reserve the destination storage without materializing a separate ordinary dataset, then load into that storage.

Shared memory avoids requiring each worker to return a large private array for the parent process to copy into place. Workers attach to the destination and write their assigned regions directly.

Memmap uses the same approach with file-backed storage. It can reduce pressure on resident memory, but it does not make the dataset smaller: the backing files still need space, and reading and writing them adds I/O. Resident memory also depends on how the operating system caches mapped pages.

Coordinate parallelism without allowing temporary buffers to grow unchecked

Parallel loading operates at two levels.

Within a process, threads decode compressed chunks. Across source files, shared and memmap loading can assign independent file batches to multiple workers.

The new _OmxReadCoordinator shares a decoder thread pool and a semaphore across participating reads. The semaphore limits how many chunks can be queued or actively decoded at once. A reader must obtain a slot before reading another raw chunk, and the decoding task releases that slot when it finishes.

This matters when Dask runs several variable-loading tasks concurrently. Without a shared limit, each read could independently queue its own set of chunks, multiplying temporary memory use. Pool reuse also avoids repeatedly creating a separate collection of decoder threads for each matrix.

The bound is a chunk-count limit, not a fixed byte budget. Large chunks consume more memory than small chunks, and separate worker processes have their own decoding resources. Increasing worker counts therefore still has a memory cost.

On non-Windows platforms, file-level parallel loading uses spawned processes so workers do not inherit open HDF5 state. On Windows, it uses threads to avoid process-startup restrictions in ordinary scripts and notebooks. h5py serializes its HDF5 calls within a process, so Windows file threads do not imply fully parallel HDF5 access; chunk decompression can still run concurrently.

The best worker count depends on the workload. More workers can help when decompression or independent file reads are limiting performance, but can add overhead or contend for the same storage device.

Make parallel reloads preserve existing behavior

reload_from_omx_3d can now load multiple files concurrently when the destination is shared-memory-backed and the sources can be reopened.

Before starting workers, Sharrow builds assignments connecting each source matrix to its destination variable and time-period page. Duplicate matrix names retain the existing “last file wins” behavior. Resolving that ownership in advance prevents workers from racing to overwrite the same destination region.

Reloading also validates destination shapes and dimensions, skips unselected variables, and supports custom dimension names. Ordinary in-process destinations continue to use serial file loading, with parallel chunk decoding available within each read.

Use OpenMatrix 0.4 while keeping the fast HDF5 path

The PR requires openmatrix>=0.4.0 and locks version 0.4.0. OpenMatrix now supplies an h5py-backed file object, allowing Sharrow to use its OMX interface while passing the underlying HDF5 datasets to the existing fast decoder.

One compatibility detail is important: OpenMatrix treats relative keys such as file["TIME__AM"] as matrix names. Sharrow therefore uses absolute paths such as file["/data"] and file["/lookup"] when accessing the format’s groups.

The loaders accept paths, OpenMatrix handles, plain h5py handles, and legacy wrappers exposing a usable filename. Handles supplied by callers remain caller-owned; Sharrow closes the handles it opens itself.

Standard gzip/shuffle files work with the base installation. Additional filter packages are moved into the hdf5-plugins extra. When Sharrow cannot decode a dataset through its fast path, it falls back to h5py, which still needs the appropriate installed plugin for nonstandard compression.

Improve cleanup and supporting utilities

The shared-memory changes also remove an avoidable allocation during optional buffer initialization. The buffer is now zeroed in place instead of constructing a same-sized Python bytes object first.

Failure paths release allocated storage, and memmap cleanup flushes and closes mappings before deleting their files. This is particularly important on Windows, where an open mapping can prevent deletion. Memmap creation also refuses to overwrite an existing backing file or metadata file.

The OMX utilities receive related improvements:

  • split_omx no longer requires Larch, preserves dataset compression and metadata, handles output paths correctly, and removes stale contents when rewriting an output.
  • omx_to_zarr uses the common OMX loader to assemble complete variables before writing them. It retains upstream’s explicit Zarr format-2 output.
  • Documentation, examples, dependency environments, and benchmark scripts are updated for the new loading options.

Python 3.14 support and the general Zarr 3 compatibility work already exist on the current upstream main; this PR preserves those changes.

Validation and measured results

After rebasing onto upstream main, validation completed with 183 unit tests passing and one skipped. The loading-skims notebook and all pre-commit checks also passed. Tests cover OpenMatrix and h5py handles, all four loading modes, dtype conversion, shared-memory reloads, cleanup, and OMX utilities.

A local synthetic benchmark compared the previous main and this OpenMatrix-based port. It used six gzip/shuffle-compressed 2,048 × 2,048 float64 matrices, producing two three-period float32 variables. Each case ran in a fresh process, with one warmup and three measured repetitions.

Implementation and mode Median load time Median peak process RSS
Previous main, lazy materialized 0.382 s 610 MiB
This PR, eager 0.364 s 326 MiB

The resulting arrays were byte-identical across the compared implementations. For this workload, the clearest measured benefit was lower peak memory: eager loading used approximately 47% less peak RSS than the previous main lazy path. The timing improvement was modest.

These measurements demonstrate the behavior of the loading changes on one synthetic workload, not a universal speedup or a full-scale regional-model result. The included benchmark tools support broader comparisons, including full-scale MTC skims and ActivitySim-style shared-memory reloads.

@jpn--
jpn-- requested a review from janzill November 12, 2026 18:03

@jpn-- jpn-- left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review: 10 inline comments. The ones that matter most are the memmap release segfault (shared_memory.py), reload ignoring caller-owned/in-memory handles, and reload_from_omx_3d spawning a process pool by default. The rest are memory, efficiency, and robustness concerns.

🤖 Generated with Claude Code

Comment thread sharrow/shared_memory.py Resolved
Comment thread sharrow/dataset.py Resolved
Comment thread sharrow/dataset.py Resolved
Comment thread sharrow/dataset.py Resolved
Comment thread sharrow/dataset.py Resolved
Comment thread sharrow/translate.py Resolved
Comment thread sharrow/dataset.py Resolved
Comment thread sharrow/dataset.py Resolved
Comment thread sharrow/dataset.py
return filenames, assignments


def _is_reopenable(filename) -> bool:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repeated opens. _is_reopenable opens each file only to probe it. Combined with the path recursion in from_omx/from_omx_3d and the fast-path reopen, from_omx(path) opens the same file three times. On network filesystems with many skim files this adds up. Sources given as paths that were just opened successfully could be treated as reopenable without probing.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Triaged as a non-blocking optimization and deferred for this PR, as agreed. Redundant probes/opens can matter on network storage, but some probing already exists on main and there is no measured open-latency regression here. The correctness fix preserves caller-owned handles during serial reload; broader changes to reopenability probing remain separate follow-up work. Leaving this thread open to make that deferral explicit.

Comment thread sharrow/omx_reader.py
try:
offset = dset_id.get_chunk_info(i).chunk_offset
filter_mask, raw_bytes = dset_id.read_direct_chunk(offset)
if pending_semaphore is None:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the two executor.submit branches differ only in the release wrapper. A no-op release (or always wrapping) would collapse them into one call. Relatedly, the source-normalization and TypeError message are copy-pasted across from_omx_3d, _load_omx_assignments, and _prepare_omx_reload_assignments, and only one copy closes partially opened handles on error. A shared _open_omx_source() helper would keep them in sync.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Triaged as cleanup and deferred for this PR, as agreed. The duplicated submit branches are not themselves a correctness problem, and I did not establish an additional handle leak: single-source paths use context managers, while the multi-source path explicitly closes its retained handles on failure. The concrete serial-handle bug is fixed in 4470f82. A shared source-opening helper and submission-branch refactor remain follow-up work; leaving this thread open.

@jpn--

jpn-- commented Nov 12, 2026

Copy link
Copy Markdown
Member Author

Pushed seven review-fix commits through b4a4aa6. Replied to all ten inline threads and resolved the eight addressed concerns. The repeated-open optimization and code-deduplication suggestions remain explicitly deferred and open.

Validation on the updated branch: 200 tests passed, one skipped (unit suite plus loading-skims notebook); all pre-commit checks passed. Isolated before/after checks verified values for whole-dataset loading, selected periods, and shared-memory realignment. All seven commit signatures verified.

Memmap file retention remains unchanged by agreement: successfully created data and metadata files persist after process exit and require explicit deletion. Release now safely preserves live array views; Windows deletion must wait until all mappings have been released. No automatic file-deletion policy was added.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants