Repository navigation
Conversation
Port the performance and portability work from codex/replace-openmatrix-with-h5py (b024fb3) onto main (fc175b2). Retain bounded parallel chunk decoding, dtype conversion into destination arrays, variable-level lazy tasks, direct shared-memory/memmap loading, parallel reloads, cleanup fixes, OMX utilities, and benchmark tools. Require OpenMatrix >=0.4.0 and lock 0.4.0. Open OMX paths through OpenMatrix and use absolute HDF5 group paths to respect its matrix-name indexing. Support caller-owned OpenMatrix and h5py handles, including in-memory files, and retain filename-based compatibility with legacy wrappers. Update the notebook, environments, CI, and benchmark dependencies accordingly. Validation: 128 unit tests passed, 1 skipped; loading-skims notebook passed; all pre-commit hooks passed. Synthetic gzip/shuffle comparison against main and the original branch produced byte-identical arrays, retained the old branch's timings, and reduced peak memory versus main. Original implementation: c816dc7 through b024fb3. Authored by codex/astra-6-medium
|
This PR improves how Sharrow loads OMX matrices into Xarray datasets while adopting OpenMatrix 0.4, which uses h5py instead of PyTables. It preserves the existing default of returning a lazy, Dask-backed dataset and adds explicit loading modes for ordinary NumPy arrays, shared memory, and memory-mapped files. The main performance improvement comes from reducing intermediate arrays. Wherever possible, Sharrow now allocates the destination in its final shape and dtype, then decodes the source data directly into that destination. The PR also coordinates parallel decoding across matrices and allows multiple source files to populate shared storage concurrently. Why loading OMX data can require much more memory than the finished dataset OMX files store two-dimensional matrices. Sharrow often combines several of these into one three-dimensional variable. For example: Loading this variable involves more than reading bytes from disk. Sharrow must decompress the stored matrices, arrange them into the requested dimensions, and sometimes convert their numeric type. On For scale, a single 5,000 × 5,000 matrix contains 25 million values. It occupies about 191 MiB as These numbers describe array storage, not a prediction of total process memory. Actual peak memory also depends on task scheduling, compression, concurrent operations, and library overhead. Decode into the destination dtype Sharrow already has a parallel HDF5 chunk decoder on An HDF5 chunk is a rectangular block of a larger matrix. For supported compression filters, Sharrow reads the compressed chunk, decompresses it, and writes its values into the corresponding region of the output array. Previously, the decoder required the output array to have the same dtype as the source. When a file contained The decoder now distinguishes between:
It decodes each chunk using the source dtype and converts the values as it assigns them into the destination. This removes the need for a complete The existing Some temporary storage remains necessary for compressed and decompressed chunks. The improvement is that conversion can operate chunk by chunk instead of requiring a second full matrix. Arrange memory to match how OMX stores the data A three-dimensional array can have the same logical dimensions while using different physical layouts in memory. For a conventional C-contiguous array shaped as: (origin, destination, time_period)the time-period values for each origin–destination pair sit next to each other. Consequently, selecting one entire period with OMX presents the opposite arrangement: each period is already a separate, contiguous two-dimensional matrix. The new loader uses a layout that stores each complete time-period matrix together. NumPy strides describe how to interpret that storage as the same logical This lets Sharrow fill the final three-dimensional result one source matrix at a time. It also allows h5py’s ordinary direct-read fallback to write into those pages without first creating a complete matrix-sized temporary. The shared-memory metadata now records these strides, so another process attaching to the same buffer reconstructs the correct array layout. There is a tradeoff: the complete three-dimensional array is not necessarily C-contiguous. Downstream code that explicitly requires a C-contiguous array may still need to make a copy. Group lazy work by logical variable The default lazy loader now creates one Dask loading task per logical variable, rather than one per physical time-period matrix followed by a stacking operation. For
This reduces task-graph overhead, reuses file handles within a variable, and avoids a separate stack of fully materialized period arrays inside the loading task. The result is still lazy: constructing the dataset generally reads metadata and builds tasks; the matrix values are loaded when requested. Sources that cannot be reopened, such as some in-memory handles, need an eager snapshot so the returned dataset remains usable after the caller closes the source. Variable-level loading is especially useful when the caller will consume all periods. If only selected periods are needed, grouping everything into one task may read more data than necessary. The PR therefore also supports: task_granularity="matrix"This retains separate tasks for individual matrices and can be a better fit for selective time-period loading. Offer explicit choices for where and when data is loaded
For example, a caller that needs all of the data immediately can use: skims = sharrow.dataset.from_omx_3d(
skim_files,
time_periods=["EA", "AM", "MD", "PM", "EV"],
load="eager",
)The eager loader allocates the output arrays once and fills them directly. It processes source files in the calling process while still using the parallel chunk decoder. The shared and memmap modes first construct a lightweight description of the output shapes, coordinates, and dtypes. They reserve the destination storage without materializing a separate ordinary dataset, then load into that storage. Shared memory avoids requiring each worker to return a large private array for the parent process to copy into place. Workers attach to the destination and write their assigned regions directly. Memmap uses the same approach with file-backed storage. It can reduce pressure on resident memory, but it does not make the dataset smaller: the backing files still need space, and reading and writing them adds I/O. Resident memory also depends on how the operating system caches mapped pages. Coordinate parallelism without allowing temporary buffers to grow unchecked Parallel loading operates at two levels. Within a process, threads decode compressed chunks. Across source files, shared and memmap loading can assign independent file batches to multiple workers. The new This matters when Dask runs several variable-loading tasks concurrently. Without a shared limit, each read could independently queue its own set of chunks, multiplying temporary memory use. Pool reuse also avoids repeatedly creating a separate collection of decoder threads for each matrix. The bound is a chunk-count limit, not a fixed byte budget. Large chunks consume more memory than small chunks, and separate worker processes have their own decoding resources. Increasing worker counts therefore still has a memory cost. On non-Windows platforms, file-level parallel loading uses spawned processes so workers do not inherit open HDF5 state. On Windows, it uses threads to avoid process-startup restrictions in ordinary scripts and notebooks. h5py serializes its HDF5 calls within a process, so Windows file threads do not imply fully parallel HDF5 access; chunk decompression can still run concurrently. The best worker count depends on the workload. More workers can help when decompression or independent file reads are limiting performance, but can add overhead or contend for the same storage device. Make parallel reloads preserve existing behavior
Before starting workers, Sharrow builds assignments connecting each source matrix to its destination variable and time-period page. Duplicate matrix names retain the existing “last file wins” behavior. Resolving that ownership in advance prevents workers from racing to overwrite the same destination region. Reloading also validates destination shapes and dimensions, skips unselected variables, and supports custom dimension names. Ordinary in-process destinations continue to use serial file loading, with parallel chunk decoding available within each read. Use OpenMatrix 0.4 while keeping the fast HDF5 path The PR requires One compatibility detail is important: OpenMatrix treats relative keys such as The loaders accept paths, OpenMatrix handles, plain h5py handles, and legacy wrappers exposing a usable filename. Handles supplied by callers remain caller-owned; Sharrow closes the handles it opens itself. Standard gzip/shuffle files work with the base installation. Additional filter packages are moved into the Improve cleanup and supporting utilities The shared-memory changes also remove an avoidable allocation during optional buffer initialization. The buffer is now zeroed in place instead of constructing a same-sized Python bytes object first. Failure paths release allocated storage, and memmap cleanup flushes and closes mappings before deleting their files. This is particularly important on Windows, where an open mapping can prevent deletion. Memmap creation also refuses to overwrite an existing backing file or metadata file. The OMX utilities receive related improvements:
Python 3.14 support and the general Zarr 3 compatibility work already exist on the current upstream Validation and measured results After rebasing onto upstream A local synthetic benchmark compared the previous
The resulting arrays were byte-identical across the compared implementations. For this workload, the clearest measured benefit was lower peak memory: eager loading used approximately 47% less peak RSS than the previous These measurements demonstrate the behavior of the loading changes on one synthetic workload, not a universal speedup or a full-scale regional-model result. The included benchmark tools support broader comparisons, including full-scale MTC skims and ActivitySim-style shared-memory reloads. |
jpn--
left a comment
There was a problem hiding this comment.
Code review: 10 inline comments. The ones that matter most are the memmap release segfault (shared_memory.py), reload ignoring caller-owned/in-memory handles, and reload_from_omx_3d spawning a process pool by default. The rest are memory, efficiency, and robustness concerns.
🤖 Generated with Claude Code
| return filenames, assignments | ||
|
|
||
|
|
||
| def _is_reopenable(filename) -> bool: |
There was a problem hiding this comment.
Repeated opens. _is_reopenable opens each file only to probe it. Combined with the path recursion in from_omx/from_omx_3d and the fast-path reopen, from_omx(path) opens the same file three times. On network filesystems with many skim files this adds up. Sources given as paths that were just opened successfully could be treated as reopenable without probing.
There was a problem hiding this comment.
Triaged as a non-blocking optimization and deferred for this PR, as agreed. Redundant probes/opens can matter on network storage, but some probing already exists on main and there is no measured open-latency regression here. The correctness fix preserves caller-owned handles during serial reload; broader changes to reopenability probing remain separate follow-up work. Leaving this thread open to make that deferral explicit.
| try: | ||
| offset = dset_id.get_chunk_info(i).chunk_offset | ||
| filter_mask, raw_bytes = dset_id.read_direct_chunk(offset) | ||
| if pending_semaphore is None: |
There was a problem hiding this comment.
Nit: the two executor.submit branches differ only in the release wrapper. A no-op release (or always wrapping) would collapse them into one call. Relatedly, the source-normalization and TypeError message are copy-pasted across from_omx_3d, _load_omx_assignments, and _prepare_omx_reload_assignments, and only one copy closes partially opened handles on error. A shared _open_omx_source() helper would keep them in sync.
There was a problem hiding this comment.
Triaged as cleanup and deferred for this PR, as agreed. The duplicated submit branches are not themselves a correctness problem, and I did not establish an additional handle leak: single-source paths use context managers, while the multi-source path explicitly closes its retained handles on failure. The concrete serial-handle bug is fixed in 4470f82. A shared source-opening helper and submission-branch refactor remain follow-up work; leaving this thread open.
Authored by codex/astra-6-medium
Authored by codex/astra-6-medium
Authored by codex/astra-6-medium
Authored by codex/astra-6-medium
Authored by codex/astra-6-medium
Authored by codex/astra-6-medium
Authored by codex/astra-6-medium
|
Pushed seven review-fix commits through b4a4aa6. Replied to all ten inline threads and resolved the eight addressed concerns. The repeated-open optimization and code-deduplication suggestions remain explicitly deferred and open. Validation on the updated branch: 200 tests passed, one skipped (unit suite plus loading-skims notebook); all pre-commit checks passed. Isolated before/after checks verified values for whole-dataset loading, selected periods, and shared-memory realignment. All seven commit signatures verified. Memmap file retention remains unchanged by agreement: successfully created data and metadata files persist after process exit and require explicit deletion. Release now safely preserves live array views; Windows deletion must wait until all mappings have been released. No automatic file-deletion policy was added. |
Authored by codex/astra-6-medium
OMX loading can allocate large intermediate arrays while decoding, stacking time periods, and converting dtypes. This change retains OpenMatrix as the OMX interface, requires its h5py-backed 0.4 release, and loads matrices through Sharrow's bounded parallel decoder into their destination arrays.
Validation:
Addresses both #90 and #91