Skip to content

Support reading skims in Parquet format #970

Closed

Description

@amsamimi

Description:
Currently, skims are read from OMX files, but based on our experience, storing skims in Parquet format can reduce file size by ~30%. This would significantly reduce storage requirements and improve efficiency.
We already support reading Parquet instead of CSV for the population synthesizer inputs, which has proven very valuable for reducing the size of model inputs. Extending this capability to skims would provide similar benefits.

Proposed enhancement:
Add support for reading skims directly from Parquet format.
Maintain backward compatibility with existing OMX inputs.

Benefits:
Reduced disk usage.
Faster I/O performance.

Activity

  1. lmz commented on Sep 30, 2025

    @lmz
    Contributor

    Just curious - is there a recommended way to store skims in parquet?
    Also, any notes on how much faster i/o would be would be useful...

  2. AmirSamimi-TfNSW commented on Oct 2, 2025

    @AmirSamimi-TfNSW

    @lmz
    Skims can be stored in Parquet format, similar to how they’re currently handled in OMX. Since our visualizer supports Parquet-formatted skims, we convert them using the same structure. This conversion reduces the file size from 7.23 GB to 4.98 GB.
    We expect ActivitySim to load these files more quickly, though the extent of the performance improvement still needs to be tested. The disk space savings are around 30%, and while faster I/O is anticipated, the actual impact will need to be measured.

  3. jpn-- commented on Nov 4, 2025

    @jpn--
    Member

    There are a few things that the HDF5 format handles transparently which we will need to address to manage skim data in parquet format:

    • Lookups, aka row and column headers: This data is one-dim, a different shape from the 2-dim regular matrix tables. Do we expect the lookups to be stored in exploded format in the parquet? e.g. OAZ = [1,2,3,1,2,3,1,2,3] and DTAZ = [1,1,1,2,2,2,3,3,3]. Or should they be provided in a seperate file?
    • Do we allow sparse arrays or only dense arrays? Implicitly OMX is only dense so it's probably easiest to start with dense only, but planning to eventually allow sparse could be useful.
    • Ordering: are the 2-d arrays to be stored in row major or column major format?

    None of this is super hard to handle but I don't think we can ignore having a parquet spec if we want to do parquet, or it will bite us eventually.

  4. moved this to ToDo in Phase 11on Nov 6, 2025
  5. jpn-- commented on Nov 6, 2025

    @jpn--
    Member

    @amsamimi You noted that you are already experimenting with tools that output skims in parquet format. Are you able to describe what that output format looks like for the tools you are using? In particular, addressing the questions outlined above on lookups/taz labels, sparseness, and ordering?

  6. AmirSamimi-TfNSW commented on Nov 6, 2025

    @AmirSamimi-TfNSW

    I used this tool to convert skims.omx to skims.parquet for use in a visualizer. @asiripanich, could you provide more details on the questions Jeff raised in the comment above?

  7. asiripanich commented on Nov 14, 2025

    @asiripanich
    Contributor

    @amsamimi You noted that you are already experimenting with tools that output skims in parquet format. Are you able to describe what that output format looks like for the tools you are using? In particular, addressing the questions outlined above on lookups/taz labels, sparseness, and ordering?

    Hi @jpn--, the tool that @AmirSamimi-TfNSW mentioned converts an OMX file to a Parquet file with the following format:

    Origin Destination Matrix 1 Value Matrix 2 Value ... Matrix n Value

    This format is dense (it includes all OD pairs) and as you can see in the code below, there are no other specific optimizations.

    with omx.open_file(omx_path) as matrix:
                zones = matrix.shape()[1]
                od_pairs = list(
                    itertools.product(range(1, zones + 1), range(1, zones + 1))
                )
                origins = np.array([t[0] for t in od_pairs], dtype=np.int64)
                destinations = np.array([t[1] for t in od_pairs], dtype=np.int64)
                table_contents = {
                    "origin": origins,
                    "destination": destinations,
                }
                for table_name in matrix.list_matrices():
                    column = extract_matrix(matrix, table_name)
                    table_contents[table_name] = column
    
            pa_table = pa.table(table_contents)
            return pa_table

    The main purpose of the skims parquet file is for a dashboard we developed which relies on lazy loading via Polars Lazy DataFrame.

  8. self-assigned this
    on Dec 4, 2025
  9. moved this from ToDo to Under Review in Phase 11on Aug 13, 2026
  10. moved this from Under Review to Subordinate in Phase 11on Aug 13, 2026
  11. moved this from Subordinate to Done in Phase 11on Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

FeatureNew feature or request

Type

No type

Projects

  • Status
    Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions