Skip to content

Reduce memory of summarize step by dropping unused variables before wide merge - #1125

Open
i-am-sijia wants to merge 3 commits into
ActivitySim:mainfrom
wsp-sag:reduce-summarize-mem
Open

i-am-sijia wants to merge 3 commits into
ActivitySim:mainfrom
wsp-sag:reduce-summarize-mem

Conversation

@i-am-sijia

Copy link
Copy Markdown
Member

This PR addresses #1124.

This pull request introduces a new feature to the summarize model that trims unused columns from tables before merging, improving memory usage and performance. It also adds a corresponding test to ensure this behavior works as intended. The most important changes are grouped below:

Core feature: Trimming unused columns

  • Added a new function trim_unused_columns to summarize.py that drops columns from input tables that are not referenced in the summarize specification or related preprocessor specs, with logic to retain necessary columns for merging and aggregation.
  • Introduced a new settings flag DROP_UNUSED_COLUMNS (default True) in SummarizeSettings to control whether unused columns are dropped, with logic to skip trimming if EXPORT_PIPELINE_TABLES is True.
  • Integrated trim_unused_columns into the main summarize function so that tables are trimmed before merging, based on the new setting.

Refactoring and utility

  • Extracted logic for deleting temporary variables into a new helper function delete_temporary_variables for clarity and reuse.

Testing

  • Added a new test test_summarize_drop_unused_columns to verify that columns are correctly trimmed when the feature is enabled, and that the output remains correct and consistent.

@jpn--

jpn-- commented Nov 5, 2026

Copy link
Copy Markdown
Member

abench results

Auto-generated by abench 0.2.0.

PR: #1125 · Baseline: base

Run Result Elapsed (s) Change Peak (GiB) Change
base valid 1203.39 +0.0% 28.434 +0.0%
pr_off valid 1204.72 +0.1% 28.389 -0.2%
pr_on valid 1245.52 — 28.262 —

Negative changes mean lower elapsed time or memory. Changes are omitted for invalid runs or differing comparison settings. Single runs do not establish statistical significance.

Configuration and source commits

base — base
Households: 500000; multiprocess: False; processes: 1; Sharrow: True; profile: MTC prototype; platform: None.

  • activitysim: ActivitySim/activitysim at e44be4dc232e02cd0e95a9042e4cfa6278b53c88
  • sharrow: driftlesslabs/sharrow at b024fb343c9c391263c77ee086fa610758702b9f

pr_off — pr_off
Households: 500000; multiprocess: False; processes: 1; Sharrow: True; profile: MTC prototype; platform: None.

  • activitysim: ActivitySim/activitysim at 1752e2f6d77bc6380a33bdbbb03e8997ea4fc1d2
  • sharrow: driftlesslabs/sharrow at b024fb343c9c391263c77ee086fa610758702b9f

pr_on — pr_on
Households: 500000; multiprocess: False; processes: 1; Sharrow: True; profile: MTC prototype; platform: None.

  • activitysim: ActivitySim/activitysim at 1752e2f6d77bc6380a33bdbbb03e8997ea4fc1d2
  • sharrow: driftlesslabs/sharrow at b024fb343c9c391263c77ee086fa610758702b9f

Memory traces

Memory traces

Blue: total cgroup memory; green dashed: anonymous + shared memory when available. Panels share axes; invalid runs are labeled.

Component runtimes

Component runtimes

Valid runs only. Mean ± population standard deviation across worker executions, not confidence intervals. Parallel component times must not be summed as wall time.

@jpn-- jpn-- left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

abench results posted above show the fix is effective at reducing memory usage in the summarize step, at least in the MTC model. I see no reason why these results wouldn't generalize to other models. Overall runtime shows some variation that is purely exogenous; the increased runtime on the run with the feature enabled is entirely on components that this PR does not touch, and is possibly due to thermal throttling due to the relevant test running last.

I found 3 problems in the code that should be addressed. None are surfaced by existing tests. In addition to fixes, new tests should be included that cover all three problems.

expressions_df = spec[["Expression"]].dropna().astype(str)
if model_settings.preprocessor is not None:
preprocessor_spec = pd.read_csv(
state.filesystem.get_config_file_path(model_settings.preprocessor.SPEC),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The existing preprocessor loader explicitly supports omitting the filename extension. This is valid:

EXPORT_PIPELINE_TABLES: false

preprocessor:
  DF: trips_merged
  SPEC: summarize_preprocessor

The actual file on disk is:

summarize_preprocessor.csv

Before the PR, compute_columns() normalizes the name:

if not expressions_spec_name.endswith(".csv"):
    expressions_spec_name = f"{expressions_spec_name}.csv"

It therefore finds and evaluates the CSV successfully.

The new trimming helper needs to inspect the preprocessor expressions before merging. But it resolves the raw SPEC value directly:

state.filesystem.get_config_file_path(model_settings.preprocessor.SPEC)

For the example above, it searches for a file literally named summarize_preprocessor, without an extension. That fails with FileNotFoundError before the normal preprocessor loader gets a chance to add .csv.

The same configuration works when trimming is disabled, or when pipeline-table export bypasses trimming. That makes this a configuration compatibility regression rather than a missing input file.

The immediate fix is to apply the same extension normalization before resolving the file. More robustly, the dependency scanner should share the existing preprocessor specification-loading logic so the two paths interpret settings consistently. A regression test should use an extensionless SPEC, provide only the .csv file, and compare trimming-enabled and trimming-disabled results.

For existing users, DROP_UNUSED_COLUMNS: false avoids all three regressions while preserving the previous behavior. These cases all use explicit, statically identifiable column names or an already-supported filename convention; they don’t depend on dynamically constructed column references.

keep |= {"origin", "destination"}
keep |= {i.removesuffix(s) for i in identifiers for s in ("_trip", "_tour")}

for table_name in ("persons_merged", "trips_merged", "tours_merged", "land_use"):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The existing summarize step exposes several tables in its expression namespace, including:

trips
trips_merged
tours
tours_merged

It also processes BIN and AGGREGATE settings for those names. In particular, tours is an alias for tours_merged.

However, the new dependency collector only examines these settings sections:

("persons_merged", "trips_merged", "tours_merged", "land_use")

It omits trips and tours, even though those are the tables affected by trimming.
For example:

EXPORT_PIPELINE_TABLES: false

trips:
  AGGREGATE:
    - column: trip_mode
      label: broad_mode
      map:
        WALK: active
        BIKE: active
        DRIVEALONEFREE: auto

With a summarize expression:

trips.groupby("broad_mode").size().rename("trips").reset_index()

The intended sequence is straightforward: read trip_mode, create broad_mode, then count trips by that grouping.

The trimmer sees broad_mode in the expression, but it never inspects the trips settings section to discover its dependency on trip_mode. If no other expression requires trip_mode, it drops that column. The existing aggregation code then attempts:

df["trip_mode"].map(...)

and raises:

KeyError: 'trip_mode'

The same problem occurs with a tours: section that bins or aggregates a column used nowhere else.

This is separate from the first bug: no merge suffixes are involved. Even a plain source-column name is lost because its entire settings section is skipped.

The fix is to include trips and tours when collecting YAML dependencies—or, more generally, derive the settings names from the same table namespace that the classification loop uses. Tests should cover both aliases with a source column referenced only in their YAML settings.

for table_name in ("persons_merged", "trips_merged", "tours_merged", "land_use"):
meta = getattr(model_settings, table_name, None) or {}
for slicer in (meta.get("AGGREGATE") or []) + (meta.get("BIN") or []):
keep.add(slicer["column"])

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The merge combines trips and tours_merged. When both tables contain a column with the same name, pandas adds _trip and _tour suffixes.

For example, suppose the input tables contain:

Input table Column Value
trips duration 15
tours_merged duration 90

After merging, those become duration_trip = 15 and duration_tour = 90.

A valid configuration could classify trip durations before summarizing:

EXPORT_PIPELINE_TABLES: false

trips_merged:
  BIN:
    - column: duration_trip
      label: duration_category
      type: manual_breaks
      bin_breaks: [0, 30, 120]
      bin_labels: [short, long]

The corresponding summarize expression only needs the resulting category:

trips_merged.groupby("duration_category").size().rename("trips").reset_index()

Before the PR, the sequence works:

  1. Merge the original tables, producing duration_trip and duration_tour.
  2. Read duration_trip to create duration_category.
  3. Count trips by category.

With trimming enabled, the helper discovers duration_category in the expression and duration_trip in the YAML. But the input tables contain duration, not duration_trip. Consequently, duration is removed from both tables before merging. The binning operation subsequently raises:

KeyError: 'duration_trip'

The PR already strips suffixes from identifiers found in expressions. The problem is that it adds YAML column references after performing that expansion, preserving them literally.

This affects both BIN and AGGREGATE, and both _trip and _tour references. It can be masked if another expression happens to reference duration or duration_trip, causing the necessary source column to survive.

The fix is to collect expression and YAML references first, then apply suffix expansion to the combined set. Retain the unsuffixed column in both input tables so the merge still produces the expected suffixed names. A regression test should reference the source column only through YAML, with the summarize expression using only the derived label.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants