Repository navigation
Reduce memory of summarize step by dropping unused variables before wide merge - #1125
i-am-sijia wants to merge 3 commits into
Conversation
abench resultsAuto-generated by abench 0.2.0. PR: #1125 · Baseline: base
Negative changes mean lower elapsed time or memory. Changes are omitted for invalid runs or differing comparison settings. Single runs do not establish statistical significance. Configuration and source commitsbase — base
pr_off — pr_off
pr_on — pr_on
Memory tracesBlue: total cgroup memory; green dashed: anonymous + shared memory when available. Panels share axes; invalid runs are labeled. Component runtimesValid runs only. Mean ± population standard deviation across worker executions, not confidence intervals. Parallel component times must not be summed as wall time. |
There was a problem hiding this comment.
abench results posted above show the fix is effective at reducing memory usage in the summarize step, at least in the MTC model. I see no reason why these results wouldn't generalize to other models. Overall runtime shows some variation that is purely exogenous; the increased runtime on the run with the feature enabled is entirely on components that this PR does not touch, and is possibly due to thermal throttling due to the relevant test running last.
I found 3 problems in the code that should be addressed. None are surfaced by existing tests. In addition to fixes, new tests should be included that cover all three problems.
| expressions_df = spec[["Expression"]].dropna().astype(str) | ||
| if model_settings.preprocessor is not None: | ||
| preprocessor_spec = pd.read_csv( | ||
| state.filesystem.get_config_file_path(model_settings.preprocessor.SPEC), |
There was a problem hiding this comment.
The existing preprocessor loader explicitly supports omitting the filename extension. This is valid:
EXPORT_PIPELINE_TABLES: false
preprocessor:
DF: trips_merged
SPEC: summarize_preprocessorThe actual file on disk is:
summarize_preprocessor.csv
Before the PR, compute_columns() normalizes the name:
if not expressions_spec_name.endswith(".csv"):
expressions_spec_name = f"{expressions_spec_name}.csv"It therefore finds and evaluates the CSV successfully.
The new trimming helper needs to inspect the preprocessor expressions before merging. But it resolves the raw SPEC value directly:
state.filesystem.get_config_file_path(model_settings.preprocessor.SPEC)For the example above, it searches for a file literally named summarize_preprocessor, without an extension. That fails with FileNotFoundError before the normal preprocessor loader gets a chance to add .csv.
The same configuration works when trimming is disabled, or when pipeline-table export bypasses trimming. That makes this a configuration compatibility regression rather than a missing input file.
The immediate fix is to apply the same extension normalization before resolving the file. More robustly, the dependency scanner should share the existing preprocessor specification-loading logic so the two paths interpret settings consistently. A regression test should use an extensionless SPEC, provide only the .csv file, and compare trimming-enabled and trimming-disabled results.
For existing users, DROP_UNUSED_COLUMNS: false avoids all three regressions while preserving the previous behavior. These cases all use explicit, statically identifiable column names or an already-supported filename convention; they don’t depend on dynamically constructed column references.
| keep |= {"origin", "destination"} | ||
| keep |= {i.removesuffix(s) for i in identifiers for s in ("_trip", "_tour")} | ||
|
|
||
| for table_name in ("persons_merged", "trips_merged", "tours_merged", "land_use"): |
There was a problem hiding this comment.
The existing summarize step exposes several tables in its expression namespace, including:
trips
trips_merged
tours
tours_mergedIt also processes BIN and AGGREGATE settings for those names. In particular, tours is an alias for tours_merged.
However, the new dependency collector only examines these settings sections:
("persons_merged", "trips_merged", "tours_merged", "land_use")It omits trips and tours, even though those are the tables affected by trimming.
For example:
EXPORT_PIPELINE_TABLES: false
trips:
AGGREGATE:
- column: trip_mode
label: broad_mode
map:
WALK: active
BIKE: active
DRIVEALONEFREE: autoWith a summarize expression:
trips.groupby("broad_mode").size().rename("trips").reset_index()The intended sequence is straightforward: read trip_mode, create broad_mode, then count trips by that grouping.
The trimmer sees broad_mode in the expression, but it never inspects the trips settings section to discover its dependency on trip_mode. If no other expression requires trip_mode, it drops that column. The existing aggregation code then attempts:
df["trip_mode"].map(...)and raises:
KeyError: 'trip_mode'
The same problem occurs with a tours: section that bins or aggregates a column used nowhere else.
This is separate from the first bug: no merge suffixes are involved. Even a plain source-column name is lost because its entire settings section is skipped.
The fix is to include trips and tours when collecting YAML dependencies—or, more generally, derive the settings names from the same table namespace that the classification loop uses. Tests should cover both aliases with a source column referenced only in their YAML settings.
| for table_name in ("persons_merged", "trips_merged", "tours_merged", "land_use"): | ||
| meta = getattr(model_settings, table_name, None) or {} | ||
| for slicer in (meta.get("AGGREGATE") or []) + (meta.get("BIN") or []): | ||
| keep.add(slicer["column"]) |
There was a problem hiding this comment.
The merge combines trips and tours_merged. When both tables contain a column with the same name, pandas adds _trip and _tour suffixes.
For example, suppose the input tables contain:
| Input table | Column | Value |
|---|---|---|
trips |
duration |
15 |
tours_merged |
duration |
90 |
After merging, those become duration_trip = 15 and duration_tour = 90.
A valid configuration could classify trip durations before summarizing:
EXPORT_PIPELINE_TABLES: false
trips_merged:
BIN:
- column: duration_trip
label: duration_category
type: manual_breaks
bin_breaks: [0, 30, 120]
bin_labels: [short, long]The corresponding summarize expression only needs the resulting category:
trips_merged.groupby("duration_category").size().rename("trips").reset_index()Before the PR, the sequence works:
- Merge the original tables, producing
duration_tripandduration_tour. - Read
duration_tripto createduration_category. - Count trips by category.
With trimming enabled, the helper discovers duration_category in the expression and duration_trip in the YAML. But the input tables contain duration, not duration_trip. Consequently, duration is removed from both tables before merging. The binning operation subsequently raises:
KeyError: 'duration_trip'
The PR already strips suffixes from identifiers found in expressions. The problem is that it adds YAML column references after performing that expansion, preserving them literally.
This affects both BIN and AGGREGATE, and both _trip and _tour references. It can be masked if another expression happens to reference duration or duration_trip, causing the necessary source column to survive.
The fix is to collect expression and YAML references first, then apply suffix expansion to the combined set. Retain the unsuffixed column in both input tables so the merge still produces the expected suffixed names. A regression test should reference the source column only through YAML, with the summarize expression using only the derived label.
This PR addresses #1124.
This pull request introduces a new feature to the
summarizemodel that trims unused columns from tables before merging, improving memory usage and performance. It also adds a corresponding test to ensure this behavior works as intended. The most important changes are grouped below:Core feature: Trimming unused columns
trim_unused_columnstosummarize.pythat drops columns from input tables that are not referenced in the summarize specification or related preprocessor specs, with logic to retain necessary columns for merging and aggregation.DROP_UNUSED_COLUMNS(defaultTrue) inSummarizeSettingsto control whether unused columns are dropped, with logic to skip trimming ifEXPORT_PIPELINE_TABLESisTrue.trim_unused_columnsinto the mainsummarizefunction so that tables are trimmed before merging, based on the new setting.Refactoring and utility
delete_temporary_variablesfor clarity and reuse.Testing
test_summarize_drop_unused_columnsto verify that columns are correctly trimmed when the feature is enabled, and that the output remains correct and consistent.