Repository navigation
Calibration - #1107
Calibration#1107dhensle wants to merge 102 commits into
Conversation
…nto calibration
also fixing some bugs captured by AI review
At the component iteration limit, calibration already ran an extra simulation with the newly persisted coefficients. However, that call bypassed expression evaluation and reporting, leaving convergence status, histories, and reports describing the preceding simulation. Final model outputs could match a target while calibration still reported failure. Move the existing final simulation into an evaluation-only loop pass. Rebuild the expression context, evaluate targets and convergence, and refresh histories and enabled generic and bespoke reports. Skip coefficient adjustments, bound clipping, and coefficient persistence on this pass so reported results describe the coefficients actually simulated. Give the final evaluation its own history index at submodel_max_iterations + 1, preserving the preceding coefficient update and recording zero final coefficient change. Retain the existing count of calibration update iterations and the early-convergence behavior. Add regression coverage for final-pass convergence and nonconvergence, early convergence, refreshed reports and CSV histories, and evaluation without coefficient adjustment or bound clipping. Document the final-pass behavior and history numbering. Validation: 39 calibration unit tests and 6 integration tests passed, including single/multiprocess equivalence, restart, continuation, and rewind scenarios. Black and git diff --check passed.
The calibration CSV reader selected only required columns, silently removing the optional default_increment column. Although the coefficient updater supported this setting, it always received rows without it and fell back to 2.0. A configured increment of 0.25 therefore produced a 2.0 adjustment when a log-ratio or odds-ratio calculation needed the fallback. Retain default_increment during CSV loading and validate it as a finite, nonnegative numeric magnitude. Use 2.0 for omitted columns and blank cells, and preserve explicit zero values so users can disable fallback adjustments for individual rows. Leave the separate fallback damping behavior unchanged. Document these semantics and add CSV-to-update regression coverage for both adjustment methods and directions, custom and zero increments, missing and blank values, and invalid inputs. Validation: all 51 calibration unit tests passed. Black and staged whitespace checks passed.
The updater preserved held or converged values before applying bounds, which could still change them. For example, a held coefficient of 5 with bounds [-1, 1] was silently clipped to 1. Clipping a converged coefficient also changed the value used by the simulation that established convergence, leaving the reported result inconsistent with the persisted coefficient. Calculate adjustments and apply bounds only to coefficients eligible for an update: neither held nor converged, and outside an evaluation-only pass. Continue evaluating target differences and reporting bound flags for fixed values; hold_fast does not imply convergence. Retain the existing warning policy for initial values outside bounds and document that held or converged values remain unchanged. Users requiring a hard constraint must correct starting values or bounds before calibration. Bounds continue to clip eligible coefficient updates normally. Add regression coverage for held and converged coefficients above and below bounds, unchanged coefficient records, target convergence status, initial warnings, and clipping of eligible updates in both directions. Validation: all 59 calibration unit tests passed. Black and staged whitespace checks passed.
… rows Templated models previously accepted every coefficient listed anywhere in COEFFICIENT_TEMPLATE, including entries whose generic names never appeared in the utility specification. This allowed calibration to adjust unused coefficients without affecting the model's utilities. Extract tokens from utility columns first, then map only referenced template rows to their segment-specific coefficient names. Preserve direct-token validation for models without templates and the template reader's handling of blank cells. Labels, descriptions, expressions, and comments do not count as coefficient references in utility columns. Add regression coverage for referenced and unused template rows, multiple segments, blank-cell inheritance, supplemental specifications, and direct coefficient references. Document that this is a static reference check, not a guarantee that a segment occurs in the simulation sample. Validation: 61 calibration unit tests passed. The stricter check detects exactly the six disconnected zero-auto SHARED2 coefficients in the original SANDAG calibration tables; corrected SANDAG configurations pass five tests. Both prototype MTC calibration configurations also validate. Formatting and staged whitespace checks passed. No full regional calibration was run.
Calibration constants were appended to four expected parameter tables, while Larch includes them in alphabetical order. Positional CSV comparisons therefore paired different coefficients and failed despite passing likelihood checks. Reorder those expected rows without changing any values or tolerances, and sort unique-name parameter tables before regression comparison. Preserve the existing grouped order for concatenated segment tables with repeated names. Add focused checks that row reordering passes while missing, extra, renamed, duplicate, and materially changed parameters fail. Check that the four affected baselines retain unique, sorted names. Add a focused Windows estimation workflow using the same locked dependencies as Core Testing. Rebuild estimation data from a fresh cache and upload complete CSV comparisons and JUnit results so numerical discrepancies can be inspected without relying on truncated console output. Local validation: seven parameter-comparison tests passed; all expected CSV row contents are unchanged. Windows estimation validation follows in Actions.
Integrate estimation-test-fix while excluding the dedicated .github/workflows/estimation-tests.yml validation workflow from the calibration branch. Sort uniquely named parameter results and reorder the four affected regression baselines without changing expected values or tolerances. Preserve grouped comparisons with repeated parameter names and add checks that meaningful parameter changes still fail regression validation. The source fix passed all 28 existing estimation mode tests and seven new regression checks on Windows in driftlesslabs/activitysim Actions run 35681053967. Only the standalone workflow is excluded from this merge; the existing Core Testing estimation job remains unchanged.
Fix calibration diagnostics, coefficient handling, and estimation regressions
|
I've reviewed the PR and I think overall the workflow is easy enough to follow and run and consistent with ActivitySim's choice model specifications. It's also quite flexible and well documented, with a good example. The only thing I would like to understand a bit better is the implementation of multi-processing and associated helpers. The calibration version seems to provide similar functionality to the core implementation, so I wonder if we could extend the core instead. I think there are a couple of issues that opus5.5 found that are a consequence of having two versions here, see 2, 3, 4 below. Is there a technical reason that makes them hard to combine? Opus5.5 review results:
|
This pull request introduces the new automated calibration workflow for ActivitySim, modularizing calibration logic into a dedicated package and integrating it into the main simulation runner. The changes include the addition of a comprehensive calibration package, updates to the CLI to support calibration runs and output preservation, and enhancements to model settings validation to include calibration configuration.
Calibration package introduction and integration:
activitysim.core.calibrationpackage, splitting calibration logic into focused modules such asorchestrator.py,settings.py,component.py, and more, with a clear public API. [1] [2] [3]activitysim/cli/run.pyto import and utilize the calibration package, including logic to determine when to preserve outputs and to run the calibration workflow before standard or multiprocess simulation. [1] [2] [3]Settings and validation improvements:
settings_checker.pyto explicitly validate the optionalcalibration.yamlconfiguration file using the same error aggregation as other settings. [1] [2]Random number generator and reproducibility:
initialize_households, ensured that bothhouseholdsandpersonsdataframes are registered as random number generator channels to support reproducible calibration and simulation.Documentation:
README.mdfor the calibration package, describing its architecture, configuration, iteration/recovery semantics, and integration with the main workflow.