Skip to content

review sandag one zone vs two zone setup performance #431

Closed

Description

@bstabler
  • Goal is to understand performance differences from 1 zone to 2 zone
  • Compare model specifications for model implementations
  • Tour mode choice slows from 1 zone to 2 zone, why?
  • Trip destination submodel slows down as well
  • Want to control hardware configuration as well
  • @bstabler, @jpn-- compare specs and review logging
  • @wusun2 share logfiles, run with same specs across runs, and update example_sandag (by testing on a fork)

Activity

  1. bstabler commented on Jun 17, 2021

    @bstabler
    ContributorAuthor

    I'll take the lead for this and coordinate with @jpn--. Thanks @esanchez01 for updating the examples. Can you also share the logfiles from your runs?

  2. esanchez01 commented on Jun 18, 2021

    @esanchez01
    Contributor

    I'll take the lead for this and coordinate with @jpn--. Thanks @esanchez01 for updating the examples. Can you also share the logfiles from your runs?

    @bstabler I sent over the location of the 1 & 2 zone logfiles through email.

  3. bstabler commented on Jun 24, 2021

    @bstabler
    ContributorAuthor

    @jpn-- and I did some initial review and found the following:

    1 - the expression evaluation for the tour_mode_choice is slower for ONE zone versus TWO zone

    ONE ZONE

    16/06/2021 23:47:30 - INFO - activitysim.core.mem - trace_memory_info tour_mode_choice.work.simple_simulate.chunk_1.eval_nl.eval_utils.add.expression_values rss: 62.41GB used: 122.29 GB percent: 38.2%
    16/06/2021 23:47:35 - INFO - activitysim.core.mem - trace_memory_info tour_mode_choice.work.simple_simulate.chunk_1.eval_nl.eval_utils.add.utilities rss: 62.51GB used: 122.28 GB percent: 38.2%

    TWO ZONE

    16/06/2021 15:22:09 - INFO - activitysim.core.mem - trace_memory_info tour_mode_choice.work.simple_simulate.chunk_1.eval_nl.eval_utils.add.expression_values rss: 62.72GB used: 137.57 GB percent: 43.0%
    16/06/2021 15:24:51 - INFO - activitysim.core.mem - trace_memory_info tour_mode_choice.work.simple_simulate.chunk_1.eval_nl.eval_utils.add.utilities rss: 62.83GB used: 149.6 GB percent: 46.8%

    2 - the expression files are very similar

    ONE ZONE

    https://github.com/esanchez01/activitysim/blob/develop/activitysim/examples/example_mtc/configs/tour_mode_choice.csv

    TWO ZONE

    https://github.com/esanchez01/activitysim/blob/develop/activitysim/examples/example_psrc/configs/tour_mode_choice.csv

    3 - Some more specific logging within a process shows roughly the same number of tours being processed in the chunk and similar tour mode shares but longer runtimes

    ONE ZONE

    Line 2703 in mp_households_0-activitysim.log

    The chunk runtime is 6 seconds for 7029 tours and the mode share is reported

    DRIVEALONEFREE 2044
    SHARED2FREE 1673
    SHARED3FREE 1528
    WALK 830
    WALK_LRF 298
    WALK_EXP 164
    WALK_LOC 130
    TNC_SINGLE 120
    BIKE 80
    TAXI 44

    TWO ZONE

    Line 2678 in mp_households_0-activitysim.log

    The chunk runtime is 166 sec for 7087 tours and the mode share is basically the same

    DRIVEALONEFREE 2028
    SHARED2FREE 1705
    SHARED3FREE 1560
    WALK 897
    WALK_LRF 267
    WALK_EXP 159
    WALK_LOC 115
    TNC_SINGLE 110
    BIKE 94
    TAXI 33

    4 - so it seems we need to debug/profile a bit to figure out exactly why.

    The network_los engine/setup, and potentially the MAZ to MAZ costs are what's different so it's likely something in there. Without actually profiling, we're wondering about the bellow note in the quick_loc_df() called by get_mazpairs()....

  4. bstabler commented on Jun 24, 2021

    @bstabler
    ContributorAuthor

    We're going to investigate some more to confirm the hypothesis and to hopefully develop an improvement. We'll look at the DaySim and CTRAMP code for ideas as well. Testing with the full scale setup will be important. This would benefit both 2 and 3 zone setups.

  5. added a commit that references this issue on Jun 25, 2021
  6. bstabler commented on Jun 28, 2021

    @bstabler
    ContributorAuthor

    @esanchez01- I'm in the process of testing this improvement on a full scale version of your model and hoping you can test it as well?

  7. esanchez01 commented on Jun 29, 2021

    @esanchez01
    Contributor

    @bstabler, I manually made the change you mentioned to v0.9.9.1, as this is the version my previous tests ran on (no chunking update), and reran the full SANDAG 2-Zone PSRC-based example. There are some noticeable run time improvements when compared to the equivalent 2-Zone run I shared the log files for:

    model_name Pre-Update Post-Update Pre - Post
    compute_accessibility 16.4 15.4 1
    school_location 15.8 10.8 5
    workplace_location 30.7 23.6 7.1
    mandatory_tour_scheduling 27.4 13.5 13.9
    joint_tour_destination 12 1.7 10.3
    non_mandatory_tour_destination 40.1 26.5 13.6
    tour_mode_choice_simulate 25.9 18.5 7.4
    atwork_subtour_destination 6.3 4 2.3
    trip_destination 153.9 48.9 105
    trip_purpose_and_destination 7.6 4.6 3
    trip_mode_choice 21.1 17.5 3.6
    write_trip_matrices 8 13.4 -5.4
    SUB MODEL RUN TIME 388.1 220.8 167.3
    TOTAL RUN TIME 442 258 184

    I will rerun the 3-Zone example with this change as well.

  8. bstabler commented on Jun 29, 2021

    @bstabler
    ContributorAuthor

    I like these improvements. Tour mode choice is still quite a bit slower than the 1 zone example, which, if I recall, was a couple minutes, so I'll keep investigating. @esanchez01 - can you post the 1 zone example runtimes by submodel for reference?

  9. esanchez01 commented on Jun 29, 2021

    @esanchez01
    Contributor

    Here are the results of the 3-Zone rerun:

    model_name Pre-Update Post-Update Pre - Post
    compute_accessibility 42.3 39.3 3
    school_location 21.4 13.3 8.1
    workplace_location 41.1 28.2 12.9
    mandatory_tour_scheduling 31.8 17.1 14.7
    joint_tour_destination 17.5 4.5 13
    non_mandatory_tour_destination 49.3 31.7 17.6
    tour_mode_choice_simulate 38.5 30.3 8.2
    atwork_subtour_destination 7.5 5.1 2.4
    atwork_subtour_mode_choice 4.7 3.6 1.1
    trip_destination 135.2 67.5 67.7
    trip_purpose_and_destination 7.8 5.2 2.6
    trip_mode_choice 19 15.7 3.3
    SUB MODEL RUN TIME 442 288.5 153.5
    TOTAL RUN TIME 489 328 161
  10. esanchez01 commented on Jun 29, 2021

    @esanchez01
    Contributor

    I like these improvements. Tour mode choice is still quite a bit slower than the 1 zone example, which, if I recall, was a couple minutes, so I'll keep investigating. @esanchez01 - can you post the 1 zone example runtimes by submodel for reference?

    Complete run time list for an optimized 1-Zone run:

    model_step 1-Zone
    initialize_landuse 0
    initialize_households 1.3
    compute_accessibility 0.9
    school_location 4.8
    workplace_location 10.9
    auto_ownership_simulate 0
    free_parking 0.1
    cdap_simulate 0.4
    mandatory_tour_frequency 0.1
    mandatory_tour_scheduling 6.1
    joint_tour_frequency 0.1
    joint_tour_composition 0
    joint_tour_participation 0.2
    joint_tour_destination 1
    joint_tour_scheduling 0.2
    non_mandatory_tour_frequency 3
    non_mandatory_tour_destination 10
    non_mandatory_tour_scheduling 2.6
    tour_mode_choice_simulate 1.2
    atwork_subtour_frequency 0.1
    atwork_subtour_destination 1.3
    atwork_subtour_scheduling 0.3
    atwork_subtour_mode_choice 0.1
    stop_frequency 0.4
    trip_purpose 0
    trip_destination 27.1
    trip_purpose_and_destination 0
    trip_scheduling 8.3
    trip_mode_choice 1.2
    write_data_dictionary 0.2
    write_trip_matrices 6.6
    write_tables 3.3
    SUB MODEL RUN TIME 91.8
    TOTAL RUN TIME 112
  11. 7 remaining items

  12. bstabler commented on Sep 3, 2021

    @bstabler
    ContributorAuthor

    Below are updated runtimes for 100% samples of the in-development example_sandag 1, 2, and 3 zone system setups on my Intel Xeon 2.1 GHz 32 core 512 GB RAM server with 28 num processors configured and chunking disabled. The runtimes for the 3 configurations are similar, which suggests the differences we saw earlier in the runtimes between the setups was likely due to chunking due to the use of less RAM. Long term we can certainly improve chunking, but simply using more RAM for this specific model configuration goes a long way to a more performant setup.

    SANDAG 1 zone SANDAG 2 zone SANDAG 3 zone
    process_name model_name seconds minutes process_name model_name seconds minutes process_name model_name seconds minutes
    mp_initialize initialize_landuse 0.4 0 mp_initialize initialize_landuse 11.5 0.2 mp_initialize initialize_landuse 10.3 0.2
    mp_initialize initialize_households 49.2 0.8 mp_initialize initialize_households 50.3 0.8 mp_initialize initialize_households 52.3 0.9
    mp_accessibility compute_accessibility 48.6 0.8 mp_accessibility_0 compute_accessibility 17.1 0.3 mp_accessibility_0 compute_accessibility 96.4 1.6
    mp_los initialize_los 11.7 0.2
    mp_tvpb_0 initialize_tvpb 75 1.2
    mp_households_0 school_location 351.2 5.9 mp_households_0 school_location 366.4 6.1 mp_models_0 school_location 531.7 8.9
    mp_households_0 workplace_location 1033.9 17.2 mp_households_0 workplace_location 809.8 13.5 mp_models_0 workplace_location 1142.7 19
    mp_households_0 auto_ownership_simulate 3 0.1 mp_households_0 auto_ownership_simulate 3.1 0.1 mp_models_0 auto_ownership_simulate 1.8 0
    mp_households_0 free_parking 4.2 0.1 mp_households_0 free_parking 5.1 0.1 mp_models_0 free_parking 3.5 0.1
    mp_households_0 cdap_simulate 24.9 0.4 mp_households_0 cdap_simulate 22.9 0.4 mp_models_0 cdap_simulate 20.7 0.3
    mp_households_0 mandatory_tour_frequency 6.5 0.1 mp_households_0 mandatory_tour_frequency 6.7 0.1 mp_models_0 mandatory_tour_frequency 5.1 0.1
    mp_households_0 mandatory_tour_scheduling 510.9 8.5 mp_households_0 mandatory_tour_scheduling 482.7 8 mp_models_0 mandatory_tour_scheduling 546.7 9.1
    mp_households_0 joint_tour_frequency 6.4 0.1 mp_households_0 joint_tour_frequency 7 0.1 mp_models_0 joint_tour_frequency 3.2 0.1
    mp_households_0 joint_tour_composition 4.6 0.1 mp_households_0 joint_tour_composition 5.4 0.1 mp_models_0 joint_tour_composition 1.9 0
    mp_households_0 joint_tour_participation 14.8 0.2 mp_households_0 joint_tour_participation 16.1 0.3 mp_models_0 joint_tour_participation 12.6 0.2
    mp_households_0 joint_tour_destination 43.8 0.7 mp_households_0 joint_tour_destination 54.2 0.9 mp_models_0 joint_tour_destination 73.3 1.2
    mp_households_0 joint_tour_scheduling 12.1 0.2 mp_households_0 joint_tour_scheduling 12.8 0.2 mp_models_0 joint_tour_scheduling 8.2 0.1
    mp_households_0 non_mandatory_tour_frequency 228.1 3.8 mp_households_0 non_mandatory_tour_frequency 137.3 2.3 mp_models_0 non_mandatory_tour_frequency 131.8 2.2
    mp_households_0 non_mandatory_tour_destination 871.9 14.5 mp_households_0 non_mandatory_tour_destination 789.7 13.2 mp_models_0 non_mandatory_tour_destination 1177.9 19.6
    mp_households_0 non_mandatory_tour_scheduling 200.4 3.3 mp_households_0 non_mandatory_tour_scheduling 168.8 2.8 mp_models_0 non_mandatory_tour_scheduling 138.6 2.3
    mp_households_0 tour_mode_choice_simulate 58.3 1 mp_households_0 tour_mode_choice_simulate 846.7 14.1 mp_models_0 tour_mode_choice_simulate 681.5 11.4
    mp_households_0 atwork_subtour_frequency 10.6 0.2 mp_households_0 atwork_subtour_frequency 14.4 0.2 mp_models_0 atwork_subtour_frequency 7 0.1
    mp_households_0 atwork_subtour_destination 140.7 2.3 mp_households_0 atwork_subtour_destination 101.8 1.7 mp_models_0 atwork_subtour_destination 184.6 3.1
    mp_households_0 atwork_subtour_scheduling 17.9 0.3 mp_households_0 atwork_subtour_scheduling 16 0.3 mp_models_0 atwork_subtour_scheduling 11 0.2
    mp_households_0 atwork_subtour_mode_choice 11.7 0.2 mp_households_0 atwork_subtour_mode_choice 91 1.5 mp_models_0 atwork_subtour_mode_choice 77.1 1.3
    mp_households_0 stop_frequency 21.3 0.4 mp_households_0 stop_frequency 23.5 0.4 mp_models_0 stop_frequency 21.3 0.4
    mp_households_0 trip_purpose 7.8 0.1 mp_households_0 trip_purpose 8.5 0.1 mp_models_0 trip_purpose 3.4 0.1
    mp_households_0 trip_destination 2432.2 40.5 mp_households_0 trip_destination 2244.2 37.4 mp_models_0 trip_destination 2140.2 35.7
    mp_households_0 trip_purpose_and_destination 9.5 0.2 mp_households_0 trip_purpose_and_destination 25 0.4 mp_models_0 trip_purpose_and_destination 76.6 1.3
    mp_households_0 trip_scheduling 24.9 0.4 mp_households_0 trip_scheduling 27.6 0.5 mp_models_0 trip_scheduling 19.1 0.3
    mp_households_0 trip_mode_choice 71.5 1.2 mp_households_0 trip_mode_choice 99.1 1.7 mp_models_0 trip_mode_choice 261.4 4.4
    mp_summarize write_data_dictionary 11.6 0.2 mp_summarize write_data_dictionary 14 0.2 mp_summarize write_data_dictionary 15.5 0.3
    mp_summarize track_skim_usage 13.4 0.2 mp_summarize track_skim_usage 11.5 0.2
    mp_summarize write_trip_matrices 354.2 5.9 mp_summarize write_trip_matrices 378.7 6.3 mp_summarize write_trip_matrices 163.8 2.7
    mp_summarize write_tables 196.8 3.3 mp_summarize write_tables 203.4 3.4 mp_summarize write_tables 215.8 3.6
    submodel total 113 submodel total 118 submodel total 132
    overall total 125 overall total 145 overall total 144
  13. esanchez01 commented on Sep 3, 2021

    @esanchez01
    Contributor

    @bstabler, thank you for sharing. If possible, can you add the total (sub model + non sub model) run times? Also, it would be good to know what the RAM usage of each run was to get a sense of how much may be needed and how the different zone systems compare.

  14. wusun2 commented on Sep 9, 2021

    @wusun2

    @bstabler, these are interesting results. I am surprised to see the location choices of 1-zone performed worse than the 2-zone system, some are even worse than the 3-zone system. For example, the trip location time for 1, 2, and 3 zones are 40.5, 37.4, and 35.7 minutes (this is the most time consuming step also, roughly 30-40% of total runtime). This is counter-intuitive, because 1-zone location choice operates at TAZ rather than the MAZ level in 2 and 3 zone systems, shouldn't it be faster or at least not worse than the 2/3 zone system?

  15. joecastiglione commented on Sep 9, 2021

    @joecastiglione

    I was also a bit confused by these results. While super-encouraging from a 3-zone perspective, it is my understanding that there are simply fewer calculations in the 1- and 2-zone systems due to the absence of TVPB, so why is there no performance advantage to these simple setups? Is this a matter of more time having been spent optimizing the 3-zone? If so, I think we need to dig into the 1- and 2- zone performance.

  16. wusun2 commented on Sep 9, 2021

    @wusun2

    @joecastiglione , I agree we need to dig into the 1 and 2 zone performance more. Especially, some of us will use the 1 or 2 zone system as their region's production model.

  17. esanchez01 commented on Sep 19, 2021

    @esanchez01
    Contributor

    @bstabler, I am attaching a workbook that breaks down the 1, 2, and 3 zone run times (ran on a SANDAG server) and contains notable logging for certain sub models that appeared to have performance issues. This includes what was covered during the 9/14 technical call.

    zone_run_time_analysis.xlsx

  18. bstabler commented on Sep 29, 2021

    @bstabler
    ContributorAuthor

    Thanks @esanchez01. I looked into the reported runtime differences for your 1, 2, and 3 zone setups and have a couple of findings and some improvements (in the commit above) to the 2 zone setup to reduce runtimes.

    1. Some of the difference in runtime is due to memory needs, memory available, and the use of chunking. The runtimes for tour and trip mode choice from your 320GB chunked run and my 512GB chunking disabled run are below.

    SANDAG server

    submodel 1 zone 2 zone 3 zone
    tour mode choice 1.2 22.2 50.5
    trip mode choice 1.6 13.1 6.5

    My server

    submodel 1 zone 2 zone 3 zone
    tour mode choice 1 14.1 11.4
    trip mode choice 1.2 1.7 4.4

    I believe some of the increase in runtimes for trip mode choice from the 1 zone to 2 zone setup is due to memory and chunking since my run without chunking has a more reasonable increase in runtime from 1 zone to 2 zone to 3 zone setup. However, I do not expect my chunking disabled tour mode choice setup to go from 1 min for a 1 zone setup to 14 min for a 2 zone setup so I looked into this.

    1. What I found is that repeatedly calling the sparse MAZ to MAZ data lookups, such as od_skims['DISTWALK'], is largely responsible for the slow down. The example_psrc expressions have several repeat calls to get the data, which gets the data from a pandas DataFrame keyed by an OMAZ-DMAZ compound key. To try and reduce runtimes, I gradually simplified the expressions for tour_mode_choice as follows:
    SANDAG 2 zone minutes
    tour_mode_choice model runtime without changes 13
    make just one od_skims['DISTWALK'] call for walk, and also for bike and general distance, in the pre-processor and then use the result in the expression files 3
    also drop od_skims['DIST'] from the MAZ to MAZ tables so there isn't a blended skimmed DIST + a MAZ to MAZ DIST (... i.e. just use the skimmed value) 2.8
    also drop a reverse DMAZ to OMAZ lookup and just assume O->D distance can be used for D->O as well (which is common for bike and walk) 1.9

    I then ran the complete 2 zone example again, using the same setup as mentioned in an earlier comment but with these revisions, and below are the revised runtimes. The submodels run about 20% faster, and are actually faster than the 1 zone system setup. I did not revise the 3 zone expression files, but they could also be improved, as well as probably the 1 zone system files as well, but they use numpy array indexing instead of pandas DataFrame indexing.

    SANDAG 2 zone before after
    process_name model_name minutes minutes
    mp_initialize initialize_landuse 0.2 0.2
    mp_initialize initialize_households 0.8 0.8
    mp_accessibility_0 compute_accessibility 0.3 0.3
    mp_households_0 school_location 6.1 5.7
    mp_households_0 workplace_location 13.5 11.9
    mp_households_0 auto_ownership_simulate 0.1 0
    mp_households_0 free_parking 0.1 0.1
    mp_households_0 cdap_simulate 0.4 0.4
    mp_households_0 mandatory_tour_frequency 0.1 0.1
    mp_households_0 mandatory_tour_scheduling 8 7.1
    mp_households_0 joint_tour_frequency 0.1 0.1
    mp_households_0 joint_tour_composition 0.1 0.1
    mp_households_0 joint_tour_participation 0.3 0.1
    mp_households_0 joint_tour_destination 0.9 0.7
    mp_households_0 joint_tour_scheduling 0.2 0.2
    mp_households_0 non_mandatory_tour_frequency 2.3 2.1
    mp_households_0 non_mandatory_tour_destination 13.2 12
    mp_households_0 non_mandatory_tour_scheduling 2.8 2.8
    mp_households_0 tour_mode_choice_simulate 14.1 2.1
    mp_households_0 atwork_subtour_frequency 0.2 0.1
    mp_households_0 atwork_subtour_destination 1.7 1.7
    mp_households_0 atwork_subtour_scheduling 0.3 0.2
    mp_households_0 atwork_subtour_mode_choice 1.5 0.3
    mp_households_0 stop_frequency 0.4 0.3
    mp_households_0 trip_purpose 0.1 0.1
    mp_households_0 trip_destination 37.4 34.5
    mp_households_0 trip_purpose_and_destination 0.4 0.4
    mp_households_0 trip_scheduling 0.5 0.4
    mp_households_0 trip_mode_choice 1.7 1.5
    mp_summarize write_data_dictionary 0.2 0.2
    mp_summarize track_skim_usage 0.2 0.2
    mp_summarize write_trip_matrices 6.3 5.8
    mp_summarize write_tables 3.4 3.1
    submodel total 118 96
    overall total 145 113

    We should think about ways to improve MAZ to MAZ impedance storage and retrieval speed to speed this up. Possible ideas are by using numpy arrays, a different smarter index/search method, and/or caching.

    @esanchez01 - can you test on your end and do a more thorough expression file update of your various setups? You may also want to update the full scale maz_to_maz_bike.csv on the activitysim resources repo by dropping DIST if you do not want to blend general distance as well.

    P.S. I also noticed that the transit skims in the expressions have /100 in them, which is a hold over from the MTC skims, which are Cube TRANPLAN skims, which are scaled ints and therefore need to be /100 before use. These should be dropped if not needed.

  19. esanchez01 commented on Oct 4, 2021

    @esanchez01
    Contributor

    @bstabler, I included your changes in the SANDAG 2 zone setup and reran on our server. Both the sub models and the overall run ran about 25% faster -- down to ~140 minutes and ~180 minutes, respectively. Tour mode choice saw the most improvement (~19 minutes to 3 minutes) and improvements were seen across other sub models, consistent with what was observed from your test. I am attaching the results here: maz_maz_update_runtime.xlsx

    I will look further into the expression files and make adjustments.

    NOTE: The 1, 2, and 3 zone run times I previously shared in the workbook are all of MTC based setups, whereas the SANDAG 2 zone setup is PSRC based (as you tested). I did this to allow a more apples to apples comparison of the different zone systems. I believe this to be the reason why there is a big difference in the 2 zone trip mode choice run time between your test and mine, as you noted. For the reruns, I used the PSRC based setup and observed similar trip mode choice run times as in your tests.

  20. bstabler commented on Oct 7, 2021

    @bstabler
    ContributorAuthor

    @esanchez01 - did you fix the /100 issue as well? If so, can you PR your improvements to the fewer-maz-maz-data-calls-for-sandag-two-zone branch?

  21. bstabler commented on Oct 7, 2021

    @bstabler
    ContributorAuthor

    This issue will be closed once we update the examples with the reduced and reused expressions

  22. bstabler commented on Oct 7, 2021

    @bstabler
    ContributorAuthor

    And we'll create a new issue to look into the trip_destination model since it remains the slowest submodel.

  23. esanchez01 commented on Oct 12, 2021

    @esanchez01
    Contributor

    @bstabler, I have not yet fixed the /100 issue but will get to it, as well as updating additional expressions, soon. Since the SANDAG examples pull MTC and PSRC expression files, I won't be making updates directly to these files but probably just storing the updated expression files in the SANDAG example directories.

  24. bstabler commented on Oct 13, 2021

    @bstabler
    ContributorAuthor

    We're going to create a new issue for the /100 issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions