You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
review sandag one zone vs two zone setup performance #431
3 - Some more specific logging within a process shows roughly the same number of tours being processed in the chunk and similar tour mode shares but longer runtimes
ONE ZONE
Line 2703 in mp_households_0-activitysim.log
The chunk runtime is 6 seconds for 7029 tours and the mode share is reported
DRIVEALONEFREE 2044
SHARED2FREE 1673
SHARED3FREE 1528
WALK 830
WALK_LRF 298
WALK_EXP 164
WALK_LOC 130
TNC_SINGLE 120
BIKE 80
TAXI 44
TWO ZONE
Line 2678 in mp_households_0-activitysim.log
The chunk runtime is 166 sec for 7087 tours and the mode share is basically the same
DRIVEALONEFREE 2028
SHARED2FREE 1705
SHARED3FREE 1560
WALK 897
WALK_LRF 267
WALK_EXP 159
WALK_LOC 115
TNC_SINGLE 110
BIKE 94
TAXI 33
4 - so it seems we need to debug/profile a bit to figure out exactly why.
The network_los engine/setup, and potentially the MAZ to MAZ costs are what's different so it's likely something in there. Without actually profiling, we're wondering about the bellow note in the quick_loc_df() called by get_mazpairs()....
We're going to investigate some more to confirm the hypothesis and to hopefully develop an improvement. We'll look at the DaySim and CTRAMP code for ideas as well. Testing with the full scale setup will be important. This would benefit both 2 and 3 zone setups.
@bstabler, I manually made the change you mentioned to v0.9.9.1, as this is the version my previous tests ran on (no chunking update), and reran the full SANDAG 2-Zone PSRC-based example. There are some noticeable run time improvements when compared to the equivalent 2-Zone run I shared the log files for:
model_name
Pre-Update
Post-Update
Pre - Post
compute_accessibility
16.4
15.4
1
school_location
15.8
10.8
5
workplace_location
30.7
23.6
7.1
mandatory_tour_scheduling
27.4
13.5
13.9
joint_tour_destination
12
1.7
10.3
non_mandatory_tour_destination
40.1
26.5
13.6
tour_mode_choice_simulate
25.9
18.5
7.4
atwork_subtour_destination
6.3
4
2.3
trip_destination
153.9
48.9
105
trip_purpose_and_destination
7.6
4.6
3
trip_mode_choice
21.1
17.5
3.6
write_trip_matrices
8
13.4
-5.4
SUB MODEL RUN TIME
388.1
220.8
167.3
TOTAL RUN TIME
442
258
184
I will rerun the 3-Zone example with this change as well.
I like these improvements. Tour mode choice is still quite a bit slower than the 1 zone example, which, if I recall, was a couple minutes, so I'll keep investigating. @esanchez01 - can you post the 1 zone example runtimes by submodel for reference?
I like these improvements. Tour mode choice is still quite a bit slower than the 1 zone example, which, if I recall, was a couple minutes, so I'll keep investigating. @esanchez01 - can you post the 1 zone example runtimes by submodel for reference?
Complete run time list for an optimized 1-Zone run:
Below are updated runtimes for 100% samples of the in-development example_sandag 1, 2, and 3 zone system setups on my Intel Xeon 2.1 GHz 32 core 512 GB RAM server with 28 num processors configured and chunking disabled. The runtimes for the 3 configurations are similar, which suggests the differences we saw earlier in the runtimes between the setups was likely due to chunking due to the use of less RAM. Long term we can certainly improve chunking, but simply using more RAM for this specific model configuration goes a long way to a more performant setup.
@bstabler, thank you for sharing. If possible, can you add the total (sub model + non sub model) run times? Also, it would be good to know what the RAM usage of each run was to get a sense of how much may be needed and how the different zone systems compare.
@bstabler, these are interesting results. I am surprised to see the location choices of 1-zone performed worse than the 2-zone system, some are even worse than the 3-zone system. For example, the trip location time for 1, 2, and 3 zones are 40.5, 37.4, and 35.7 minutes (this is the most time consuming step also, roughly 30-40% of total runtime). This is counter-intuitive, because 1-zone location choice operates at TAZ rather than the MAZ level in 2 and 3 zone systems, shouldn't it be faster or at least not worse than the 2/3 zone system?
I was also a bit confused by these results. While super-encouraging from a 3-zone perspective, it is my understanding that there are simply fewer calculations in the 1- and 2-zone systems due to the absence of TVPB, so why is there no performance advantage to these simple setups? Is this a matter of more time having been spent optimizing the 3-zone? If so, I think we need to dig into the 1- and 2- zone performance.
@joecastiglione , I agree we need to dig into the 1 and 2 zone performance more. Especially, some of us will use the 1 or 2 zone system as their region's production model.
@bstabler, I am attaching a workbook that breaks down the 1, 2, and 3 zone run times (ran on a SANDAG server) and contains notable logging for certain sub models that appeared to have performance issues. This includes what was covered during the 9/14 technical call.
Thanks @esanchez01. I looked into the reported runtime differences for your 1, 2, and 3 zone setups and have a couple of findings and some improvements (in the commit above) to the 2 zone setup to reduce runtimes.
Some of the difference in runtime is due to memory needs, memory available, and the use of chunking. The runtimes for tour and trip mode choice from your 320GB chunked run and my 512GB chunking disabled run are below.
SANDAG server
submodel
1 zone
2 zone
3 zone
tour mode choice
1.2
22.2
50.5
trip mode choice
1.6
13.1
6.5
My server
submodel
1 zone
2 zone
3 zone
tour mode choice
1
14.1
11.4
trip mode choice
1.2
1.7
4.4
I believe some of the increase in runtimes for trip mode choice from the 1 zone to 2 zone setup is due to memory and chunking since my run without chunking has a more reasonable increase in runtime from 1 zone to 2 zone to 3 zone setup. However, I do not expect my chunking disabled tour mode choice setup to go from 1 min for a 1 zone setup to 14 min for a 2 zone setup so I looked into this.
What I found is that repeatedly calling the sparse MAZ to MAZ data lookups, such as od_skims['DISTWALK'], is largely responsible for the slow down. The example_psrc expressions have several repeat calls to get the data, which gets the data from a pandas DataFrame keyed by an OMAZ-DMAZ compound key. To try and reduce runtimes, I gradually simplified the expressions for tour_mode_choice as follows:
SANDAG 2 zone
minutes
tour_mode_choice model runtime without changes
13
make just one od_skims['DISTWALK'] call for walk, and also for bike and general distance, in the pre-processor and then use the result in the expression files
3
also drop od_skims['DIST'] from the MAZ to MAZ tables so there isn't a blended skimmed DIST + a MAZ to MAZ DIST (... i.e. just use the skimmed value)
2.8
also drop a reverse DMAZ to OMAZ lookup and just assume O->D distance can be used for D->O as well (which is common for bike and walk)
1.9
I then ran the complete 2 zone example again, using the same setup as mentioned in an earlier comment but with these revisions, and below are the revised runtimes. The submodels run about 20% faster, and are actually faster than the 1 zone system setup. I did not revise the 3 zone expression files, but they could also be improved, as well as probably the 1 zone system files as well, but they use numpy array indexing instead of pandas DataFrame indexing.
SANDAG 2 zone
before
after
process_name
model_name
minutes
minutes
mp_initialize
initialize_landuse
0.2
0.2
mp_initialize
initialize_households
0.8
0.8
mp_accessibility_0
compute_accessibility
0.3
0.3
mp_households_0
school_location
6.1
5.7
mp_households_0
workplace_location
13.5
11.9
mp_households_0
auto_ownership_simulate
0.1
0
mp_households_0
free_parking
0.1
0.1
mp_households_0
cdap_simulate
0.4
0.4
mp_households_0
mandatory_tour_frequency
0.1
0.1
mp_households_0
mandatory_tour_scheduling
8
7.1
mp_households_0
joint_tour_frequency
0.1
0.1
mp_households_0
joint_tour_composition
0.1
0.1
mp_households_0
joint_tour_participation
0.3
0.1
mp_households_0
joint_tour_destination
0.9
0.7
mp_households_0
joint_tour_scheduling
0.2
0.2
mp_households_0
non_mandatory_tour_frequency
2.3
2.1
mp_households_0
non_mandatory_tour_destination
13.2
12
mp_households_0
non_mandatory_tour_scheduling
2.8
2.8
mp_households_0
tour_mode_choice_simulate
14.1
2.1
mp_households_0
atwork_subtour_frequency
0.2
0.1
mp_households_0
atwork_subtour_destination
1.7
1.7
mp_households_0
atwork_subtour_scheduling
0.3
0.2
mp_households_0
atwork_subtour_mode_choice
1.5
0.3
mp_households_0
stop_frequency
0.4
0.3
mp_households_0
trip_purpose
0.1
0.1
mp_households_0
trip_destination
37.4
34.5
mp_households_0
trip_purpose_and_destination
0.4
0.4
mp_households_0
trip_scheduling
0.5
0.4
mp_households_0
trip_mode_choice
1.7
1.5
mp_summarize
write_data_dictionary
0.2
0.2
mp_summarize
track_skim_usage
0.2
0.2
mp_summarize
write_trip_matrices
6.3
5.8
mp_summarize
write_tables
3.4
3.1
submodel total
118
96
overall total
145
113
We should think about ways to improve MAZ to MAZ impedance storage and retrieval speed to speed this up. Possible ideas are by using numpy arrays, a different smarter index/search method, and/or caching.
@esanchez01 - can you test on your end and do a more thorough expression file update of your various setups? You may also want to update the full scale maz_to_maz_bike.csv on the activitysim resources repo by dropping DIST if you do not want to blend general distance as well.
P.S. I also noticed that the transit skims in the expressions have /100 in them, which is a hold over from the MTC skims, which are Cube TRANPLAN skims, which are scaled ints and therefore need to be /100 before use. These should be dropped if not needed.
@bstabler, I included your changes in the SANDAG 2 zone setup and reran on our server. Both the sub models and the overall run ran about 25% faster -- down to ~140 minutes and ~180 minutes, respectively. Tour mode choice saw the most improvement (~19 minutes to 3 minutes) and improvements were seen across other sub models, consistent with what was observed from your test. I am attaching the results here: maz_maz_update_runtime.xlsx
I will look further into the expression files and make adjustments.
NOTE: The 1, 2, and 3 zone run times I previously shared in the workbook are all of MTC based setups, whereas the SANDAG 2 zone setup is PSRC based (as you tested). I did this to allow a more apples to apples comparison of the different zone systems. I believe this to be the reason why there is a big difference in the 2 zone trip mode choice run time between your test and mine, as you noted. For the reruns, I used the PSRC based setup and observed similar trip mode choice run times as in your tests.
@bstabler, I have not yet fixed the /100 issue but will get to it, as well as updating additional expressions, soon. Since the SANDAG examples pull MTC and PSRC expression files, I won't be making updates directly to these files but probably just storing the updated expression files in the SANDAG example directories.
Activity
bstabler commentedon Jun 17, 2021
I'll take the lead for this and coordinate with @jpn--. Thanks @esanchez01 for updating the examples. Can you also share the logfiles from your runs?
esanchez01 commentedon Jun 18, 2021
@bstabler I sent over the location of the 1 & 2 zone logfiles through email.
bstabler commentedon Jun 24, 2021
@jpn-- and I did some initial review and found the following:
1 - the expression evaluation for the tour_mode_choice is slower for ONE zone versus TWO zone
ONE ZONE
16/06/2021 23:47:30 - INFO - activitysim.core.mem - trace_memory_info tour_mode_choice.work.simple_simulate.chunk_1.eval_nl.eval_utils.add.expression_values rss: 62.41GB used: 122.29 GB percent: 38.2%
16/06/2021 23:47:35 - INFO - activitysim.core.mem - trace_memory_info tour_mode_choice.work.simple_simulate.chunk_1.eval_nl.eval_utils.add.utilities rss: 62.51GB used: 122.28 GB percent: 38.2%
TWO ZONE
16/06/2021 15:22:09 - INFO - activitysim.core.mem - trace_memory_info tour_mode_choice.work.simple_simulate.chunk_1.eval_nl.eval_utils.add.expression_values rss: 62.72GB used: 137.57 GB percent: 43.0%
16/06/2021 15:24:51 - INFO - activitysim.core.mem - trace_memory_info tour_mode_choice.work.simple_simulate.chunk_1.eval_nl.eval_utils.add.utilities rss: 62.83GB used: 149.6 GB percent: 46.8%
2 - the expression files are very similar
ONE ZONE
https://github.com/esanchez01/activitysim/blob/develop/activitysim/examples/example_mtc/configs/tour_mode_choice.csv
TWO ZONE
https://github.com/esanchez01/activitysim/blob/develop/activitysim/examples/example_psrc/configs/tour_mode_choice.csv
3 - Some more specific logging within a process shows roughly the same number of tours being processed in the chunk and similar tour mode shares but longer runtimes
ONE ZONE
Line 2703 in mp_households_0-activitysim.log
The chunk runtime is 6 seconds for 7029 tours and the mode share is reported
DRIVEALONEFREE 2044
SHARED2FREE 1673
SHARED3FREE 1528
WALK 830
WALK_LRF 298
WALK_EXP 164
WALK_LOC 130
TNC_SINGLE 120
BIKE 80
TAXI 44
TWO ZONE
Line 2678 in mp_households_0-activitysim.log
The chunk runtime is 166 sec for 7087 tours and the mode share is basically the same
DRIVEALONEFREE 2028
SHARED2FREE 1705
SHARED3FREE 1560
WALK 897
WALK_LRF 267
WALK_EXP 159
WALK_LOC 115
TNC_SINGLE 110
BIKE 94
TAXI 33
4 - so it seems we need to debug/profile a bit to figure out exactly why.
The network_los engine/setup, and potentially the MAZ to MAZ costs are what's different so it's likely something in there. Without actually profiling, we're wondering about the bellow note in the quick_loc_df() called by get_mazpairs()....
activitysim/activitysim/core/util.py
Line 233 in a57efa1
activitysim/activitysim/core/los.py
Line 517 in a57efa1
bstabler commentedon Jun 24, 2021
We're going to investigate some more to confirm the hypothesis and to hopefully develop an improvement. We'll look at the DaySim and CTRAMP code for ideas as well. Testing with the full scale setup will be important. This would benefit both 2 and 3 zone setups.
for #431
bstabler commentedon Jun 28, 2021
@esanchez01- I'm in the process of testing this improvement on a full scale version of your model and hoping you can test it as well?
esanchez01 commentedon Jun 29, 2021
@bstabler, I manually made the change you mentioned to v0.9.9.1, as this is the version my previous tests ran on (no chunking update), and reran the full SANDAG 2-Zone PSRC-based example. There are some noticeable run time improvements when compared to the equivalent 2-Zone run I shared the log files for:
I will rerun the 3-Zone example with this change as well.
bstabler commentedon Jun 29, 2021
I like these improvements. Tour mode choice is still quite a bit slower than the 1 zone example, which, if I recall, was a couple minutes, so I'll keep investigating. @esanchez01 - can you post the 1 zone example runtimes by submodel for reference?
esanchez01 commentedon Jun 29, 2021
Here are the results of the 3-Zone rerun:
esanchez01 commentedon Jun 29, 2021
Complete run time list for an optimized 1-Zone run:
7 remaining items
bstabler commentedon Sep 3, 2021
Below are updated runtimes for 100% samples of the in-development example_sandag 1, 2, and 3 zone system setups on my Intel Xeon 2.1 GHz 32 core 512 GB RAM server with 28 num processors configured and chunking disabled. The runtimes for the 3 configurations are similar, which suggests the differences we saw earlier in the runtimes between the setups was likely due to chunking due to the use of less RAM. Long term we can certainly improve chunking, but simply using more RAM for this specific model configuration goes a long way to a more performant setup.
esanchez01 commentedon Sep 3, 2021
@bstabler, thank you for sharing. If possible, can you add the total (sub model + non sub model) run times? Also, it would be good to know what the RAM usage of each run was to get a sense of how much may be needed and how the different zone systems compare.
wusun2 commentedon Sep 9, 2021
@bstabler, these are interesting results. I am surprised to see the location choices of 1-zone performed worse than the 2-zone system, some are even worse than the 3-zone system. For example, the trip location time for 1, 2, and 3 zones are 40.5, 37.4, and 35.7 minutes (this is the most time consuming step also, roughly 30-40% of total runtime). This is counter-intuitive, because 1-zone location choice operates at TAZ rather than the MAZ level in 2 and 3 zone systems, shouldn't it be faster or at least not worse than the 2/3 zone system?
joecastiglione commentedon Sep 9, 2021
I was also a bit confused by these results. While super-encouraging from a 3-zone perspective, it is my understanding that there are simply fewer calculations in the 1- and 2-zone systems due to the absence of TVPB, so why is there no performance advantage to these simple setups? Is this a matter of more time having been spent optimizing the 3-zone? If so, I think we need to dig into the 1- and 2- zone performance.
wusun2 commentedon Sep 9, 2021
@joecastiglione , I agree we need to dig into the 1 and 2 zone performance more. Especially, some of us will use the 1 or 2 zone system as their region's production model.
esanchez01 commentedon Sep 19, 2021
@bstabler, I am attaching a workbook that breaks down the 1, 2, and 3 zone run times (ran on a SANDAG server) and contains notable logging for certain sub models that appeared to have performance issues. This includes what was covered during the 9/14 technical call.
zone_run_time_analysis.xlsx
reduce and reuse maz to maz network impedance calls to speed up runtime
bstabler commentedon Sep 29, 2021
Thanks @esanchez01. I looked into the reported runtime differences for your 1, 2, and 3 zone setups and have a couple of findings and some improvements (in the commit above) to the 2 zone setup to reduce runtimes.
SANDAG server
My server
I believe some of the increase in runtimes for trip mode choice from the 1 zone to 2 zone setup is due to memory and chunking since my run without chunking has a more reasonable increase in runtime from 1 zone to 2 zone to 3 zone setup. However, I do not expect my chunking disabled tour mode choice setup to go from 1 min for a 1 zone setup to 14 min for a 2 zone setup so I looked into this.
od_skims['DISTWALK'], is largely responsible for the slow down. Theexample_psrcexpressions have several repeat calls to get the data, which gets the data from a pandas DataFrame keyed by an OMAZ-DMAZ compound key. To try and reduce runtimes, I gradually simplified the expressions for tour_mode_choice as follows:od_skims['DISTWALK']call for walk, and also for bike and general distance, in the pre-processor and then use the result in the expression filesod_skims['DIST']from the MAZ to MAZ tables so there isn't a blended skimmed DIST + a MAZ to MAZ DIST (... i.e. just use the skimmed value)I then ran the complete 2 zone example again, using the same setup as mentioned in an earlier comment but with these revisions, and below are the revised runtimes. The submodels run about 20% faster, and are actually faster than the 1 zone system setup. I did not revise the 3 zone expression files, but they could also be improved, as well as probably the 1 zone system files as well, but they use numpy array indexing instead of pandas DataFrame indexing.
We should think about ways to improve MAZ to MAZ impedance storage and retrieval speed to speed this up. Possible ideas are by using numpy arrays, a different smarter index/search method, and/or caching.
@esanchez01 - can you test on your end and do a more thorough expression file update of your various setups? You may also want to update the full scale maz_to_maz_bike.csv on the activitysim resources repo by dropping DIST if you do not want to blend general distance as well.
P.S. I also noticed that the transit skims in the expressions have
/100in them, which is a hold over from the MTC skims, which are Cube TRANPLAN skims, which are scaled ints and therefore need to be/100before use. These should be dropped if not needed.esanchez01 commentedon Oct 4, 2021
@bstabler, I included your changes in the SANDAG 2 zone setup and reran on our server. Both the sub models and the overall run ran about 25% faster -- down to ~140 minutes and ~180 minutes, respectively. Tour mode choice saw the most improvement (~19 minutes to 3 minutes) and improvements were seen across other sub models, consistent with what was observed from your test. I am attaching the results here: maz_maz_update_runtime.xlsx
I will look further into the expression files and make adjustments.
NOTE: The 1, 2, and 3 zone run times I previously shared in the workbook are all of MTC based setups, whereas the SANDAG 2 zone setup is PSRC based (as you tested). I did this to allow a more apples to apples comparison of the different zone systems. I believe this to be the reason why there is a big difference in the 2 zone trip mode choice run time between your test and mine, as you noted. For the reruns, I used the PSRC based setup and observed similar trip mode choice run times as in your tests.
bstabler commentedon Oct 7, 2021
@esanchez01 - did you fix the /100 issue as well? If so, can you PR your improvements to the fewer-maz-maz-data-calls-for-sandag-two-zone branch?
bstabler commentedon Oct 7, 2021
This issue will be closed once we update the examples with the reduced and reused expressions
bstabler commentedon Oct 7, 2021
And we'll create a new issue to look into the trip_destination model since it remains the slowest submodel.
esanchez01 commentedon Oct 12, 2021
@bstabler, I have not yet fixed the /100 issue but will get to it, as well as updating additional expressions, soon. Since the SANDAG examples pull MTC and PSRC expression files, I won't be making updates directly to these files but probably just storing the updated expression files in the SANDAG example directories.
bstabler commentedon Oct 13, 2021
We're going to create a new issue for the /100 issue.