You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Should we freeze activitysim at pandas 1.2, keep it flexible with the hope that this issue will resolve itself, or update activitysim? In the past when this has happened, the answer typically depends on the level of effort for the fix. @jpn-- do you have an idea of the LOE?
Total level of effort in the end was just a few hours. In addition to the line noted above, pandas now complains with a performance warning about memory fragmentation when building certain dataframes. The default settings for ActivitySim promoted the warning to an error. I tried to fix the problem by defragmenting the relevant data frames when the warning occurs, but it turned out that's not so easy as the pandas 1.3 block manager appears to allocate separate blocks for every bool column even after running a copy command. So I just instructed ActivitySim not to promote this warning to an error. Longer term, we may want to study whether fixing this and getting better block consolidation might lead to improved performance.
All the fixes needed have been incorporated into #440 which now passes tests with pandas 1.3, and so we'll be good on this front when that PR is eventually merged.
Activity
bstabler commentedon Jul 8, 2021
Should we freeze activitysim at pandas 1.2, keep it flexible with the hope that this issue will resolve itself, or update activitysim? In the past when this has happened, the answer typically depends on the level of effort for the fix. @jpn-- do you have an idea of the LOE?
jpn-- commentedon Jul 9, 2021
Total level of effort in the end was just a few hours. In addition to the line noted above, pandas now complains with a performance warning about memory fragmentation when building certain dataframes. The default settings for ActivitySim promoted the warning to an error. I tried to fix the problem by defragmenting the relevant data frames when the warning occurs, but it turned out that's not so easy as the pandas 1.3 block manager appears to allocate separate blocks for every bool column even after running a copy command. So I just instructed ActivitySim not to promote this warning to an error. Longer term, we may want to study whether fixing this and getting better block consolidation might lead to improved performance.
All the fixes needed have been incorporated into #440 which now passes tests with pandas 1.3, and so we'll be good on this front when that PR is eventually merged.