Skip to content

Breaking changes in pandas 1.3 #449

Closed

Description

@jpn--

Pandas 1.3 was released today. There are changes that break ActivitySim.

At least this line no longer works:

template = template.where(~template.isnull(), template.index)

As an interim solution, use the latest flavor of pandas 1.2.

Activity

  1. self-assigned this
    on Jul 2, 2021
  2. bstabler commented on Jul 8, 2021

    @bstabler
    Contributor

    Should we freeze activitysim at pandas 1.2, keep it flexible with the hope that this issue will resolve itself, or update activitysim? In the past when this has happened, the answer typically depends on the level of effort for the fix. @jpn-- do you have an idea of the LOE?

  3. jpn-- commented on Jul 9, 2021

    @jpn--
    MemberAuthor

    Total level of effort in the end was just a few hours. In addition to the line noted above, pandas now complains with a performance warning about memory fragmentation when building certain dataframes. The default settings for ActivitySim promoted the warning to an error. I tried to fix the problem by defragmenting the relevant data frames when the warning occurs, but it turned out that's not so easy as the pandas 1.3 block manager appears to allocate separate blocks for every bool column even after running a copy command. So I just instructed ActivitySim not to promote this warning to an error. Longer term, we may want to study whether fixing this and getting better block consolidation might lead to improved performance.

    All the fixes needed have been incorporated into #440 which now passes tests with pandas 1.3, and so we'll be good on this front when that PR is eventually merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions