Skip to content

Share global string identifiers in deepfreeze #90868

Closed
Closed
Share global string identifiers in deepfreeze#90868
Labels
3.11only security fixesperformancePerformance or resource usage

Description

@kumaraditya303
kumaraditya303
opened on Feb 10, 2022
BPO 46712
Nosy @gvanrossum, @ericsnowcurrently, @sweeneyde, @kumaraditya303
PRs
  • bpo-46712: Share global string identifiers in deepfreeze #31261
  • bpo-46430: fix error-handling in _Py_Deepfreeze_Init #31596
  • bpo-46712: Let generate_global_objects.py Run on Earlier Python Versions #31637
  • bpo-46712: Do not Regen Deep-Frozen Modules before Generating Global Objects #32061
  • bpo-46712: share more global strings in deepfreeze #32152
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2022-02-25.18:08:29.439>
created_at = <Date 2022-02-10.14:18:03.087>
labels = ['3.11', 'performance']
title = 'Share global string identifiers in deepfreeze'
updated_at = <Date 2022-03-28.09:14:38.402>
user = 'https://github.com/kumaraditya303'

    bugs.python.org fields:

    activity = <Date 2022-03-28.09:14:38.402>
actor = 'kumaraditya'
assignee = 'none'
closed = True
closed_date = <Date 2022-02-25.18:08:29.439>
closer = 'gvanrossum'
components = []
creation = <Date 2022-02-10.14:18:03.087>
creator = 'kumaraditya'
dependencies = []
files = []
hgrepos = []
issue_num = 46712
keywords = ['patch']
message_count = 7.0
messages = ['413003', '413005', '414028', '414295', '415665', '415811', '415882']
nosy_count = 4.0
nosy_names = ['gvanrossum', 'eric.snow', 'Dennis Sweeney', 'kumaraditya']
pr_nums = ['31261', '31596', '31637', '32061', '32152']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue46712'
versions = ['Python 3.11']

    Activity

    1. kumaraditya303 commented on Feb 10, 2022

      @kumaraditya303
      kumaraditya303
      on Feb 10, 2022
      ContributorAuthor

      Since bpo-46541, the global strings are statically allocated so they can now be referenced by deep-frozen modules just like any other singleton. Sharing identifiers with deepfreeze will reduce the duplicated strings hence it would save space.

      See faster-cpython/ideas#218
      See faster-cpython/ideas#230

    2. kumaraditya303
      added
      3.11only security fixes
      on Feb 10, 2022

    3. kumaraditya303 commented on Feb 10, 2022

      @kumaraditya303
      kumaraditya303
      on Feb 10, 2022
      ContributorAuthor

      I have refactored generate_global_objects.py, and now instead of hard-coding every identifier manually, it now scans *.c files extracts the identifiers used in it and then generate the header file. This has multiple advantages:

      • No need to manually add identifiers, as soon as it is used in a c file it is added to the global identifiers struct.
      • It simplifies the codegen a lot.
      • Remove the need of special casing certain file for checking now it is just a set of identifiers and auto removes unused global strings.

    4. gvanrossum commented on Feb 25, 2022

      @gvanrossum
      gvanrossum
      on Feb 25, 2022
      Member

      New changeset eb002db by Kumar Aditya in branch 'main':
      bpo-46712: Share global string identifiers in deepfreeze (GH-31261)
      eb002db

    5. gvanrossum
      closed this as completedon Feb 25, 2022
    6. gvanrossum
      added
      performancePerformance or resource usage
      on Feb 25, 2022
    7. gvanrossum
      closed this as completedon Feb 25, 2022
    8. gvanrossum
      added
      performancePerformance or resource usage
      on Feb 25, 2022

    9. ericsnowcurrently commented on Mar 1, 2022

      @ericsnowcurrently
      ericsnowcurrently
      on Mar 1, 2022
      Member

      New changeset 21099fc by Eric Snow in branch 'main':
      bpo-46712: Let generate_global_objects.py Run on Earlier Python Versions (gh-31637)
      21099fc

    10. sweeneyde commented on Mar 21, 2022

      @sweeneyde
      sweeneyde
      on Mar 21, 2022
      Member

      In bpo-47067, there was concern about the addition of the makefile target from PR 31637:

      regen-global-objects: regen-deepfreeze

      After a new &_Py_ID(__orig_class__) is added to Objects/genericaliasobject.c, running make regen-global-objects starts

      gcc -pthread -c [snipped] -DPy_BUILD_CORE -o [Objects/genericaliasobject.o](https://github.com/python/cpython/blob/main/Objects/genericaliasobject.o) [Objects/genericaliasobject.c](https://github.com/python/cpython/blob/main/Objects/genericaliasobject.c)

      which fails with a compilation error because that identifier is not yet defined. Is there a good way to convince make to regenerate the global objects without this sort of circular dependency? Am I missing a step?

    11. ericsnowcurrently commented on Mar 22, 2022

      @ericsnowcurrently
      ericsnowcurrently
      on Mar 22, 2022
      Member

      After a new &_Py_ID(__orig_class__) is added to Objects/genericaliasobject.c, running make regen-global-objects starts

      gcc -pthread -c [snipped] -DPy_BUILD_CORE -o [Objects/genericaliasobject.o](https://github.com/python/cpython/blob/main/Objects/genericaliasobject.o) [Objects/genericaliasobject.c](https://github.com/python/cpython/blob/main/Objects/genericaliasobject.c)

      which fails with a compilation error because that identifier is not yet defined. Is there a good way to convince make to regenerate the global objects without this sort of circular dependency? Am I missing a step?

      I'm looking into this. A temporary workaround is to run Tools/scripts/generate-global-objects.py directly.

    12. ericsnowcurrently commented on Mar 23, 2022

      @ericsnowcurrently
      ericsnowcurrently
      on Mar 23, 2022
      Member

      New changeset febf54b by Eric Snow in branch 'main':
      bpo-46712: Do not Regen Deep-Frozen Modules before Generating Global Objects (gh-32061)
      febf54b

    13. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    14. ericsnowcurrently
      added a commit that references this issue on Apr 19, 2022

      bpo-46712: share more global strings in deepfreeze (gh-32152)

      ab0d35d
    15. ericsnowcurrently
      added a commit that references this issue on Nov 8, 2022

      gh-90868: Adjust the Generated Objects (gh-99223)

      52f91c6
    16. vstinner
      added a commit that references this issue on Nov 9, 2022

      gh-90868: Add _PyStaticObject_CheckRefcnt() function (#99261)

      0124b5d
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.11only security fixesperformancePerformance or resource usage

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions