Repository navigation
[subinterpreters] Don't share Python objects between interpreters #84713
Description
| BPO | 40533 |
|---|---|
| Nosy | @vstinner, @ericsnowcurrently, @shihai1991 |
| PRs |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2020-05-06.15:11:35.602>
labels = ['expert-subinterpreters', '3.9']
title = "[subinterpreters] Don't share Python objects between interpreters"
updated_at = <Date 2022-01-13.17:32:56.034>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2022-01-13.17:32:56.034>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Subinterpreters']
creation = <Date 2020-05-06.15:11:35.602>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 40533
keywords = ['patch']
message_count = 8.0
messages = ['368262', '368274', '368276', '368279', '380105', '388966', '408606', '410499']
nosy_count = 3.0
nosy_names = ['vstinner', 'eric.snow', 'shihai1991']
pr_nums = ['19958', '19961']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue40533'
versions = ['Python 3.9']
Activity
vstinner commentedon May 6, 2020
To get one GIL per interpreter (bpo-40512), either PyObject.ob_refcnt member must become an atomic variable, or subinterpreters must not share any object.
Right now, subinterpreters share Python objects. For example, PyModule_Type is declared statically and so shared by all interpreters and so PyModule_Type.tp_mro tuple is accessed in parallel by multiple interpreters. If PyObject.ob_refcnt is not atomic, Py_INCREF() and Py_DECREF() are unsafe and tp_mro tuple can be destroyed whereas it is still used.
I propose to make PyObject.ob_refcnt atomic for now, when Python is built with EXPERIMENTAL_ISOLATED_SUBINTERPRETERS macro defined. It's a temporary workaround until subinterpreters stop sharing objects.
ericsnowcurrently commentedon May 6, 2020
Yep, before per-interpreter GIL is official we must get to the point where *no* PyObject objects are shared.
Making PyObject.ob_refcnt atomic until then (only as part of the experiment) should be fine.
vstinner commentedon May 6, 2020
I would like to add: "no PyObject objects are shared in the stdlib" ;-)
vstinner commentedon May 6, 2020
New changeset d8135e9 by Victor Stinner in branch 'master':
bpo-40533: Disable GC in subinterpreters (GH-19961)
d8135e9
[-]Subinterpreters: don't share Python objects between interpreters[/-][+][subinterpreters] Don't share Python objects between interpreters[/+][-]Subinterpreters: don't share Python objects between interpreters[/-][+][subinterpreters] Don't share Python objects between interpreters[/+]vstinner commentedon Oct 31, 2020
Another example of this issue are parallel Py_INCREF/Py_DECREF calls on the tp_bases member of a type.
vstinner commentedon Mar 17, 2021
The current plan is to not share any object between two interpreters, so this PR is not needed. I close my PR 19958 which marked PyObject.ob_refcnt as atomic.
vstinner commentedon Dec 15, 2021
Until all Python stdlib C extensions and all third-party C extensions will be modified to no longer use and define static types and will stop shared Python objects between two interpreters, we can no longer micro-optimize the comparison of two interned strings by only comparing their memory addresse. See bpo-46006.
vstinner commentedon Jan 13, 2022
Sharing objects between multiple interpreters can cause complex bugs leading to crashes: https://bugs.python.org/issue46070#msg410493 For this specific bug, I wrote a workaround (GH-30577).
vstinner commentedon Nov 3, 2022
I don't have the bandwidth to work on this issue, so I just close it.