Repository navigation
[3.11.0rc1] Undocumented behavior: Large refcounts on small objects #96730
Description
Activity
Assuming it’s not a bug, I’m not sure it needs to be documented. There are no guarantees aside from >0. Does this cause a problem? I’m genuinely curious.
The large initial reference count is
_PyObject_IMMORTAL_REFCNT(999999999), which is set by_PyObject_IMMORTAL_INITand_PyVarObject_IMMORTAL_INIT, and is used by_PyUnicode_ASCII_BASE_INIT,_PyBytes_SIMPLE_INIT,_PyLong_DIGIT_INIT, etc. AFAICT, it's an ad hoc initial value, and the reference count can still be incremented and decremented normally. PEP 683 (Immortal Objects, Using a Fixed Refcount) is tracked for Python 3.12, after which the reference count on such immortal objects will be a constant value.I don't think there's anything to document at this point. Also,
sys.getrefcount()should continue to return the current value of the reference count as is instead of trying to 'fix' this.I agree we don't want
sys.getrefcountto do anything clever, but would there be value in documenting this in sys.getrefcount since this might be the first place Python programmers might encounter immortal references? (Also Cc @ericsnowcurrently).For some people who work with native extensions or similar, who may be shocked upon suddenly seeing a massive refcount, having this at least documented somewhere in the docs would be nice to have, that way you don't have to dig through all peps to figure out this is intended behavior and not a bug in your code.
There's no need to specify all objects that behave like this, but just saying that there are some with behavior like this would be incredibly helpful.With myself as an example: It took me 2-3 days of debugging and digging through documentation until someone pointed me to the relevant PEP, and I wouldn't have thought to look there on my own.
It should be mentioned in what’s new in 3.11 as well. That’s where people look when something breaks while upgrading to a new version.
Reacted by Martmists, Eric Snow, Eryk Sun, Scott Maxwell and Petr ViktorinFor some people who work with native extensions or similar, who may be shocked upon suddenly seeing a massive refcount,
I'd like to hear more about the expectations extension authors/maintainers might have about refcounts. In particular, what is shocking about the large refcount values? Are there particular objects that stood out? Finally, how would you describe the expectations some have about refcounts, as you've alluded?
FWIW, we've been aware that large refcounts may be confusing for some extension maintainers (and changes in refcount semantics, a la PEP 683, may lead to some unit test failures in the community). That's the motivation for the doc change in the "Immortal Objects" PEP (683):
- context: https://peps.python.org/pep-0683/#public-refcount-details
- proposed change: https://peps.python.org/pep-0683/#documentation
Basically, we're going to clearly document what should be expected (a very limited expectation, as @ericvsmith suggested), whereas right now it isn't so clear.
So how does that apply here? I think we should make those proposed doc changes (from PEP 683), regardless of the PEP, and backport them to 3.11.
having this at least documented somewhere in the docs would be nice to have, that way you don't have to dig through all peps to figure out this is intended behavior and not a bug in your code. There's no need to specify all objects that behave like this, but just saying that there are some with behavior like this would be incredibly helpful.
+1
With myself as an example: It took me 2-3 days of debugging and digging through documentation until someone pointed me to the relevant PEP, and I wouldn't have thought to look there on my own.
Sorry about that!
It should be mentioned in what’s new in 3.11 as well. That’s where people look when something breaks while upgrading to a new version.
@kumaraditya303, would you like to take a stab at this? If not, I'll add it to my TODO.
My opinion as an ordinary user who occasionally looks at reference counts to check something for myself or explain something to others:
There are no guarantees aside from >0.
There's a kind of guarantee in a name. Something called "X" should be X. If it isn't X, it shouldn't be called "X". There's leeway if "X" is ambiguous, but I'd say "reference count" is unambiguously "The number of references to an object", as the glossary entry puts it. So if it's not just that, then it's a misnomer.
You can define it to mean something different (e.g., including an exception for immortality), but that definition then should be easily visible to people looking at them, and documentation about them shouldn't be wrong. If the
sys.getrefcountdocumentation just says "reference count" but actually returns what it did above, I consider that wrong. And if I saw those large numbers (or reference counts not changing despite deleting/adding references), I'd be confused and investigate and likewise waste time if it's not easily found.My suggestion: I assume this new meaning of "reference count" is or will be documented somewhere, perhaps in the devguide? Add a brief sentence about the deviation from the ordinarily understood meaning to the glossary entry with a link to that "somewhere". And in the
sys.getrefcountdocumentation, make the term "reference count" a link to the glossary entry.There's a kind of guarantee in a name. Something called "X" should be X. If it isn't X, it shouldn't be called "X".
Unfortunately, calling it something more precise isn't really an option at this point.
I assume this new meaning of "reference count" is or will be documented somewhere
See the proposed documentation in PEP 683 (links in #96730 (comment)). We'd love your feedback if you think those proposed changes are insufficient.
- Reacted by Eddie Elizondo
Yes, this should be documented, and we have plans for that, though it probably got a bit derailed with PEP683 taking much longer to be accepted.
@ericsnowcurrently Given that we already have some instances of these larger values, we might want to be a bit more explicit about it and incorporate pieces of 683 as part of the documentation today?
Given that we already have some instances of these larger values, we might want to be a bit more explicit about it and incorporate pieces of 683 as part of the documentation today?
Yeah, I've been thinking along those lines for a while, but hadn't gotten around to it yet. Certainly anyone could pick this up in the meantime. PEP 683 outlines what needs to be changed.
Some objects seem to have unreasonably large refcounts, and this is not documented in the docs nor on the changelog.
3.10.6:
3.11.0rc1: