Repository navigation
Deprecate pickle support for itertools #101588
Description
Activity
Hi @rhettinger, would you like a hand on this?
@JosephSBoyle I am already working on a patch for this issue. Please find a different issue to work on.
Reacted by JosephSBoyleThanks for working on this. When the PR is ready for review, please assign to me :-)
@JosephSBoyle Do you pick up this task?
Reacted by JosephSBoyleI still plan to create a pull request for this issue (within the next two weeks, hopefully).
Reacted by JosephSBoyleHi @rhettinger, I'd be happy to work on this.
@ZackerySpytz, perhaps we could work together somehow? Up to you - I appreciate that you were working on this before I volunteered to help.
I was about to use the pickling feature on my end, and I wouldn't have noticed this had I not checked in the doc, noticed it wasn't there and propose it to be in the issue above.
So, I wouldn't be surprised if others than me use the feature as well, and while I fully understand and support the rationale for deprecating, I think making it follow a deprecation cycle wouldn't hurt.
In my experience and opinion, iterators in python generally fail spectacularly when you try to pickle them, so having the itertools one not fail may have felt for many to be a low-key guarantee on that being a supported feature, even if it was not explicitly documented (and even though it's not a very good practice).@JosephSBoyle You no longer need to wait. There is no particular reason for @ZackerySpytz to reserve this one. There are plenty of other open bugs.
Sure, @rhettinger. Where would you recommend starting on this?
Thomas, can this deprecation still go into 3.12? It deprecates an undocumented feature.
It may be undocumented but pickle use is so viral, it's very likely depended on without people even knowing :) Let's get the deprecation into beta 2 if we can (it's scheduled for next week), but given that it's not actually problematic and just something we don't want to support, I would suggest holding back on removing it until we have a clearer picture of what it would break.
- added a commit that references this issue
on May 26, 2023 I missed when this was deprecated, but I support this. It was inefficient, it leaked internal state, and it often not worked as expected for complex iterators like
chainortee.Should not this also be deprecated for builtins
zip,mapandfilter, and for iterators of builtin collections?Reacted by Gouvernathor, inoperable and Agriya KhetarpalA little late to the party, but exposing this surface makes for a relatively easy crash in Py3.13.1:
Python 3.13.1 (main, Dec 28 2024, 13:25:56) [GCC 13.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import itertools >>> tdo = itertools._tee_dataobject(object(), [1], None) >>> tob = itertools._tee(()) >>> tob.__setstate__((tdo, 0)) <python-input-3>:1: DeprecationWarning: Pickle, copy, and deepcopy support will be removed from itertools in Python 3.14. tob.__setstate__((tdo, 0)) >>> next(tob) 1 >>> next(tob) Segmentation fault
This seems to work on any non-iterable object in the first argument to the TDO constructor. I suppose it's because the TDO calls
PyIter_Nexton this object without ever having done aPyIter_Check.Arguably that's a separate bug, but removing this support should probably see the interface to
_tee_dataobjectfrom user code go away. TDOs seem to be a fragile internal detail in any case.Reacted by Enzan umo and Joshua-Luke Bevan
Pickle support was long ago added to some itertools. It was done mostly to support an atypical use case for a single company. It was implemented in a very inefficient manner, essentially replaying iteration from the beginning to the mid-stream state where it was frozen. The implementation was of low quality and had many bugs. Also, it was not a documented or advertised feature. Newer itertools don't support pickling and no one has noticed or cared. The popular third-party package
more-itertoolsis implemented with generators which do not have pickle support — again, none of their users seems to have noticed or cared.IMO, this is just cruft that has made maintenance more difficult and we should get rid of it. As an undocumented feature, we could just remove it directly. But to be on the safe side, we can go through a deprecation cycle.
Linked PRs