Generator expression behavior changed in 3.13.4 - it does not throw exception anymore

Bug report

Bug description:

In 3.13.3 and before

>>> bool(_ for item in False)
Traceback (most recent call last):
  File "<python-input-14>", line 1, in <module>
    bool(_ for item in False)
                       ^^^^^
TypeError: 'bool' object is not iterable
>>> (_ for item in False)
Traceback (most recent call last):
  File "<python-input-15>", line 1, in <module>
    (_ for item in False)
                   ^^^^^
TypeError: 'bool' object is not iterable

Now in 3.13.4

>>> bool(_ for item in False)
True
>>> (_ for item in False)
<generator object <genexpr> at 0xffffa9c02200>

This is very fundamental change. Is it expected?

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

Most probably related to fix #132384 of #127682

We should restore an old behavior which was intentional: GET_ITER should be called before passing argument to generator expression, not after.

Sounds like the change should be reverted. We're doing an expedited 3.13.5 for other reasons, and the rollback can go in there.

We need two rollbacks -- one for the change that removed GET_ITER before calling the generator expression, and other for the change that added GET_ITER in a generator expression (that may not be trivial).

What is the bug here?

bool(_ for item in False) should be True. All generators are truthy.

If the intent was to write any(_ for item in False) or all(_ for item in False) then the expected TypeError is raised.

Consider the equivalent generator function:

def gen():
     for item in False:
        yield _

creating the generator does not raise an exception, but iterating over it does.

>>> bool(gen())
True

Iterating over an iterable is a two step process.:

Convert the iterable into an iterator (using the GET_ITER instruction)
Repeatedly get the next item from the iterator (using the FOR_ITER instruction)

For for loops and generators, the FOR_ITER and GET_ITER are together and everything works fine.

Historically, for reasons I am unaware of, in generator expressions the conversion from iterable to iterator occurred when the generator was created, not when used. This left a gap between conversion and the (unchecked) iteration which could mean that iteration could crash.
#125038.

We attempted to fix that by putting a GET_ITER in the generator expression as well, but that breaks iterators without __iter__ methods, or where the __iter__ has side effects. #127682
It also introduced a crash, but in SWIG generated code: rpm-software-management/libdnf#1682

So we must do one of the following:

Allow a crash. I'm strongly opposed to this option
Break working iterators that do not implement __iter__, which have always worked.
Allow the creation of some generator expressions that was previously not allowed, but will still fail as expected when iterated over.

Option 3 is what we now have and seems to me to be easily the best option. It breaks no working code, and produces the same exceptions for faulty code in a way that is largely indistinguishable from the behavior in 3.12.

That generator expressions and generator functions behaved differently could also be considered a bug.

See also #127682 (comment)

It should be a TypeError, because False is not iterable. It's been that way since the beginning, and I believe it was intentional, and that the existing code depends on it. It helps to detect bugs. Even if we want to change the behavior, it should be preceded by a general discussion, and we can only do that in a new Python version, not in a bugfix release.

We just have to return the old behavior. The crash you mentioned is only possible after low level hacking. The behavior change is visible to common users.

@markshannon

@markshannon I do not insist that it must be fixed, but I thought it could be considered for a major release. I encountered this issue in legacy code, and there’s no question it should be refactored to avoid this. The exception is used there to exit recursion while parsing nested dict-like structures, if you need a real-world scenario.
Casting to bool is not important, a more generic example would be something like x = (_ for _ in None). If the examples in the bug description seem confusing, I can replace them.

@Roman513

@Roman513 the problem is not that your examples are hard to use, but they appear contrived. What was the original issue you found (in actual code)?

@serhiy-storchaka

@serhiy-storchaka What about the other issues I listed, do they do not deserve to be fixed? Why this issue and not those?

One other thing. what about generator expressions with multiple iterables e.g. ((i,j) for i in range(3) for j in range(n))?

Python 3.12 and earlier were already inconsistent here:

Python 3.12
>>> ((i,j) for i in range(3) for j in False)
<generator object <genexpr> at 0x7447eaba43c0>
>>> ((i,j) for i in False for j in range(n))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'bool' object is not iterable

@Yhg1s

Overall, I believe the current behavior is the best. It is the robust, efficient and is consistent with generator functions. So I definitely want to keep it for 3.15+.

Regarding 3.13 and 3.14 there is one more thing we could attempt:

Move GET_ITER back to the creation of the generator expression for 3.13 and 3.14
Add a tp_iternext function to object so that FOR_ITER never crashes, just throws the expected exception.

The downside of this is that any C extensions doing Py_TYPE(op)->tp_iternext != NULL to determine if an object is an iterator will be broken. Hyrum's law states that someone will be doing this 🙁

@Yhg1s @hugovk It's your call.

https://xkcd.com/1172/

[3.14] gh-135171: Update documentation for the generator expression (G…

[3.13] gh-135171: Update documentation for the generator expression (G…

pythongh-135171: Update documentation for the generator expression (p…

[3.14] GH-135171: Revert async generator expressions behavior (#135352)

@serhiy-storchaka

Should we close this issue? PRs for 3.13 and 3.14 are being merged.

But for Python 3.15+ we have to decide what to do.
I'll create a d.p.o topic to discuss this decision (see #135352 (comment)).

CC @serhiy-storchaka @Yhg1s @hugovk

Yes, let's. Thanks all!

If close this issue, we should open a new issue for main. It is better to solve it earlier, because the more time will pass, the more changes we will need to roll back, and it may conflict with other code.

pythongh-135171: Update documentation for the generator expression (p…

I've started related d.p.o. thread here: https://discuss.python.org/t/generator-expressions-behavior-with-non-iterable-objects/96329.

pythongh-135171: Update documentation for the generator expression (p…

[3.13] pythonGH-135171: Roll back all fixes for pythonGH-127682 as th…

pythongh-135171: Update documentation for the generator expression (p…

Navigation Menu

Repository navigation

Generator expression behavior changed in 3.13.4 - it does not throw exception anymore #135171

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

32 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Generator expression behavior changed in 3.13.4 - it does not throw exception anymore #135171

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

Roman513 commented on Jun 5, 2025

serhiy-storchaka commented on Jun 5, 2025

Yhg1s commented on Jun 5, 2025

serhiy-storchaka commented on Jun 5, 2025

markshannon commented on Jun 5, 2025

serhiy-storchaka commented on Jun 5, 2025

Roman513 commented on Jun 5, 2025

markshannon commented on Jun 6, 2025

markshannon commented on Jun 6, 2025

markshannon commented on Jun 6, 2025

markshannon commented on Jun 6, 2025

32 remaining items

efimov-mikhail commented on Jun 16, 2025

hugovk commented on Jun 17, 2025

serhiy-storchaka commented on Jun 17, 2025

efimov-mikhail commented on Jun 22, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions