Include Pdb._exec_in_closure() as code-path in exec() itself, or make it a public utility otherwise? #136280
Description
Feature or enhancement
Proposal:
When IPython is embedded into a non-IPython interpreter, it evaluates code using frame locals (FrameLocalsProxy) in the same way as pdb does in Python stdlib.
This does not work with objects which require access to the closure scope:
- comprehensions in Python <3.13 (fixed for 3.13+ by inlining comprehensions introduced in PEP 709)
- generators
This bug affected pdb module too (#65360) but it was fixed/worked around by:
For example, the following does not work:
import sys
call_frame = sys._getframe(0).f_back
local_ns = call_frame.f_locals
exec('x = 1; sum(x * i for i in range(5))', locals=local_ns)Produces:
Traceback (most recent call last):
File "<python-input-0>", line 4, in <module>
exec('x = 1; sum(x * i for i in range(5))', locals=local_ns)
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<string>", line 1, in <module>
File "<string>", line 1, in <genexpr>
NameError: name 'x' is not definedI know that FrameLocalsProxy has a few undocumented limitations, I am not sure if those are relevant here - I am linking the relevant issue just in case if that helps maintainers to refresh the context:
I am coming here from IPython, to explore if the solution belongs in IPython, or in CPython repo:
- a) could
exechandleFrameLocalsProxyspecially in some way? - b) as one option, could the same code as added for
pdbin gh-83151: Make closure work on pdb #111094 (_exec_in_closuremethod) be included in the defaultexecimplementation for whenFrameLocalsProxyis passed? - c) if adding
_exec_in_closureintoexecproper does not make sense, is it a good idea to explore moving_exec_in_closureout ofpdband making it public?
The fix/workaround from #111094 is not perfect yet, though I think most of the problems could be resolved with a little bit more work. These issues were reported in:
If _exec_in_closure were exposed as public, there would be a bigger incentive for community to fix issues outlined in #126958, basically centralising the effort to make that work well.
On the other hand, if there are plans to inline generators in the future (I saw that PEP 709 leaved that as a possibility), maybe _exec_in_closure would no longer be necessary in the first place.
As an alternative to reusing _exec_in_closure, IPython could use locals() call to populate locals_ns which get passed down to exec via locals argument when in embed mode; I am somewhat apprehensive to make this change as I worry it might break downstream code.
However, if you all advise that _exec_in_closure shall remain a private pdb utility, this would likely tilt the the trade-off towards using the locals() call, as otherwise IPython would need to maintain its own version of _exec_in_closure adding to maintenance cost on already over-stretched team.
Has this already been discussed elsewhere?
This is a minor feature, which does not need previous discussion elsewhere
Links to previous discussion of this feature:
Not directly, but this is the third oldest unresolved issue in IPython dating back over 15 years:
- Global variables undefined in interactive use of embedded ipython shell ipython/ipython#62
- IPythonShellEmbed fails to recognize local variables ipython/ipython#136
- Variable not recognized in nested or filtered list comprehension under IPython.embed() ipython/ipython#12199
CC @gaogaotiantian if I may, as author of #111094 and assignee on #126958
Activity
gaogaotiantian commentedon Jul 4, 2025
First of all, could you make the example code show your actual ideas next time? Your current code will just raise an exception because there's no
f_back. A helpful reproducible example is something that we can copy & paste then execute.To answer your questions. I don't believe this has anything to do with
FrameLocalsProxy. It will have the exact same issue if you passlocals(). It's the code you compile - the generator believesxis a global variable but it does not exist in the global name space.So your question a) is not valid - it's not the
FrameLocalsProxythat caused the problem.Can we use the pdb's closure implementation in exec? No we can't do that. That would break backwards compatibility. Also it's very implicit. The fact
exec()takes aclosureargument shows that you should deal with it in your debugger - it's an advanced function and we want to keep it relatively straightforward. You have the option to do advanced things, but the default behavior does not assume the code was executed in a potential closure environment.And we probably won't expose the internal function of pdb either - as you said, it's not perfect. It's kind of a workaround and we don't want to make a promise to it.
Actually, making it to work is very difficult. I must disagree on your thought that "most of the problems could be resolved with a little bit more work", because I've spent plenty of time into it and did not find any good solution without hacking heavily on the code.
The problem will not be fully solved even if generators are inlined - the user can always create a function scope in their code.
I understand that I've talked plenty but did not give any helpful directions, sorry about that. I did put plenty of thoughts into this and if you have a better idea of how to make it work better, I'm happy to look at the solution. I did look at how the original author of that issue solved it in their project - it's a lot of code and full of black magics. It's impossible for us to maintain that in stdlib.
krassowski commentedon Jul 4, 2025
Apologies, you are probably right. This is what I see
I do not understand why you say there is no
f_back:I know that. This is what I tried to achieve, I understand this is not the example you would expect. Apologies for annoying you but it is hard for me to isolate the thing from all of the IPython codebase.
I don't understand why you say that.
gaogaotiantian commentedon Jul 4, 2025
Your
locals()andf_localsare not equivalent. Yourf_localsis not the local scope of your interpreter, it's thepyreplobject. So you are running everything in the interpreter and that makes it a bit complicated.If you do
instead of
You'll find that everything just works - that's the equivalent form as your
locals()solution. It's not aboutFrameLocalsProxy, you are just using different frames.Normally when we try to reproduce an example, we copy the lines in a file, save it and run it with python. If you do that you'll realize that
call_frame = sys._getframe(0).f_backgivesNonebecausesys._getframe(0)is already at the bottom. It does not raise an exception because of how current Python shell works.krassowski commentedon Jul 4, 2025
I know. I think it would make more sense to you in context of ipython/ipython#62
Yes, I know. I said I am coming from IPython :)
Fine, indeed my reproducer assumed that one would be running it in a Python shell. It still stands that this is a real issue, and this is the smallest reproducer I was able to create.
Anyways, since it seems you are not very enthusiastic about adding a code path to
execwhich would handle generators well (whether that would be triggered byFrameLocalsProxytype or a new argument or whatever - you know better than me), feel free to close this issue.gaogaotiantian commentedon Jul 4, 2025
This is a real issue, it just has nothing to do with
FrameLocalsProxy. The problem is that when we compile a piece of code, we have no way to know that it is supposed to be in another function (closured). We can't change that behavior because it's not backward compatible - it's not that we don't want to make it better, we can't make it better without breaking a lot of the old code.The way
pdbdeals with it is very hacky and should only live in a debugger. You need to change a lot of things to makeexecreally work in such scenario - I think you probably need to change the compiler.