pdb cannot find source code for frozen stdlib modules

Bug report

Context: this issue came out of a discussion in #python on the Libera.chat IRC network, where a user wanted to peek into importlib._bootstrap with pdb while chasing a bug.

pdb is capable of stepping into function calls into frozen modules, but the list command cannot locate the source code necessary to display the source being stepped through.

# repro.py
import importlib._bootstrap

# some function call that we want to step into with pdb
importlib._bootstrap._resolve_name("os", ".", 1)

$ python3 -m pdb repro.py
> /home/snoopjedi/repro.py(2)<module>()
-> import importlib._bootstrap
(Pdb) n
> /home/snoopjedi/repro.py(5)<module>()
-> importlib._bootstrap._resolve_name("os", ".", 1)
(Pdb) s
--Call--
> <frozen importlib._bootstrap>(883)_resolve_name()
(Pdb) l
[EOF]

Note that executing source importlib._bootstrap from the frame that calls into this module does successfully locate the source, but this isn't very useful to a user of pdb.

I believe that bringing the frame's co_filename into agreement with the module's __file__ would fix this issue without changes to pdb (see #89815), but thought it would be good to track the issue with pdb in a separate ticket since that fix is more nuanced and I think I have an interim patch for the debugger.

Your environment

CPython versions tested on: 3.9.4, 3.11.0b3
Operating system and architecture: Ubuntu 20.04, x86_64

Linked PRs

gh-93696: Fixed the breakpoint display error for frozen modules #132862

Merge branch 'main' into pythongh-93696-pdb-find-frozen-source

Likely duplicate of #60115

@exarkun

Likely duplicate of #60115

Definitely related inasmuch as importlib._bootstrap is involved, @exarkun is the user I mentioned in my report. Note that the issue does happen with other frozen modules, though, so the scope of the problem is expanding in 3.11 because of #89183. Here's an example showing the same problem with os.path

# repro2.py
import os.path

os.path.basename("/foo/bar/baz.py")  # implemented in Python

With Python 3.10:

$ python3.10 -m pdb repro2.py 
> /home/snoopjedi/repos/cpython/repro2.py(2)<module>()
-> import os.path
(Pdb) n
> /home/snoopjedi/repos/cpython/repro2.py(4)<module>()
-> os.path.basename("/foo/bar/baz.py")  # implemented in Python
(Pdb) s
--Call--
> /home/snoopjedi/.pyenv/versions/3.10.0/lib/python3.10/posixpath.py(140)basename()
-> def basename(p):
(Pdb) l
135         return p[:0], p
136  
137  
138     # Return the tail (basename) part of a path, same as split(path)[1].
139  
140  -> def basename(p):
141         """Returns the final component of a pathname"""
142         p = os.fspath(p)
143         sep = _get_sep(p)
144         i = p.rfind(sep) + 1
145         return p[i:]

With Python 3.11:

$ python3.11 -m pdb repro2.py 
> /home/snoopjedi/repos/cpython/repro2.py(2)<module>()
-> import os.path
(Pdb) n
> /home/snoopjedi/repos/cpython/repro2.py(4)<module>()
-> os.path.basename("/foo/bar/baz.py")  # implemented in Python
(Pdb) s
--Call--
> <frozen posixpath>(140)basename()
(Pdb) l
[EOF]
(Pdb)

Merge branch 'main' into pythongh-93696-pdb-find-frozen-source

Pinging this issue per the devguide in the hopes of getting review on the PR, which has been open for a little while. I understand if things are too busy for review because of the release issues with 3.11, though!

Merge branch 'main' into pythongh-93696-pdb-find-frozen-source

This should mostly be resolved in 3.10. See issue gh-65935 (and PR gh-28656). We now set __file__ for frozen stdlib modules. Could you give it a try?

This should mostly be resolved in 3.10. See issue gh-65935 (and PR gh-28656). We now set __file__ for frozen stdlib modules. Could you give it a try?

I get the same behavior as in the initial report against the current tip of 3.10 (b60c3d2):

13:29 [snoopjedi@denton ~/repos/cpython (3.10)]
$ ./python -c "import sys; print(sys.version)"
3.10.5+ (remotes/upstream/3.10:b60c3d2f7e, Jul 21 2022, 12:34:09) [GCC 9.4.0]
13:29 [snoopjedi@denton ~/repos/cpython (3.10)]
$ ./python -m pdb repro.py 
> /home/snoopjedi/repos/cpython/repro.py(2)<module>()
-> import importlib._bootstrap
(Pdb) n
> /home/snoopjedi/repos/cpython/repro.py(5)<module>()
-> importlib._bootstrap._resolve_name("os", ".", 1)
(Pdb) s
--Call--
> <frozen importlib._bootstrap>(902)_resolve_name()
(Pdb) l
[EOF]

The __file__ attribute is reported correctly (I believe this has been the case since before I filed this ticket, actually), but pdb is relying on co_filename and #89815 still applies

Yeah, it may be worth adding the stopgap fix in the meantime.

@ericsnowcurrently

FYI I am looking into this. I'll probably make deepfreeze generate the correct co_filename but since this would have to be backported the stopgap solutions seems worth it. @ericsnowcurrently ?

Merge branch 'main' into pythongh-93696-pdb-find-frozen-source

gh-93696: Locate frozen module source with __file__ (#93697)

pythongh-93696: Locate frozen module source with __file__ (pythonGH-9…

gh-93696: Locate frozen module source with __file__ (GH-93697)

I'm adding this here in case anyone has to debug an older version of python and you still need pdb to step through "<frozen ...>".
Adapt it as needed according to the modules/files you need to debug.

This is an example of how to make the source lines available for <frozen importlib._bootstrap> on python3.9:
Early in your script(s), insert this:

import linecache

getline_original = linecache.getline
def getline_patched(filename, lineno, module_globals=None):
    if filename == '<frozen importlib._bootstrap>':
        filename = '/usr/lib/python3.9/importlib/_bootstrap.py'
    return getline_original(filename, lineno, module_globals)

linecache.getline = getline_patched

I also found this, but it didn't work for me; hence the code above.

That workaround looks reasonable to me. I would suggest that users could replace the hard-coded /usr/lib/python3.9/ part with sysconfig.get_path("stdlib") to be slightly more portable across installations and Python versions, but the truly portable fix that works for all frozen files is in the associated PR 😸

@SnoopJ

I completely agree with you @SnoopJ, the associated PR is indeed the permanent solution. 🏆

In my situation, I had to work on a legacy system, and recompiling/changing the filesystem was not an option available to me at the time. 🥲

gh-93696: Fixed the breakpoint display error for frozen modules (#132862

Navigation Menu

Repository navigation

pdb cannot find source code for frozen stdlib modules #93696

Linked PRs

2 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

pdb cannot find source code for frozen stdlib modules #93696

Description

Linked PRs

Activity

jacobtylerwalls commented on Jun 12, 2022

SnoopJ commented on Jun 12, 2022

SnoopJ commented on Jul 16, 2022

ericsnowcurrently commented on Jul 21, 2022

SnoopJ commented on Jul 21, 2022

2 remaining items

ericsnowcurrently commented on Jul 22, 2022

kumaraditya303 commented on Aug 21, 2022

rh314 commented on May 16, 2024

SnoopJ commented on May 16, 2024

rh314 commented on May 18, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions