Repository navigation
Deprecate and remove code execution in pth files #78125
Description
| BPO | 33944 |
|---|---|
| Nosy | @mhammond, @warsaw, @brettcannon, @terryjreedy, @jaraco, @ncoghlan, @pitrou, @ericvsmith, @tiran, @nedbat, @aroberge, @methane, @ericsnowcurrently, @takluyver, @zooba, @matrixise, @vedgar, @native-api, @yan12125, @asottile, @ethanhs, @csabella, @miss-islington, @chrisjbillington, @Qix- |
| PRs | |
| Dependencies |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2018-06-22.17:22:20.335>
labels = ['3.8', 'type-feature', 'library']
title = 'Deprecate and remove code execution in pth files'
updated_at = <Date 2021-01-05.10:12:30.573>
user = 'https://github.com/warsaw'bugs.python.org fields:
activity = <Date 2021-01-05.10:12:30.573>
actor = 'lkollar'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-06-22.17:22:20.335>
creator = 'barry'
dependencies = ['14803']
files = []
hgrepos = []
issue_num = 33944
keywords = ['patch']
message_count = 120.0
messages = ['320246', '320249', '320253', '320266', '320277', '320279', '320283', '320284', '320286', '320287', '320292', '320293', '320342', '320386', '320393', '320724', '320754', '320850', '320997', '321005', '321026', '321125', '321134', '321340', '328488', '328564', '329607', '329764', '329802', '330115', '333235', '333536', '333567', '333568', '333569', '333572', '333591', '333592', '333613', '333637', '333638', '333639', '333640', '333642', '333644', '333645', '333698', '333699', '333705', '333706', '333716', '333997', '334199', '335774', '335926', '336351', '336662', '336705', '336709', '336710', '336711', '336714', '336716', '336721', '336722', '336725', '336726', '336809', '336853', '336856', '336860', '336863', '336875', '336882', '336939', '336944', '336961', '336970', '336983', '336984', '336992', '337064', '337351', '337353', '337354', '337365', '337368', '337370', '337396', '337399', '337406', '337408', '337409', '337410', '337414', '337417', '337418', '337421', '337422', '337424', '337426', '337427', '337430', '337434', '337437', '337438', '337439', '337446', '337920', '337954', '350625', '351861', '351872', '358909', '358915', '358953', '368712', '368732', '371334', '384148']
nosy_count = 31.0
nosy_names = ['mhammond', 'barry', 'brett.cannon', 'terry.reedy', 'jaraco', 'ncoghlan', 'pitrou', 'eric.smith', 'christian.heimes', 'nedbat', 'aroberge', 'ionelmc', 'methane', 'SilentGhost', '__Vano', 'eric.snow', 'takluyver', 'steve.dower', 'matrixise', 'veky', 'Ivan.Pozdeev', 'yan12125', 'Anthony Sottile', 'Michel Desmoulin', 'ethan smith', 'cheryl.sabella', 'lkollar', 'miss-islington', 'Chris Billington', 'Peter L3', 'qix-']
pr_nums = ['10131', '12107', '12110', '15942']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue33944'
versions = ['Python 3.8']Reacted by Pat Hawks and Adrian Friedli
Pinned by picnixz
merwokI think this is a PEP-level change, so am closing the discussion here.
Please continue on the forum threads:
Activity
warsaw commentedon Jun 22, 2018
pth files are evil. They are very difficult to debug because they're processed too early. They usually contain globs of inscrutable code. Exceptions in pth files can get swallowed in some cases. They are loaded in indeterminate order.
They are also unnecessary to support namespace packages in Python 3 (ignoring straddling code).
Let's start the process for removing them.
Deprecate pth files in Python 3.8 and turn them off with the -3 option.
Kill off pth file support once Python 2 is EOL'd.
tiran commentedon Jun 22, 2018
+1
ericvsmith commentedon Jun 22, 2018
Also +1.
takluyver commentedon Jun 22, 2018
I'm generally in favour of getting rid of .pth files. But I did accept a PR adding support for them in Flit to act as a substitute for symlinks on Windows, to achieve something like a 'development install'. I'm not sure what the alternative is if they go away.
brettcannon commentedon Jun 22, 2018
Windows has symlinks now I believe, you just have to turn them on.
And I would say there is no need for alternative. If a package needs to do something funky they can do it in their __init__.py file. Otherwise if I don't import a package it shouldn't get to do anything crazy through a .pth file.
takluyver commentedon Jun 22, 2018
I don't want to use the execution features of .pth files, just their original functionality of adding extra directories to sys.path. I'd be very happy to see the arbitrary code execution 'feature' of .pth files go away.
Windows supports symlinks, but the last I heard was that creating them requires some obscure permission bit. It seems to be awkward enough that Windows users aren't happy with the "just use symlinks" approach, which was what I was originally trying.
ericvsmith commentedon Jun 22, 2018
My understanding about symlinks on Windows is that they require a permission ("Create symbolic links"), that normal users by default do not have. I'm not sure if this has changed recently.
ethanhs commentedon Jun 22, 2018
I am in favor of symlinks no longer being able to execute arbitrary code, however, I do think having them add to the path cannot be killed in two releases. Here is why:
Windows support for symlinks is still not automatic. In the creators update of Windows 10 (released March 2017), CreateSymbolicLink added a dwflag SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE. This requires the user to be in developer mode to work. CPython currently doesn't use this flag. (I will open an issue to add that in a moment). I worry that giving people little time to update will be troublesome.
All editable installs everywhere (AFAIK) and setuptools eggs (still somewhat common) use easy-install.pth to list where they are. I think breaking editable installs is a bad idea, as there is no clear solution for this. Also setuptools has a fair amount of work to do before it can replace egg installs.
So I think removing adding to the path will require much more thought and break a lot more code than removing arbitrary code execution.
brettcannon commentedon Jun 22, 2018
My only answer to Ethan is "don't use eggs". :)
warsaw commentedon Jun 22, 2018
There are lots of problems with pth files, although arbitrary code execution is probably the most egregious. They are also notoriously difficult to debug, and happen before any control is given to user code. They certainly are unnecessary for namespace packages, which I think they currently get used for often in Py 2/3 straddling code.
Maybe it will be okay to just fall back to sys.path extension, but I'd like to have a better understanding of exactly what the use cases are (in a pure Python 3 world), and we have to address the other problems about discovery and debuggability.
ncoghlan commentedon Jun 23, 2018
Strong -1 without a functional replacement that provides comparable LD_PRELOAD capabilities (it also needs a full PEP that analyses all of the ways that setuptools and other packaging utilities use these files, such as for the implementation of "develop" mode, and the processing of ".lnk" mode).
This change also needs to account for the Windows-only "._pth" files that override the path completely.
The main discussion list for such a PEP should be distutils-sig, *not* python-ideas or import-sig (since distutils-sig is where we're more likely to find folks that are actually relying on the feature, and hence have a clearer idea of what will need to change to maintain a comparable level of ecosystem level capability).
https://bugs.python.org/issue14803 is also related, as pth file processing should at least be delayed to run later than it does currently, and because "run code at startup" is one of the capabilities that would need replacing.
ncoghlan commentedon Jun 23, 2018
Concrete use case for the original path extension capability: "pew add", which chains virtual environments together (allowing shared environments with a common default dependency set, and then additional per-application dependencies)
ncoghlan commentedon Jun 24, 2018
Brett pointed out that may initial reaction above came across as quite blunt and demanding, so attempting to phrase that more clearly as a user experience consideration:
It may be tempting to view this as purely a clean-up of the import system implementation, removing a quirky and error prone construct for the sake of improved maintainability of both the import system itself, and the maintainability of end user installations.
My request (wearing my "BDFL-delegate for packaging interoperability standards" hat) is that proponents of the change resist the temptation to view the problem that way :)
Path files are used extensively across the Python packaging ecosystem to implement additional environment management features beyond those provided natively by interpreter implementations, and while we've added native equivalents for some of them (namespace packages, virtual environments), we're far from having added support for all of them (dynamic package version selection, virtual environment chaining, editable package installs that still publish correct PEP-376 package metadata, etc).
This means that any changes in this area pose significant backwards compatibility risks, and need to be approached carefully, and cautiously, with a strong emphasis on surveying real world code and seeing how the feature is currently being used.
Or, alternatively, the idea can be broken up into smaller, lower impact changes that still help to address the import system and end user environment maintainability issues, but don't involve breaking backwards compatibility.
(For an example of the latter: if "python -m site --list-pth-files" printed a list of all of the pth files and "python -m site --dump-pth-files" listed both the files and their contents, then environment debuggability would improve significantly without any compatibility impacts whatsoever)
145 remaining items
webknjaz commentedon Mar 25, 2026
(packaging notes)
Just wanted to add to this, FTR, that we shouldn't forget the downstream distro packagers that are a bit farther away from Python that would have to implement the same — things like dnf, apt, emerge, homebrew and others.
This format is pip-specific and does not have a standard. Some tools just happen to parse it sometimes but there's no rules other than "whatever pip does". The respective accepted standard would be the pylock one (PEP 751) that the ecosystem is still catching up with.
danielzgtg commentedon Apr 1, 2026
PyTorch and 50% of my ML frameworks already break on every Python update. People using
.pthlegitimately can join them and find a workaround for any backwards-incompatible change.There's a simpler reason stated above as to why we should remove
.pthsupport. It would reduce code complexity and ease the maintainability burden slightly. And simpler code leads to a slightly reduced attack surface.FFY00 commentedon Apr 1, 2026
There seems to be a consensus from the maintainers that we can't remove
.pthfile support without a replacement.I'd like to ask everyone to please refrain from leaving drive-by comments with this blanket request. Instead, please start a thread in https://discuss.python.org/c/ideas/6 with a concrete proposal.
pwilkin commentedon Apr 1, 2026
I wouldn't know how to get the statistics honestly, but I'd start by actually doing concrete steps on what @sethmlarson already mentioned above:
It seems to me like a lot of people here have floated a lot of various concepts around, but having a structured table with the exact functionalities and an idea on how to get the list of affected packages would be a good start, no?
danielzgtg commentedon Apr 2, 2026
How can we ensure reduced complexity and increased maintainability if we introduce "a replacement"?
danielzgtg commentedon Apr 2, 2026
I have started a thread at https://discuss.python.org/t/prevent-open-2-ing-of-pth-files-during-startup/106792 with a concrete proposal using
seccomp(2).I found 3 on my system, which I described in a "structured table" in the thread.
merwok commentedon Apr 2, 2026
I think this is a PEP-level change, so am closing the discussion here.
Please continue on the forum threads:
warsaw commentedon Apr 7, 2026
With the creation of PEP 829 I'm reopening this issue so that it's easier to find previous discussions. I'll link to it in the next draft of 829.
Replace deprecated URI, in `2019-03-11-flatpak-problems.markdown`, wi…
warsaw commentedon Jun 3, 2026
Closing now as I believe we're done. True,
importlines aren't removed yet, but PEP 829 in Python 3.15 puts us on that path so we'll remove the support when the time comes.