Repository navigation
PEP 829: resolve the design of the site dir batch processing API #150228
Description
Feature or enhancement
Proposal:
(flagging as a feature, since it isn't really a bug report, just revisiting an API design detail for 3.15 where we had to make a design decision with less time for consideration than we might have wished)
The initial PEP 829 implementation relies on hidden global state in site.py to allow deferring site directory processing until several directories have been added:
known_paths = sys.path[:]
for dir_to_add in site_dirs_to_add:
site.addsitedir(dir_to_add, known_paths, defer_processing_start_files=True)
site.process_startup_files() # Defined, but not covered in the API docs yetThe hidden global state is a mixed blessing, as it means we can transparently make any site.addsitedir calls participate in a currently active batch (as was done to resolve #149504), but also makes the management and documentation of that hidden state a bit finicky.
With a slight tweak to the implementation of site._StartupState to also cover the known_paths functionality, the batch processing API could instead look like:
site_batch = site.StartupState(sys.path)
for dir_to_add in site_dirs_to_add:
site.addsitedir(dir_to_add, startup_state=site_batch)
site_batch.process()(passing both known_paths and startup_state would be an error)
Has this already been discussed elsewhere?
This is a minor feature, which does not need previous discussion elsewhere
Links to previous discussion of this feature:
The previous discussion was in the initial PEP 829 PR, where we settled for the boolean option in the public API because it was the simplest way to solve the problem before the beta 1 implementation deadline.
The fix for #149504 has subsequently done most of the work needed to enable switching the public API over to a state object rather than the boolean flag.
Linked PRs
Metadata
Metadata
Assignees
Labels
Projects
- Status
Activity
ncoghlan commentedon May 22, 2026
@warsaw @hugovk I wanted to get this idea written up properly rather than leaving it languishing in the review comments on the initial implementation PR, but with
sys.pathitself already being inherently global state, I don't actually mind the current boolean flag + hidden state based implementation all that much.If we decide to keep the existing design (which is certainly the simplest option), then this ticket would just cover adding the docs for
site.process_startup_files(), including an example of how to use it for batched site directory processing.warsaw commentedon May 22, 2026
I need to think about it a bit more, but on the face of it, I think it's a good idea. I'm also not super happy about the hidden state, so making that explicit is generally better. I'm also not necessarily opposed to trying to work this up for 3.15 if @hugovk approves.
Sadly, a lot of
site.pyjust isn't documented, but that's also not license to break things. Case in point:addsitepackages()attempted removal 😄hugovk commentedon May 22, 2026
Yes, if you can get it in by b2 (2026-06-02), and don't see any big risks when putting it together.
ncoghlan commentedon May 22, 2026
Given #149504, I'm not sure we can get rid of the hidden state completely, as any revised public API still needs to handle the startup file processing attempting to make further site directory additions.
It's technically a gap in the semantic specification for PEP 829's deferred processing (previously, with the always eager processing, startup code could assume other startup files in the newly added site directory had been processed by the time
addsitedirreturned)ncoghlan commentedon May 22, 2026
I realised this was an assumption that needed double checking, and determined that it isn't correct, but the relevant
known_pathsprocessing is a bit non-obvious.The trick is that if
known_pathsis not set, it is populated fromsys.path, andaddsitediradds the directory tosys.pathbefore processing any startup files.So I think we're fine to say startup file processing appends to the end of any site directory batch currently being processed.
We might need to tweak the definition of
known_pathsthough, as it really doesn't work right if you don't initialise it fromsys.paththe waysitedoes.ncoghlan commentedon May 24, 2026
Thinking about the batch processing prompted an experiment that revealed an unintentional backwards incompatible behaviour change.
The eager path processing in Python 3.14 inserts each site dir's path entries immediately after the containing site directory:
The current Python 3.15 batch processing puts all of the added directories after the last site directory (because the paths passed directly to
addsitedirare still processed eagerly, rather than being included in the batch processing):I didn't file a separate bug report for that, as I think it ties into the suggestion to make
StartupStatealso cover theknown_pathsparameter's functionality: when processing a batch of site directories, the site directories themselves should be considered part of the batch, so they still get added in the same order as in 3.14 and earlier relative to their associated.pthentries.There will still be one remaining discrepancy, which is that directories added with
sys.path.appendorsite.addsitedirin startup code execution will still be moved to the end rather than being interleaved (since the code execution happens after the application of the static path modifications). Rather than trying to change that outcome, I think we can instead cover that detail with a porting note in the What's New (if the directory import priority change matters, the affected startup code is going to have to be updated to ensure the order insys.pathis adjusted as desired)warsaw commentedon May 28, 2026
@ncoghlan Thanks again for the great engagement on this issue and for finding the interleaving regression. I think I've fixed the latter in the linked PR and made the whole API much more approachable and documented, while preserving the legacy behavior, hopefully all in backward compatible (to <= 3.14) ways. I think this also conforms ot @hugovk 's RM request to land before 3.15b2. I feel like with this we're in good shape for 3.15 final, and 🤞 we don't find any other regressions!
Changes based on pythongh-150228 review
gh-150228: Improve the PEP 829 batch processing APIs (#150542)
[3.15] gh-150228: Improve the PEP 829 batch processing APIs (GH-150542)…