Skip to content

venv on Windows with symlinks is broken if invoked with -I #86179

Closed
Closed
venv on Windows with symlinks is broken if invoked with -I#86179
Labels
3.10 (EOL)end of life3.9 (EOL)end of lifeOS-windows

Description

@gaborbernat
gaborbernat
mannequin
opened on Oct 12, 2020
BPO 42013
Nosy @pfmoore, @tjguk, @zware, @eryksun, @zooba, @gaborbernat

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-10-12.10:55:28.257>
labels = ['3.10', '3.9', 'OS-windows']
title = 'venv on Windows with symlinks is broken if invoked with -I'
updated_at = <Date 2020-10-12.22:39:02.541>
user = 'https://github.com/gaborbernat'

bugs.python.org fields:

activity = <Date 2020-10-12.22:39:02.541>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2020-10-12.10:55:28.257>
creator = 'gaborjbernat'
dependencies = []
files = []
hgrepos = []
issue_num = 42013
keywords = []
message_count = 12.0
messages = ['378485', '378487', '378488', '378490', '378491', '378492', '378493', '378494', '378499', '378507', '378512', '378532']
nosy_count = 6.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'gaborjbernat']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue42013'
versions = ['Python 3.9', 'Python 3.10']

Linked PRs

Activity

  1. gaborbernat commented on Oct 12, 2020

    @gaborbernat
    gaborbernatmannequin
    on Oct 12, 2020
    MannequinAuthor

    Here's a small reproducible, run it on a Windows OS that has symlinks enabled:

    import shutil
import venv
import subprocess

shutil.rmtree("venv", ignore_errors=True)
venv.EnvBuilder(with_pip=False, symlinks=True).create("venv")

    # works
    subprocess.check_call(["venv\\Scripts\\python.exe", "-c", "import sys; print(sys.executable)"])

    # fails with No module named 'encodings'
    subprocess.check_call(["venv\\Scripts\\python.exe", "-Ic", "import sys; print(sys.executable)"])

  2. gaborbernat
    added
    3.9 (EOL)end of life
    3.10 (EOL)end of life
    OS-windows
    on Oct 12, 2020

  3. eryksun commented on Oct 12, 2020

    @eryksun
    eryksun
    on Oct 12, 2020
    Contributor

    I can't reproduce the issue with the normal 3.9.0 distribution from python.org. For example:

        >>> venv.EnvBuilder(with_pip=False, symlinks=True).create("venv")
    >>> subprocess.check_call(["venv\\Scripts\\python.exe", "-Ic", "import sys; print(sys.executable)"])
    C:\Temp\env\venv\Scripts\python.exe
    0

    Which Python version(s) and distribution(s) did you test? Do you have the PYTHONHOME and/or PYTHONPATH environment variables set?

  4. gaborbernat commented on Oct 12, 2020

    @gaborbernat
    gaborbernatmannequin
    on Oct 12, 2020
    MannequinAuthor

    ❯ py -c 'import sys; print(sys.version)'
    3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)]

  5. FFY00 commented on Oct 12, 2020

    @FFY00
    FFY00
    on Oct 12, 2020
    Member

    This error most likely happens because sys.path isn't being set properly and the standard library isn't being included.

    We need more information about the specific environment this is triggering.

    Which CPython build are you using? The one from Github Actions, right?
    We also need the environment variables.

  6. FFY00 commented on Oct 12, 2020

    @FFY00
    FFY00
    on Oct 12, 2020
    Member

    Well, actually the environment variables /should/ not matter as -I implies -E.

  7. gaborbernat commented on Oct 12, 2020

    @gaborbernat
    gaborbernatmannequin
    on Oct 12, 2020
    MannequinAuthor

    Ok, the missing link is that the python you run into needs to be also a symlink venv:

    ❯ py -m venv env --without-pip --clear --symlinks
    ❯ .\env\Scripts\python.exe -c "import venv, subprocess; venv.EnvBuilder(with_pip=False, symlinks=True).create('venv'); subprocess.check_call(['venv\\Scripts\\python.exe', '-Ic', 'import sys; print(sys.executable)'])"

    Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
    Python runtime state: core initialized
    ModuleNotFoundError: No module named 'encodings'

  8. gaborbernat commented on Oct 12, 2020

    @gaborbernat
    gaborbernatmannequin
    on Oct 12, 2020
    MannequinAuthor

    We need more information about the specific environment this is triggering.

    Both Gitub Actions Windows CPython3.9 or installer as downloaded from (on Windows 10) https://www.python.org/downloads/release/python-390/.

    PYTHONHOME should not be set all. PYTHONPATH needs to be set carefully. It should never include standard-library path

    The issue manifests independent of those environment variables, netiher of them are set.

  9. eryksun commented on Oct 12, 2020

    @eryksun
    eryksun
    on Oct 12, 2020
    Contributor

    Well, actually the environment variables /should/ not matter as -I
    implies -E.

    The operative word there is "should". I was grasping for anything that might explain why I couldn't reproduce the issue.

    Ok, the missing link is that the python you run into needs to
    be also a symlink venv

    Thank you, that reproduces the problem for me.

  10. eryksun commented on Oct 12, 2020

    @eryksun
    eryksun
    on Oct 12, 2020
    Contributor

    This issue is partly due to bpo-8901, which changed the behavior of the -E and -I command-line options to make them ignore the default PythonPath value in the registry key "Software\Python\PythonCore\X.Y\PythonPath". The change itself is not wrong. It's just exposing an underlying problem.

    The home path in pyvenv.cfg is from sys._base_executable. In a launcher-based environment that's created from the base installation, sys._base_executable is the real base executable. OTOH, in a symlink-based virtual environment, sys._base_executable is the same as sys.executable. Consequently, if a virtual environment is created from a symlink-based virtual environment, the home path in pyvenv.cfg refers to the creating environment instead of the base installation. In this case, with the current implementation, the standard library can only be found by falling back on the default PythonPath in the registry.

  11. 16 remaining items

  12. zooba
    added a commit that references this issue on Dec 14, 2023

    pythongh-86179: Implement realpath() on Windows for getpath.py calcul…

    23703a9

  13. zooba commented on Dec 14, 2023

    @zooba
    zooba
    on Dec 14, 2023
    Member

    I'm hesitant to backport the fix, since people always get upset when we change initialisation paths at all.

    If anyone has a strong case for backporting, let me know. Otherwise I'll close this and #113121 in a few days and just call it done.

  14. vsalvino commented on Dec 14, 2023

    @vsalvino
    vsalvino
    on Dec 14, 2023

    I think as long as the fix is applied in 3.12 or 3.13, that's fine, it doesn't need to be backported to older versions.

  15. zooba commented on Dec 15, 2023

    @zooba
    zooba
    on Dec 15, 2023
    Member

    3.13 is where the fix will be. Going to 3.12 would be a backport.

  16. corona10
    added a commit that references this issue on Dec 15, 2023

    pythongh-86179: Avoid making case-only changes when calculating realp…

    7a7211e
  17. zooba
    added 2 commits that reference this issue on Jan 16, 2024

    pythongh-86179: Skip test case that fails on POSIX with unversioned b…

    7635b92

    gh-86179: Skip test case that fails on POSIX with unversioned binary (G…

    b204c4b
  18. kulikjak
    added a commit that references this issue on Jan 22, 2024

    pythongh-86179: Skip test case that fails on POSIX with unversioned b…

    84ce4fd
  19. aisk
    added 3 commits that reference this issue on Feb 11, 2024

    pythongh-86179: Implement realpath() on Windows for getpath.py calcul…

    1af21d5

    pythongh-86179: Avoid making case-only changes when calculating realp…

    887a49c

    pythongh-86179: Skip test case that fails on POSIX with unversioned b…

    7cd38cd
  20. Glyphack
    added 3 commits that reference this issue on Sep 2, 2024

    pythongh-86179: Implement realpath() on Windows for getpath.py calcul…

    390b1fa

    pythongh-86179: Avoid making case-only changes when calculating realp…

    ecaf5fd

    pythongh-86179: Skip test case that fails on POSIX with unversioned b…

    2f09e11
  21. zooba
    closed this as completedon Apr 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.10 (EOL)end of life3.9 (EOL)end of lifeOS-windows

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions