Hosted and lock-only scans treat a UTF-16 requirements.txt (what Windows PowerShell's `pip freeze >` writes) as absent: exit 0, no warning, and pip keeps installing the unpatched pin

[mikolalysenko]

[agent] Found by the scheduled pip / requirements.txt bug-hunt routine (ledger #309).

Summary

Windows PowerShell 5.1 redirects with UTF-16 LE plus a BOM, so pip freeze > requirements.txt there produces a UTF-16 file. pip reads it without complaint: req_file.py auto-decodes UTF-8, UTF-16 and UTF-32 BOMs, and pip 20.3.4 through 26.2.1 all install from it. socket-patch's hosted scan reads requirements.txt only as UTF-8. On a decode failure it drops the file as though it weren't there, with no diagnostic:

• Hosted scan with a venv (six 1.16.0 installed, patch available): exit 0, status: "success", redirect: {redirected: 0, skipped: [], warnings: []}. Human mode prints pkg:pypi/six@1.16.0: no lockfile entry pinning it could be rewritten, which is wrong because the file pins it.
• Lock-only hosted scan (fresh checkout, empty venv): No packages found. Run your package manager's install first. / scannedPackages: 0, exit 0.

The project then installs upstream six with no signal that the file was skipped.

Other commands are loud or correct on the same file. vex warns lockfile_unreadable ("cannot read requirements.txt: stream did not contain valid UTF-8"), and vendored vendor / scan --mode vendored fails with pypi_no_requirements "cannot read …/requirements.txt", exit 1. So only the hosted / discovery path is silent.

Impact

A Windows team whose requirements.txt was generated in PowerShell 5.1 (still the default powershell.exe on Windows 10 and 11) runs socket-patch scan --mode hosted in CI and gets exit 0 with a clean JSON envelope, but nothing is patched. On a fresh checkout it even says there are no packages at all.

Repro

This uses the hosted wiremock from crates/socket-patch-cli/tests/mode_migration_pypi.rs::mount_hosted_api (batch / by-package / package / wheel), with $MOCK as its URI.

python3 -m venv /tmp/v && /tmp/v/bin/pip install six==1.16.0 idna==3.7
mkdir proj && cd proj
python3 -c "open('requirements.txt','wb').write('idna==3.7\r\nsix==1.16.0\r\n'.encode('utf-16'))"   # = PowerShell 5.1 `pip freeze >` output
/tmp/v/bin/pip install --dry-run -r requirements.txt          # pip reads it: would install idna, six

A="--api-url $MOCK --org test-org --api-token fake --patch-server-url $MOCK"
VIRTUAL_ENV=/tmp/v socket-patch scan --mode hosted --yes $A --json      # exit 0, redirected 0, skipped [], warnings []
VIRTUAL_ENV=/tmp/v socket-patch scan --mode hosted --yes $A             # "no lockfile entry pinning it could be rewritten"
mkdir -p ../empty/lib/python3.13/site-packages
VIRTUAL_ENV=$PWD/../empty socket-patch scan --mode hosted --yes $A      # "No packages found. Run your package manager's install first."
socket-patch vex --output vex.json --json                               # (contrast) warnings: lockfile_unreadable

The same file in UTF-8 (identical text, CRLF) is redirected (redirected: 1), and a fresh pip install -r then installs the patched six.

Expected vs actual

• Expected: CLI_CONTRACT.md defines lockfile_unreadable as "A supported file exists but could not be read (permissions, a FIFO squatting the name, non-UTF-8)", and vex already emits it here. For socket.yml, the contract makes UTF-16 an explicit error ("UTF-16 and NUL bytes are errors"). The comment above unconfirmed in scan/hosted.rs says an unpinned granted package must never vanish silently. So hosted scan should either decode the BOM the way pip does, or report the file as unreadable in redirect.warnings[] / skipped[] (and on stderr). Lock-only discovery should say it couldn't read requirements.txt rather than "No packages found".
• Actual: the file is invisible to hosted scan and to lock-only discovery. Exit 0 with a success envelope; the human message blames a missing pin.

Matrix

OS	pip / Python	Case	Reproduces
Linux	26.2.1 / CPython 3.13	hosted, venv with six (--json and human)	yes (2/2)
Linux	26.2.1 / CPython 3.13	hosted, lock-only	yes (2/2)
Linux	20.3.4 / 3.8, 24.3.1 / 3.12, 25.1.1 / 3.12, 26.2.1 / 3.13	pip install -r on the UTF-16 file	pip installs idna + six on all four
Linux	—	vex on a UTF-16 file	no: lockfile_unreadable is emitted (correct)
Linux	—	vendor on a UTF-16 file	no: pypi_no_requirements, exit 1 (loud)
Linux	26.2.1	the same text in UTF-8	no: redirected and patched

macOS and Windows weren't probed. The decode is OS-independent; the file shape is what Windows PowerShell 5.1 writes.

First bad version: none. Published v4.0.0 (PyPI socket-patch==4.0.0) behaves the same way: Redirected 0 package(s), and lock-only says "No packages found". It isn't a regression.

Suspect code

• crates/socket-patch-core/src/hosted/engine.rs:324: CandidateFiles::read uses view.read_text(rel).await.ok(), so an InvalidData decode error becomes "file absent". The memory-view branch at :334 says so explicitly ("a non-UTF-8 one is absent to it as well"). Then patch/redirect/requirements.rs::rewrite returns early on files.get("requirements.txt") being None, with no warning.
• crates/socket-patch-core/src/vendor/lock_inventory/pypi.rs:711: requirements_tree does view.read_text(ROOT).await.ok()?, so lock-only discovery drops the root file silently.
• Related but distinct: Hosted scan --json on a PyPI project with no root requirements.txt (e.g. only requirements-dev.txt or requirements/base.txt) reports success with empty skipped and warnings, though the patched package is never pinned #638 (no root requirements.txt → silent JSON). Here the file exists and pip installs from it; a fix for Hosted scan --json on a PyPI project with no root requirements.txt (e.g. only requirements-dev.txt or requirements/base.txt) reports success with empty skipped and warnings, though the patched package is never pinned #638 that only checks file presence would still miss this. Hosted and vendored NuGet reject a packages.lock.json with a UTF-8 BOM that dotnet restores fine: hosted skips the redirect and exits 0 success, vendored fails apply_failed #623 is the NuGet UTF-8-BOM analogue.

Probe runs: none (OS-independent text decoding; Linux only).

[mikolalysenko]

[agent] Triaged as priority:p1 (pip / requirements.txt). Not a duplicate: #623 is the NuGet UTF-8 BOM reader, and I found no other issue, open or closed, about UTF-16 requirements files. Confirmed on main (045d7ec): CandidateFiles::read in crates/socket-patch-core/src/hosted/engine.rs treats a disk read_text error as "absent" (view.read_text(rel).await.ok()), so an InvalidData decode is dropped with no lockfile_unreadable warning, while vex reports one for the same file.

---

Generated by Claude Code

[mikolalysenko]

[agent] Claiming this issue (shared root cause: the hosted candidate reader and lock-only discovery treat a requirements file they can't decode as absent). Branch: agent/fix-hosted-unreadable-candidate. Claim-ID: 2026-10-03T21:20:37Z-13ca73

---

Generated by Claude Code

[mikolalysenko]

[agent] Draft PR: #724

---

Generated by Claude Code

[mikolalysenko]

[agent] New trigger for the same root cause (pip bug-hunt, ledger #309): a requirements.txt with no BOM but a PEP 263 coding line, which pip's auto_decode honours.

printf '# -*- coding: latin-1 -*-\n# Maintainer: Jos\xe9\nsix==1.16.0\n' > requirements.txt
pip install -r requirements.txt     # pip 20.3.4 and 26.2.1: rc=0, installs six
# (without the coding line pip raises UnicodeDecodeError, so the line is what makes it valid)

On main 045d7ec, scan --mode hosted --json exits 0 with redirected: 0 and empty skipped / warnings, both lock-only (scannedPackages: 0) and with a .venv holding six 1.16.0 (scannedPackages: 3). The file is left untouched, so it's the same silent no-op as the UTF-16 case. I ran it twice.

Reading the #724 diff, utils::requirements::decode returns None for any non-BOM, non-UTF-8 bytes, so this file takes the candidate_file_unreadable path rather than being decoded. Refusing it loudly would be acceptable; the remaining risk is a path that still treats None as "absent" (the lock-only inventory doc says "None when the root file itself cannot be read"). A latin-1 + coding-line fixture next to the UTF-16 ones would pin that down.

---

Generated by Claude Code