Skip to content

Stop fetching patch views during hosted scans - #2021

Open
Mikola Lysenko (mikolalysenko) wants to merge 6 commits into
mainfrom
refactor/hosted-scan-no-patch-view
Open

Mikola Lysenko (mikolalysenko) wants to merge 6 commits into
mainfrom
refactor/hosted-scan-no-patch-view

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

LLM Description written by Claude Code:claude-opus-5-5

Stacked on #2015 (which is on #2013).

Summary

Hosted scans no longer call /patches/view. A view carries every patched file's blob contents, and v5 keeps no hosted ledger, so the scan was fetching full patch bodies only to feed checks that run within the same scan. Each of those checks now uses data the scan already has, or fetches its own:

Consumer Before Now
In-run --vex records from the scan's view fetches fetches the records it attests itself via vex_sources::fetch_records, the same path standalone vex uses; output unchanged
Gem stale-install probe patched file hashes from the view sha256 of the archive RubyGems installed from (<home>/cache/<leaf>.gem, and committed vendor/cache) vs. the grant's artifact sha256
Python stale-install probe patched file hashes from the view the patched wheel's own RECORD, downloaded through its grant URL (sha256-checked) only when an installed copy exists and its WHEEL tags match; never under --offline
vlt warm-tree heal record byte checks vlt's hidden lock alone: the pinned integrity means healthy, anything else means stale, and no readable hidden lock means undeterminable
In-memory engine / Node binding fetchPatch provider, presence-checked only not called; the binding still accepts a fetchPatch method and ignores it, so existing hosts (depscan's autopatch provider) keep working

When an agent-mode manifest holds a record for the same patch, the gem and Python probes use that local record instead, so an agent→hosted migration stays quiet.

Vendored mode, agent mode, get <uuid>, vendor eject and standalone vex are unchanged and still use the view.

Behavior changes

  • record_fetch_failed no longer exists. A view failure during in-run VEX is reported as vex_record_fetch_failed.
  • A hosted scan with --vex where every record fetch fails ends with a record_unavailable skip instead of manifest_not_found.
  • Review this one: the Python probe now downloads a patched wheel, which a hosted scan never did before, but only for a confirmed patch with an installed copy whose tags match, at most once per patch. An artifact-free alternative would be the installed dist's direct_url.json archive hash, but not every installer writes one, so I kept the wheel RECORD.
  • vlt: a FIFO or symlinked hidden lock used to get its store entry invalidated when a record proved the bytes stale. Now the entry is kept and the advisory asks for vlt ci, as it does when there is no hidden lock (new test). The advisory names the missing or unreadable hidden lock.

CLI_CONTRACT.md, index.d.ts and docs/testing/poetry-compatibility.md are updated.

Validation

Check Result
cargo fmt --check pass
cargo clippy --workspace --all-features -- -D warnings pass
core --lib (isolated HOME, see #2015 for the NuGet cache test) 7137 passed
CLI lib 860 passed
e2e_redirect_gem_stale_install 42 passed
e2e_vex_lockfile 354 passed
hosted_memory_engine 49 passed
in_process_redirect 134 passed
pdm / pipenv / poetry redirect suites 6 / 11 / 9 passed
scan 248 passed
Node smoke test 14/14 passed

An earlier run also passed cli, get, cli_parse_scan, cli_parse_get, hosted_memory_parity, hosted_memory_rollout, in_process_rollback_hosted, in_process_scan, in_process_vendor_bun_takeover, mode_migration_vlt and mode_migration_bun.

Tests that used a view request as proof that a redirect was confirmed now assert zero view calls, or confirm the redirect through a re-scan's envelope. Fixtures that lay down a gem install by hand now write the cache archive RubyGems would. The pdm, pipenv and poetry stale-install tests now serve a real patched wheel from the mock server.

Not run: the hatch and uv real-toolchain e2e suites, which already serve real wheels.

🤖 Generated with Claude Code


Note

Medium Risk
Changes hosted post-rewrite behavior (new wheel downloads, vlt heal rules, and in-run VEX fetch timing) without altering vendored/agent modes or standalone vex’s overall model.

Overview
Hosted scan / get --mode hosted no longer call /patches/view after a rewrite. v5 does not persist hosted records, and views were only feeding same-run checks that now use data the scan already has or fetch narrowly on their own.

In-run --vex takes confirmed pin UUIDs and loads records through vex_sources::fetch_records (like standalone vex), instead of a pre-built map from the scan. record_fetch_failed is gone; view failures show up as vex_record_fetch_failed / record_unavailable.

Stale-install probes no longer depend on fetched patch bodies: gems compare the installed cache/<leaf>.gem (and committed vendor cache) to grant artifact shas, with agent manifest.json afterHash still used when present; Python downloads the pinned wheel once ( RECORD + matching WHEEL tags) via the grant URL when a copy is installed—not under --offline; vlt heal/stale checks use node_modules/.vlt-lock.json integrity only, so missing/unreadable hidden lock keeps store entries and advises vlt ci instead of record-driven invalidation.

Contract docs and tests are updated (including zero-view assertions and gem cache fixtures).

Reviewed by Cursor Bugbot for commit fb3d418. Configure here.


Generated by Claude Code

A hosted scan of a vlt project downloaded every granted patch archive
the way vlt fetches it, and withheld the dependency
(`redirect_vlt_artifact_unverifiable`) when the server re-gzipped it.
That guarded against patch.socket.dev's CDN re-encoding archives,
which vlt rejects with EINTEGRITY. The patch server has sent archives
with `Cache-Control: no-transform` since depscan #26856 and #26867,
and production serves them unencoded today. On any mirror that still
re-encodes, `vlt ci` fails closed with EINTEGRITY, so nothing installs
unpatched bytes.

So the scan no longer fetches the archives. It pins vlt dependencies
directly, without a full download of every patched package per run.
The in-memory hosted engine (the Node addon and `hosted-bundle`) has
no network, so it used to withhold every vlt dependency as "offline";
it now pins them the way the disk flow does.

The warm-tree heal now checks the Socket-owned lock instances this run
confirmed in `vlt-lock.json`. With no patch record and no hidden lock
it reports the copy as undeterminable (asking for `vlt ci`) instead of
comparing it against downloaded archive bytes.

The vlt backtest still probes the live artifact itself: a must-patch
hosted cell records `blocked-by-server-encoding` when its probe sees a
content-encoded response, without running the CLI.

Assisted-by: Claude Code:claude-opus-5-5
A vendored run decides before any download which patches the engine
would refuse, so it never fetches a view, asks for a grant or strips a
hosted pin for a package it cannot vendor. That decision was assembled
in the CLI from five sources (the Bun and vlt preflights, the
package-lock project refusal, the lock-text gates and the pnpm
takeover), and each entry point merged its own subset by hand. They
had drifted: the download phase never asked the package-lock project
refusal or the pnpm takeover gates, `get <uuid>` raised only the Bun
and vlt refusals before the engine, and the vendor loop judged only
Bun and vlt before its takeover.

All of it now lives in one core function,
`socket_patch_core::vendor::refusals::vendor_refusals`, which runs the
checks in the engine's order and scopes the lock-text gates (hosted
pins, the ledger's idempotency skip, packages that never reach their
backend). The scan preview and rollout plan, the download phase, both
`get` paths and the vendor loop call it. The CLI's bun_preflight.rs
and vlt_preflight.rs are gone, and their tests moved to core.

What users see: a v1 package-lock project, or a hosted pnpm pin whose
takeover the pnpm gates refuse, is now refused before its patch view
is fetched; `get <uuid> --mode vendored` raises every such refusal
with its pre-record error envelope and writes nothing; and `vendor`
refuses them before restoring a hosted pin instead of restoring and
rolling back. Codes and details are unchanged.

Assisted-by: Claude Code:claude-opus-5-5
A hosted scan fetched the full patch view (every patched file's blob
content) for each confirmed redirect, only to judge installed copies
and feed the same run's VEX. It now makes no view call at all:

- In-run `--vex` fetches the records it attests from itself, through
  the same record path the standalone `vex` uses, so the scan pipeline
  never asks for them.
- The gem stale-install probe judges an installed gem by the archive
  RubyGems installed it from (`<home>/cache/<leaf>.gem`) against the
  patched artifact's sha256.
- The Python stale-install probe downloads the patched wheel the lock
  now pins, through its grant URL and verified against the pinned
  sha256, and compares the installed copy with the wheel's own RECORD
  when the installed WHEEL tags match. It downloads only when a copy
  is installed.
- Both probes still treat an agent-mode manifest record of the same
  patch as proof of an in-place patch.
- The vlt warm-tree heal judges a store entry by vlt's hidden lock
  alone: the pinned integrity there is healthy, another is stale, and
  without a readable hidden lock the copy is kept and reported as
  unchecked.
- The in-memory hosted engine no longer calls the provider's
  `fetchPatch` or warns `record_fetch_failed`. The Node binding still
  accepts a `fetchPatch` provider method and ignores it, so existing
  providers keep working.

Assisted-by: Claude Code:claude-opus-5-5

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

…c) e2e

judge_installed_wheel read the installed WHEEL and every RECORD path with
bare std::fs reads. Site-packages is user-writable, so a FIFO there would
block the hosted scan's blocking pool. Read through
read_regular_*_sync so a FIFO or device counts as unreadable, and add a
regression test for both the WHEEL and a RECORD path.

The real-vlt e2e for heal rule (c) without a hidden lock still expected
the old record-based heal (copies invalidated). Hosted scans no longer
fetch patch records, so that case is now undeterminable: the copies are
kept and the advisory says to run `vlt ci`. Match the in-process test and
check that `vlt ci` then installs the patched package.

Co-Authored-By: Claude <noreply@anthropic.com>
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Labeled Ready for review by the burn-down agent.

  • Head: fb3d4183852150c00139b671240b93019ba88655
  • CI: 49/49 check runs green (30 success, 19 skipped by path filters)
  • Fixed in fb3d418:
    • e2e_redirect_vlt_build rule (c) without a hidden lock still expected the old record-based heal. This PR makes that case undeterminable (copies kept, advisory says run vlt ci), so the e2e now matches the in-process test and checks that vlt ci installs the patched package.
    • Bugbot (medium): the wheel probe read site-packages files with bare std::fs reads, so a FIFO could block the scan. It now uses the regular-file guard, with a FIFO regression test.
  • Bugbot: re-reviewed fb3d418, no new issues.
  • Reviewer note: stacked on Refuse vendored patches the same way everywhere #2015, which is stacked on Stop downloading vlt artifacts during hosted scans #2013. heal_rule_c_no_record now covers the same scenario as heal_rule_c_no_hidden_lock because patch records are never fetched; consider dropping one of them.
  • Slack announcement not sent this run (Slack send tool unavailable); next run retries.

Generated by Claude Code

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Nov 4, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Final review brief (for the human approver)

What it does: hosted scans stop calling /patches/view, which returned the full patched file contents just to feed checks inside the same scan. In-run --vex now fetches its own records, the same way standalone vex does. The gem stale-install probe compares the sha256 of the cached .gem with the grant's. The Python probe downloads the patched wheel, verifies its sha256 and reads its RECORD. The vlt heal trusts only vlt's hidden lock. Stacked on #2015, so it merges into that branch, not main.

Risk: medium. I found no security problem and nothing that makes the probes warn wrongly. The risk is in the Python probe: it now judges an installed copy only when its WHEEL tags and dist-info name exactly match the first patched wheel for that patch. Where they don't, a stale venv goes unwarned with no signal (see the open questions).

Look here

  • wheel_record.rs:40-50: checks the sha256 before it parses anything. Downloads are capped at 256MB and RECORD/WHEEL reads at 16MB. Nothing is extracted. RECORD paths go through is_safe_relative_subpath.
  • wheel_record.rs:134-171:`` the tag gate.
  • python.rs:91-110:``` how the override is chosen and the --offline skip.
  • hosted.rs:387-407:```` any cached .gem whose sha isn't a patched one now warns as stale.
  • vlt_heal.rs:258-286:````` with no readable hidden lock the result is now Undeterminable (it used to be `Healthy`), and the in-run VEX drops it. That errs on the safe side.

Verified: I read the PR's own diff (42843ef..fb3d4183852150c00139b671240b93019ba88655, 31 files). No view calls remain on the scan, memory-engine or Node paths. --offline and --dry-run are honoured. CRLF in RECORD is handled. CHANGELOG.md is untouched and there is no debug leftover. Ran locally: wheel_record unit tests (5), CLI stale (23), in_process_redirect_poetry and e2e_redirect_gem_stale_install, all passing. The poetry test depends on the served wheel, so it exercises the new probe. CI: 30 passed and 19 skipped by path filter (49 checks). Bugbot: passing, and its one thread is resolved.

Changes I made: none.

Open questions / nits (none block merging)

  1. When a patch ships several platform wheels, or the patched wheel changes its dist-info name or tags (PyYAML vs pyyaml, py2.py3 → py3), the probe never judges the installed copy. Try every matching .whl override? This could be a follow-up.
  2. Every re-scan downloads the whole patched wheel again for each pypi candidate that has an installed copy and no local record. That costs about what the old view fetch did.
  3. No test covers the --offline skip of the wheel probe, or a gem with no .gem under <home>/cache (git or path sources).
  4. Nit: hosted.rs:974-976 still says "patch records (fetched below)", but that fetch is gone.

Auto-merge is not armed. This PR targets #2015's branch, which has no review rule, so arming it would merge it immediately. Approve and merge it after #2015, or retarget it to main once #2013 and #2015 land.


Generated by Claude Code

patched: patched_shas.is_some_and(|shas| shas.contains(&sha)),
stale_evidence: patched_shas.is_some_and(|shas| !shas.contains(&sha)),
},
None => Default::default(),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[agent] Dropping the patch-view record turns "can't judge" into "not stale" here, and in-run VEX attests every confirmed purl not in known_stale via assume_applied (hosted.rs ~1656, vex.rs ~905), so these probes are the only guard against an unpatched warm install.

  • Gem: with no agent-mode record and no <home>/cache/<leaf>.gem (Docker images that rm -rf $BUNDLE_PATH/cache after bundle install, or gem cleanup), sha is None → Default → not stale. The installed gems/<leaf> still holds upstream bytes, but there's no warning and it's attested. On main judge_installed_record against the fetched view's afterHash flagged it.
  • PyPI (scan/hosted/python.rs): an sdist pin (URL not ending .whl) is filtered out of candidates entirely, and a failed wheel download / sha256 check falls to InstalledRecordJudgment::default(); both were judged against the view on main.

Fix: when the archive/wheel evidence is missing or unfetchable, fall back to fetching the record (the VEX plan fetches it anyway), or put the purl in known_stale / drop it from assume_applied rather than treating it as clean.

Base automatically changed from refactor/vendor-refusals-into-core to main November 6, 2026 22:13
@mikolalysenko Mikola Lysenko (mikolalysenko) removed the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Nov 6, 2026
Main squash-merged this branch's stacked base commits as #2013 and #2015
with review changes, so the merge conflicted in the vlt heal code and
tests. Resolved by taking main and re-applying only this PR's own two
commits on top, which apply cleanly.

Co-Authored-By: Claude <noreply@anthropic.com>
The record-fetch-failure test that mounted a 404 mock is gone, so
Mock, method and path were left unused after merging main.

Co-Authored-By: Claude <noreply@anthropic.com>
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 4298af6. Configure here.

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Nov 6, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Re-labeled Ready for review by the burn-down agent after resolving merge conflicts.

  • Head: 4298af6
  • Conflicts: main squash-merged this branch's stacked base commits as Stop downloading vlt artifacts during hosted scans #2013 and Refuse vendored patches the same way everywhere #2015, so the merge conflicted in the vlt heal code and tests. The merge (1c97b7f) takes main and re-applies only this PR's own two commits (9bcb4c5, fb3d418), which apply cleanly on top. A follow-up commit drops the Mock/method/path wiremock imports that became unused once the record-fetch-failure test went away.
  • CI: 50/50 check runs settled green (31 success, 19 skipped)
  • Bugbot: reviewed 4298af6, no findings
  • Locally: cargo test -p socket-patch-core --lib and the in_process_redirect / hosted_memory_engine tests pass, except 7 chmod-based write-failure tests that cannot fail as root in the agent container (CI runs them green).
  • Reviewer note: the diff against main is now only the patch-view removal and the wheel probe change. The vlt rule (c) heal now reports "undeterminable" when there is no hidden lock, rather than using a record.
  • Slack announcement could not be sent from this run (no Slack send tool available).

Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Final reviewer: blocked on an open review thread, needs an owner decision (agent:needs-human). Head 4298af60.

The review thread on hosted.rs:405 is unresolved and has no reply. I checked it against the head: when there is no agent-mode record and no <home>/cache/<leaf>.gem, the gem probe (hosted.rs:395-405) returns InstalledRecordJudgment::default(), which means "not stale". On main the fetched patch view's afterHash judged that case. The thread says the PyPI path has the same gap (an sdist pin, or a failed wheel download or sha256 check). It also says in-run VEX then attests the purl through assume_applied.

Question for the owner: when the archive or wheel evidence is missing or can't be fetched, which should the scan do?

  1. Fall back to fetching the patch-view record for that purl only. This keeps the old safety, but it puts back some of the fetches this PR removes.
  2. Treat the purl as unjudgeable. Add it to known_stale or drop it from assume_applied, so VEX doesn't attest it. No fetch, but it may give false "stale" warnings on images that prune caches.

CI is green (31/31 non-skipped) and Bugbot is clean on this head, so this thread is the only thing blocking it. I haven't posted a review brief.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:needs-human Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants