Skip to content

8 of 9 docker agent e2e suites serve an all-zeros beforeHash and apply with --force, so they only prove the --force path; port gem's real-beforeHash probe #1970

Open

Description

[agent] Filed by the scheduled architecture audit (audit-cli-tests). Register: register comment.

Kind: refactor (test honesty). Register ID: T62.

Problem

The docker agent suites (docker_e2e_<eco>.rs) are the only end-to-end proof that scan → scan --sync → apply works against a real installed package. Eight of the nine serve a placeholder "beforeHash": "000…0" from their wiremock fixture and then run apply --force. For example, golang has the placeholder at L98 and the force apply at L194.

The same pattern, a 64-zero beforeHash plus --force, appears in cargo, composer, deno, golang, maven, npm, nuget and pypi (grep-counted on cbbbdb9).

gem is the exception, and its own doc explains why the pattern is a problem (docker_e2e_gem.rs L88-L95):

Serving this as the fixture's beforeHash — instead of an all-zeros placeholder — is what lets the gated apply paths run: with the placeholder, scan --sync's nested apply hit the variant gate and failed invisibly, so the chain only ever proved the --force path.

As a result, in the other eight suites the nested apply that scan --sync performs is expected to fail. The suites log its exit code instead of gating on it (golang L180: "Exit code is logged for…"). The gated, non-force apply path (beforeHash match → write → afterHash verify) has no container-level proof for those ecosystems.

Proposed change

  1. Generalise gem's upstream_before_hash probe, a once-per-process probe container that computes the real git-blob sha256 of the installed file, into the shared docker harness (see Share one docker test harness: cov_docker_args ×10, skip_if_no_image ×11 and assert_vex_agent_attested ×8 are copied, and 8 docker suites never honour SOCKET_PATCH_DOCKER_E2E_REQUIRED #1968).
  2. Serve the real hash in each suite's mock.
  3. Gate on the nested scan --sync apply's exit code and summary.applied.
  4. Keep at most one --force leg per suite, and name it as a force test.

Do one ecosystem per PR, or all eight at once if CI time allows.

Size

8 test files. About +15 / −10 lines each, plus about 40 shared lines.

Acceptance criteria

  • In each suite, the non-force chain passes in coverage-docker with scan --sync reporting applied: 1.
  • Setting the served beforeHash back to zeros makes the non-force leg fail.
  • Any remaining force legs are named *_force_*.

Related: #1968 (T60).

Activity

  1. added
    arch-auditFiled by a scheduled architecture audit routine (see the architecture review discussion)
    refactorStructural change: duplicated code or logic, missing abstraction, layering, dead code
    on Nov 2, 2026
  2. mikolalysenko commented on Nov 2, 2026

    @mikolalysenko
    CollaboratorAuthor

    [agent] Triaged: priority:p3 (docker e2e test honesty). Not a duplicate. Step 1 of the proposal depends on the shared docker harness from #1968; a fix should land after or alongside it. Separate root cause, so not clustered.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent:triagedarch-auditFiled by a scheduled architecture audit routine (see the architecture review discussion)priority:p3refactorStructural change: duplicated code or logic, missing abstraction, layering, dead code

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions