ci: gate main on the three regression suites, and make a skipped suite a failure - #418
Open
swapnilpaliwal-sd wants to merge 8 commits into
Open
swapnilpaliwal-sd wants to merge 8 commits into
swapnilpaliwal-sd wants to merge 8 commits into
Conversation
…e a failure
main had no gate. The suites only ran when somebody remembered to run them, and
running them is not cheap enough to be reliable as a habit.
Three required tiers: build and typecheck; the parser-free repo invariants
(no-ignored-fixtures, the staging guard for all three front ends, shell syntax);
and the java / python / typescript suites in parallel.
The part that needed care is that these suites are written for a laptop, where a
missing dependency is a good reason to step aside: they print SKIP and exit 77.
On CI that is the one outcome that must never be tolerated, because a gate that
opens when its input is missing is worse than no gate — the green gets read as
evidence. Two instances that would otherwise have gone unnoticed:
- the parser is an external repository, and without it all three suites exit 77;
- test/java/torture reads class files with java.lang.classfile, which is not
final before JDK 24, so on the runner's default JDK the ten-family oracle
exits 77 while the suite still reports success.
So .github/scripts/run-suite.sh converts every shape of "did not actually run"
into a failure: exit 77, any sub-harness that swallowed its own 77, and a suite
that reports "passed 0" and therefore asserted nothing. CI installs JDK 24 and
pinned Python 3.10 rather than relaxing that check.
The parser is pinned by SHA in .github/parser-ref. The goldens are a function of
both these rules and the IR, so tracking the parser's main branch would let a
change in the other repository turn this one red with no commit here to point at,
and would let a parser regression quietly become the new expectation. A nightly
workflow runs against the parser's main and opens an issue when the pin falls
behind, so the pin cannot rot unnoticed.
Soufflé is pinned to 2.5 for the same reason: its evaluation changes between
minor versions, and an unpinned solver makes every golden a moving target.
Branch protection is not applied here because it cannot be: rulesets are refused
with 403 on a private repository in a Free organisation. It is written as
.github/scripts/protect-main.sh, to run on the day this repository goes public,
when rulesets become free. Until then main-guard.yml records any commit that
reaches main without a pull request — detection rather than prevention, and the
file says so and says to delete it once the ruleset exists.
Also adds the contribution path for people who are not maintainers: issue
templates, a pull request template, CONTRIBUTING.md, CODEOWNERS, and build
badges on the README.
Fixes #417
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-ups on the review of the previous commit. The workflow was invalid and no job had ever started: a GitHub expression may only delimit strings with single quotes, and join(needs.*.result, " ") used double ones, which invalidates the whole file. actionlint catches this class of fault in a second and would have caught it before the push. Ground truth, not only goldens. A golden says "the same as last time", which a wrong answer satisfies perfectly well as long as it was wrong last time too, so java and typescript now run with --oracle: scored against javac and javap, and against the TypeScript compiler. That needs `npm ci` in the engine job — the compiler comes from this repo's own devDependencies, and without it every case reports "oracle refused" while the run still looks like it did something. No third-party library is downloaded to build ground truth. A case whose oracle would need an external classpath reports itself unscored instead. Soufflé is the solver the engine is compiled and linked against, so it is pinned and its package verified against the checksum upstream published for that release: what CI links is decided in the workflow file rather than by whatever the archive serves that day. The package is cached, so a run does not depend on that download succeeding twice. The parser is fetched with a token rather than a deploy key. CODEOWNERS now lists everyone on the project. CONTRIBUTING.md is removed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
This Fails. Working on this. |
…ary coverage Two faults found while getting the suites to run on a clean machine. THE TORTURE FAMILIES CANNOT BE SCORED WITHOUT THE PLATFORM IR. The harness stages the JVM platform IR as a library because half the families call java.util.List, Map and the functional interfaces; with it absent those receivers cannot be typed by any rule. Measured: the missing-edge census goes from 10 to 23 and recall vs possible falls to 0.847. The harness already says this in its own warning — the score then measures the staging, not the rules. That IR is 1.8 GB and is built from a JDK source checkout, so it cannot live in a repository, a cache, or a runner. Running the families anyway would paint a red that reads as a regression in whatever change happened to meet it, so CI passes --no-torture and the suite PRINTS that the families were excluded. An excluded family must never be mistakeable for one that passed. Java client->library resolution is still covered: six cases ship their own stub library in lib-src/ and are solved with it as --library. CLIENT->LIBRARY COVERAGE CAN VANISH WITHOUT ANYTHING NOTICING. Most assertions here are client->client. The client->library half rests on fewer fixtures, and deleting one of its goldens does not break anything: the case still runs, still passes, and quietly stops making the claim. A suite can only check the assertions it still has. test/tools/lib-coverage.sh pins the shape of that coverage — every TypeScript case with a lib/ has BOTH goldens, since the delta between them IS the client->library mapping; every Java case with a lib-src/ has its golden; and the counts cannot fall without lowering a floor in the same commit. It needs no parser and no solver, so it runs in seconds alongside the other invariants. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The secret says what it is rather than leaving "token" to carry the meaning. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
git reports every credential problem against a private repository as "remote: Write access to repository not granted ... 403", whatever the real cause was. It is not about write access, and the token is often not the thing that is wrong, so the message sends you looking in the wrong place. Ask the API first and name the fault: 401 invalid, 403 policy, 404 invisible — and 404 is what GitHub returns rather than confirming a private repository exists, so it is equally what a token with the wrong RESOURCE OWNER looks like. That last one is the trap, so the step also reports how many repositories the credential can reach and how many are the organisation's. None means the token belongs to a personal account. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The torture family failed in CI with nothing but "trace failed", because the harness sends the tracer's output to /dev/null. Run directly, the cause is exact: client/f27_with_target.py does `from typing import Self`, and typing.Self is 3.11+ (PEP 673), so on 3.10 the tracer dies at import and the whole client+lib family — the only python coverage of a library boundary — scores nothing. 3.10 cannot simply be dropped: the tier-1 attribution preflight reads CPython opcodes, whose shapes are not stable across minor versions, and it resolves python3.10 by name. So both are installed, 3.12 second because the later setup-python wins for plain python3, and a step prints both so a future divergence is visible in the log rather than inferred from a failure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Build succeeds! This has access to the parser repo (temporarily). Remove it once these two repos are merged into one. |
Contributor
Author
|
Keeping this open because we are adding more languages and more suites. |
This was referenced Sep 14, 2026
Merged
swapnilpaliwal-sd
force-pushed
the
main
branch
from
September 14, 2026 07:41
f1827c7 to
febf8b4
Compare
This was referenced Sep 14, 2026
Contributor
Author
|
For the update to the current tree (this branch predates the reshape, PR #469):
|
…layout with the in-repo parser, four suites (javascript added), the decls-vs-schema drift check, and every platform's engine build through build-engines.yml Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…and uses no npm cache Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Three items from #579 (closed in favour of this) worth folding in:
|
This was referenced Sep 14, 2026
Contributor
Author
|
On hold: not to be merged. The repository owner has held the npm engine packaging, the bundle change and the CI gating from main for now. Recorded here because several sessions are working this repo concurrently and can merge. Please do not merge this, and do not merge it on someone else's behalf. It is not a review finding and says nothing about the change's quality. |
swapnilpaliwal-sd
force-pushed
the
engine-prebuilt
branch
from
September 19, 2026 07:40
aac70d7 to
8dbac96
Compare
swapnilpaliwal-sd
force-pushed
the
ci/github-actions
branch
from
September 19, 2026 07:40
e031a1c to
8df2c1c
Compare
swapnilpaliwal-sd
added a commit
that referenced
this pull request
Sep 24, 2026
#1258 dropped both when it restructured the header, leaving Build, License and Node. They are status, not decoration: engines-not-yet-published is the one line that tells a reader why the Requirements paragraph still asks for Soufflé, and nightly-not-yet-enabled says the scheduled run is not there rather than passing. Restored as they were, static shields rather than workflow badges — which is deliberate, per the commit that made the nightly badge render by not asking for a run that has not happened. Worth a look before release: the engines badge points at #478, which is merged. The machinery landed; the publish has not run. #418 is where the CI arrives, and is the honest target for both.
swapnilpaliwal-sd
added a commit
that referenced
this pull request
Sep 25, 2026
Carries #418 onto current main as files rather than a rebase: its branch merged engine-prebuilt, whose other changes main already has, so only .github/, graph/test/tools/lib-coverage.sh and the java suite's --no-torture flag were new. Beyond #418: - C# joins the gate: a suite leg scored against the Roslyn oracle (.NET 8), its staging map and its decls-vs-parser-schema check. - C# joins build-engines, so the engine packages carry all five languages. - protect-main.sh points at the renamed repository and also makes v* tags immutable (creatable, never moved or deleted). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
swapnilpaliwal-sd
added a commit
that referenced
this pull request
Sep 25, 2026
Carries #418 onto current main as files rather than a rebase: its branch merged engine-prebuilt, whose other changes main already has, so only .github/, graph/test/tools/lib-coverage.sh and the java suite's --no-torture flag were new. Beyond #418: - C# joins the gate: a suite leg scored against the Roslyn oracle (.NET 8), its staging map and its decls-vs-parser-schema check. - C# joins build-engines, so the engine packages carry all five languages. - protect-main.sh points at the renamed repository and also makes v* tags immutable (creatable, never moved or deleted).
swapnilpaliwal-sd
added a commit
that referenced
this pull request
Sep 25, 2026
Carries #418 onto current main as files rather than a rebase: its branch merged engine-prebuilt, whose other changes main already has, so only .github/, graph/test/tools/lib-coverage.sh and the java suite's --no-torture flag were new. Beyond #418: - C# joins the gate: a suite leg scored against the Roslyn oracle (.NET 8), its staging map and its decls-vs-parser-schema check. - C# joins build-engines, so the engine packages carry all five languages. - protect-main.sh points at the renamed repository and also makes v* tags immutable (creatable, never moved or deleted).
swapnilpaliwal-sd
added a commit
that referenced
this pull request
Sep 25, 2026
…t release per bump; npm on publish (#1318) * ci: gate main on build, repo invariants and all five language suites Carries #418 onto current main as files rather than a rebase: its branch merged engine-prebuilt, whose other changes main already has, so only .github/, graph/test/tools/lib-coverage.sh and the java suite's --no-torture flag were new. Beyond #418: - C# joins the gate: a suite leg scored against the Roslyn oracle (.NET 8), its staging map and its decls-vs-parser-schema check. - C# joins build-engines, so the engine packages carry all five languages. - protect-main.sh points at the renamed repository and also makes v* tags immutable (creatable, never moved or deleted). * release: one version everywhere, a tag and draft release per bump, npm on publish - version.mjs sets and checks the version in all ten places that carry it (root, its engine pins, parser, gemini and the three plugin manifests). - The version gate (from #1213) now also requires every manifest to agree, a bump to move forward, and the new version not to be tagged already. It only demands a bump once the base version is tagged, so work before the first release joins 0.1.0. - release.yml: a push to main whose version has no tag gets one, plus a draft release with generated notes. Nothing is published. - publish-npm.yml runs when that draft is published: checks tag == manifests, builds the engines, publishes them and then @axiomcode/code-graph (which was never published before), prereleases under `next`, skips versions already on the registry, and attaches the tarballs to the release. - .github/RELEASING.md is the runbook. * test/java: the oracle-agreement golden counts case 60 Case 60 (#1269) added a case both ground-truth oracles compare, without re-recording the agreement golden, so `run-tests.sh --oracle` on main aborted before running any case: 45 compared, 45 agreeing, against a golden of 44/44. No disagreement changed. * release: publish with the CLI_BINARY_PUBLISH secret, the npm token already set on the repo * build: darwin-arm64 on the hosted macos-15 runner, every run A public repository gets GitHub's standard runners free, Apple Silicon macOS included, so the self-hosted runner and the macos switch that skipped it are gone: CI and every publish build all four platforms. * ci: no approval required; only the admin role merges into main GitHub never lets an author approve their own pull request, so a required approval would block the sole maintainer. The PR and CI rules still bind everyone; a separate ruleset restricts updates to main to the admin role, through a pull request only. * ci: run on pushes to dev, which takes direct pushes * release: merge main back into dev after every push to main main squash-merges, so a promotion lands as a commit dev lacks and the next dev->main PR would repeat it. A clean merge is pushed to dev; a conflict (a hotfix that touched lines dev changed) pushes nothing and opens one issue with the commands to resolve it by hand. * Linux: an apostrophe in a # comment, and a sandbox PATH that kept /bin Two faults only a Linux runner shows, found on the first CI run: - Soufflé on Linux preprocesses with mcpp, which tokenises the text of a trailing # comment; "every on's listener" is an unterminated character constant there, so every TypeScript solve and the engine generate step failed. macOS's clang preprocessor accepts it. Reworded; all five languages' programs now pass mcpp. - engine-id-test.sh hid souffle by dropping its directory from PATH, compared logically. On merged-/usr Ubuntu /bin -> /usr/bin survives that, and dropping /usr/bin itself would take bash with it. Directories are now compared with pwd -P and souffle's is replaced by a shadow holding everything else. * test/tools: one souffle-hiding helper, correct on merged-/usr Linux engine-package-test.sh carried the same PATH sandbox as engine-id-test.sh and failed the same way on Ubuntu: /bin -> /usr/bin kept souffle visible, so "no souffle" runs found it. Both now source hide-souffle.sh, which compares directories with pwd -P and shadows souffle's directory instead of dropping it. Proven on a simulated merged-/usr layout: the old block leaves souffle visible, the helper hides it and keeps sh. * ci: a docs-only change runs build and hygiene only; platform engines only when what they compile changed The workflow still starts on every event, so the required CI check always reports; a changes job classifies the diff and the engine suites and the platform build skip themselves. The CI job accepts a skip only where changes asked for one. Markdown under graph/ and parser/ still counts as code, since graph/bundle/SCHEMA.md is generated and checked. Pushes to main, merge queues and manual runs run everything. * ci: dev is the default branch, a nightly from scratch, engines cached by ENGINE_ID - dev takes every pull request: build, hygiene and the five suites. The four-platform engine build runs on the way into main (a promotion PR or a push to main) and in the nightly, not on every change to dev. - protect-main.sh names refs/heads/main instead of ~DEFAULT_BRANCH, so main's protection stays on main now that dev is the default, and dev gets a ruleset that only forbids deleting it. - nightly.yml: on nights dev changed, CI with fresh=true (no restored engines, every platform), then e2e-install.sh packs the tarballs, installs them into an empty project without Souffle and runs axiomcode in four languages, then npm publish --dry-run for every package. Publishes nothing; a failure opens an issue and the next green night closes it. - The suite cache never hit: the driver writes to ~/.cache/axiomcode/souffle while CI saved .souffle-cache. The suites now point the driver there and key it by the language's ENGINE_ID. - build-engines restores the previous engines per platform and recompiles only languages whose ENGINE_ID changed (about 3 min each at -O3); fresh (nightly, publish) restores nothing. Its Souffle download is now checked against the same SHA-512 as ci.yml. * ci: cache keys cover the compile flags and, for -march=native, the CPU ENGINE_ID hashes the rules and the Souffle version, not how the binary is compiled, so a flag change reused binaries built with the old flags. - build-engines: the hash of build-engines.yml (which holds the flags) prefixes both the key and the restore prefix, so a flag change restores nothing. Computed in generate, since the build jobs never check out. - suites: the key adds the hash of run-souffle.sh (the driver's flags) and the runner's CPU model: the driver compiles with -march=native, and a binary built on one CPU can die with an illegal instruction on another. * nightly: dev's daily status, published under the nightly dist-tag when green - Runs every night, commits or not: the status is daily, and with no lock file a dependency release can break a fresh install with nothing committed. - A green night publishes every package as <next>-nightly.<date>.g<sha> under `nightly`, never `latest`. The version is computed in the run and never committed or tagged; the g keeps an all-digit sha a valid semver identifier. Skipped when that commit is already the nightly, and until a first release exists, since npm makes a first publish `latest`. - README: the static "engines: not yet published" and "nightly: not yet enabled" badges become the live npm version and nightly status. * test/csharp: one compiled engine per run, not one per case; admins can merge - devrun.sh caches its compiled engine beside the work dir it is given, and run-tests.sh hands every case a fresh one (rm -rf "$w"), so each of the 20 cases and the cross-service cases recompiled the same engine, about 70s each: the suite took 32-38 min in CI. run-tests.sh now exports one AXIOM_CS_DEV_CACHE for the run (in CI, inside the saved engine cache). The cache is content-addressed by the rule text, so sharing it is safe. Locally: three cases, one compile and two reuses, 95s in total. - main-merge-admins bypass is `always`: in `pull_request` mode GitHub refused the merge itself ("Cannot update this protected ref"), even for admins. protect-main still has no bypass, so PR and CI bind admins too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #417
What changed
maingets a gate. Four required tiers, cheapest first: build and typecheck (parser anddriver, from this repository); the solver-free repo invariants (
no-ignored-fixtures, thestaging guard across all four front ends, the engine's base declarations against the
parser's generated schema, client→library coverage floors, shell syntax); the java /
typescript / python / javascript regression suites in parallel; and every language's
engine built on every platform through the reusable
build-engines.ymlfrom #478, thesame artifacts
publish-npmships, built here without publishing.Based on
engine-prebuilt(#455, which now carries #478), so the build workflow itcalls exists on the base. Lands after the C# engine, together with that base.
The part that needed care
These suites are written for a developer machine, where a missing dependency is a good
reason to step aside: they print
SKIPand exit 77. On CI that is the one outcomethat must never be tolerated, a gate that opens when its input is missing is worse than
no gate, because the green is read as evidence by whoever comes next.
Two instances, both of which would have gone unnoticed:
test/java/torturereads class files withjava.lang.classfile, which is not finalbefore JDK 24. On an older JDK it exits 77 and the whole ten-family oracle does not
run, while the suite still reports success
.github/scripts/run-suite.shturns every shape of did not actually run into afailure: exit 77, any sub-harness that swallowed its own 77, and a suite reporting
passed 0, which means the case loop matched nothing and the exit status says nothing.CI supplies JDK 24 and pinned Python 3.10 rather than relaxing that check.
Ground truth, not only goldens
A golden says "the same as last time", which a wrong answer satisfies perfectly well as
long as it was wrong last time too. Java and TypeScript therefore run with
--oracle,scored against the toolchain that defines the language, javac and javap, and the
TypeScript compiler, with no third-party analyzer and no third-party library downloaded
to do it. A case whose ground truth would need an external classpath reports itself
unscored rather than pulling one in.
Python is goldens-only for now: its ground truth is frozen CPython output authored by a
separate harness that CI cannot reach yet. That is a weaker check than the other two
legs, and the workflow says so where it matters.
Pins, and why
parser/) since the reshape, built bynpm install'sprepare script; there is no longer a separate checkout, token or commit pin, and the
nightly drift workflow is gone with them. The goldens are still a function of both the
rules and the IR, now both are in one diff.
Soufflé is the solver the engine is compiled and linked against, so what CI links is
decided in the workflow file rather than by whatever the archive serves that day.
Branch protection
.github/scripts/protect-main.shholds the ruleset formain: pull request required,an approving review, code-owner review,
CIgreen against an up-to-date branch, linearhistory, no force-push, no deletion, and no bypass actors. It is idempotent and applied
separately from this pull request.
main-guard.ymlreports any commit that reachesmainwithout a pull request. It is abackstop, not the rule, a workflow runs after a push has been accepted, so it can record
a direct push but never refuse one.
Evidence
Run locally through
.github/scripts/run-suite.sh, on this branch, with the in-repo parser:--oracle --no-torture:passed 39 failed 0(torture EXCLUDED, printed as such)--oracle:passed 53, failed 0passed 15 failed 0--oracle:passed: 19 failed: 0, tsc over allowJs/checkJs from thisrepo's devDependencies; the wrapper reads the JS suite's
passed:line toostart when
parser/dist/index.jsis absentnpm install, notnpm ci: no lock file is committed (bundle stage runs from any working directory (tsx --tsconfig); lock files uncommitted #476)This pull request's own CI run exercises the workflow, since
pull_requestevaluates itfrom the merge ref.
Checklist
Two faults found while getting the suites onto a clean machine
The java torture families cannot be scored without the platform IR. They call
java.util.List,Mapand the functional interfaces, so the harness stages the JVMplatform IR as a library. Removing it takes the missing-edge census from 10 edges to 23
and recall to 0.847, the harness itself warns that the score then measures the staging,
not the rules. That IR is 1.8 GB and is built from a JDK source checkout, so it cannot
live in a repository, a cache, or a runner. CI passes
--no-tortureand the suiteprints EXCLUDED, because an excluded family must never be mistakeable for one that
passed. Java client->library resolution is still covered by the six cases shipping a
lib-src/stub library.Client->library coverage could vanish unnoticed. Delete one of those goldens and
nothing breaks: the case still runs, still passes, and quietly stops making the claim.
test/tools/lib-coverage.shpins the shape, both goldens per TypeScript lib case, agolden per java stub case, floors that cannot fall without being lowered in the same
commit. Parser-free, runs in seconds.
Coverage as it now stands, all of it running in CI:
lib-src/stub libraryCI status on this branch: green
passed 39 failed 0passed 40, failed 0passed 15 failed 0Under four minutes wall clock, and scoring against ground truth rather than only the
goldens, java reports per case
oracle=15 engine=15 agree=15 missing=0, typescript thesame with the client->library half beside it, and the python torture links client and lib
for real at
oracle=623 engine=613 agree=577. The java torture line readsEXCLUDED (--no-torture), visible and never mistakable for a pass.Two faults a clean machine found that a developer machine could not
The python torture family scored nothing on 3.10. CI reported only
trace failed,because the harness sends the tracer to
/dev/null. Run directly the cause is exact:client/f27_with_target.pyimportstyping.Self, which is 3.11+ (PEP 673), so thetracer died at import and the only python coverage of a library boundary quietly
measured nothing. 3.10 cannot simply be dropped, the tier-1 attribution preflight reads
CPython opcodes, whose shapes are not stable across minor versions, so CI installs both
and prints both versions.
Every credential fault reports as
remote: Write access to repository not granted,whatever the real cause was. It is not about write access. CI now asks the API first and
names the fault, invalid, policy-blocked, or invisible, and reports how many of the
organisation's repositories the credential can actually reach, because a token owned by
a personal account looks identical to an unapproved one from the outside.