1.3.0: tests as mcpp states them, the backend's tests on mtest, a backend change begins a series, mcppls 0.21

.agents/docs/2026-11-10-tests-from-mcpp.md

@@ -0,0 +1,183 @@
+# Tests as mcpp states them: programs, parts, cases, live results, metrics and history
+
+Status, 2026-11-10: implemented in mcpp-ide 1.3.0 (mcpp-ide#8), with the client side of mcpp#917; the backend's own tests are mtest's.
+
+## Scope and principle
+
+The editors show tests as `mcpp test` states them. mcpp builds and runs the programs. mtest's protocol
+is how a program's cases report to mcpp. mtest's store (spec P12) is the form in which mcpp keeps a run
+(`target/mtest`). The backend reads three inputs:
+
+- the records of `mcpp test --message-format json`;
+- the listings of `--list` and `--cases`;
+- the store.
+
+It does not drive mtest's engine. A program that only mtest's own runner runs (a `[discover]` script of
+`mtest.toml`) is outside `mcpp test`, and the editors do not show it.
+
+Each editor presents the same data in its own idiom:
+
+- VS Code uses the Testing view, test messages, and editor decorations.
+- CLion uses the SMRunner test tree and its comparison view.
+- Neovim uses buffers, signs, virtual text, and the quickfix list.
+
+An editor shows only what the project's mcpp provides. A missing input removes the view that needs it.
+It does not remove the views that do not need it.
+
+## What mcpp provides, by version
+
+| mcpp | Provides | The editors show |
+|---|---|---|
+| before 2026.11.8.1 | one record per program | programs and their results |
+| 2026.11.8.1 (`test-protocol`) | `listed` and `cases` of a protocol program | cases, their results at the program's end, their failures' places, `kind`, `tags`, `expect` and `expected`/`actual` |
+| 2026.11.9.1 | a protocol program of several files runs as one process per file (`<program>/<file>`); the run is kept in `target/mtest` | a program of several files with its files as children; results of a part attach to its program; history, trends of durations, and the schedule from the store |
+| `test-parts` (mcpp#917) | `files`, `parts`, `main: null`, `part_of` | the same tree, from mcpp's statement instead of the backend's derivation |
+| `test-case-stream` (mcpp#917) | `case` records as cases begin and end | live results per case |
+| `test-metrics` (mcpp#917) | `metrics`, `bounds`, `usage`; the store's P12 fields | metrics beside their bounds, their trends, and a case-level schedule |
+
+A view that needs an input names the missing feature in `capabilities/explain` (`tests.parts`,
+`tests.live`, `tests.metrics`).
+
+## The model
+
+A **program** is what `mcpp test --list` names: `unit` for the directory `tests/unit/` under
+`[test] programs = "directory"`, and `00-a/0` for one file. Its fields are:
+
+- `files`: its own sources;
+- `main`: the file that defines `main`, if one does;
+- `layout`: `file` or `directory`;
+- `parts`: the processes a run divides it into, each `{name, file}`.
+
+With `test-parts`, these fields are mcpp's. Without it, the backend derives them from what mcpp
+2026.11.9.1 states (docs/08):
+
+- A program whose `main` file lies in a directory whose path below the test root is the program's name
+  is of layout `directory`.
+- Its files are the C++ sources directly in that directory.
+- Its parts are `<program>/<stem>`, one per file.
+- `main` is the file whose source defines `main`, found by the scanner; otherwise it is absent.
+
+A **case** belongs to a program and a file. Its fields are `id`, `file`, `line`, `kind`, `tags`,
+`expect` and, with `test-metrics`, `bounds`. Static cases are read from each file with mtest's manifest.
+A listing (`--cases`) replaces them case by case.
+
+A result names the program it belongs to:
+
+- `part_of` when mcpp states it;
+- else the longest listed program whose name is the record's name or a prefix of it followed by `/`;
+- else the record's name.
+
+## The backend's protocol (protocol 2, additions only)
+
+- `test/list`: each test gains `files`, `layout`, `parts`, and `main` as described above.
+- `source/entries`: a file of a directory program answers `test` (the program) and `part` (its own
+  part).
+- `test/cases`: reads every file of the program. Each case gains `file`, `kind`, `tags`, `expect` and
+  `bounds`.
+- `test/result` gains:
+  - `program` and `part`;
+  - `cases`, each with `id`, `status`, `durationMs`, `reason`, `failures` (`path`, `line`, `message`,
+    `expected`, `actual`), `metrics`, `bounds`, `kind` and `tags`.
+
+  A case's status is mapped as follows:
+
+  | mcpp | protocol |
+  |---|---|
+  | `pass` | `passed` |
+  | `fail` | `failed` |
+  | `skip` | `skipped` |
+  | `xfail` | `passed` with `expectedFailure` |
+  | `xpass` | `failed` with `unexpectedPass` |
+  | `invalid` | `errored` |
+
+  A whole-program run therefore marks every case.
+- `test/caseResult`: gains the fields of a case above. It is sent for every record of the job, not
+  only the last one; a run of a directory program writes one record per part.
+- `test/caseEvent` (new): `{jobId, root, program, part, case, event: "begin" | "end", tMs, ...}` with
+  the fields of a case on `end`. It is sent as mcpp writes a `case` record (`test-case-stream`).
+- `test/history {root, program?, case?, runs?}` (new): the store's last runs (20 by default) and each
+  selected case's results across them.
+- `test/insights {root, runs?}` (new):
+  - the last run: its counts and its counts by kind;
+  - its failures, each with the run since which it has failed;
+  - the slowest cases, with their median over the window;
+  - the cases whose status changed within the window (flaky);
+  - each metric's series, its bounds, its last value and its margin;
+  - the schedule: one lane per worker, each process with its start, duration and build, and the
+    critical path (the chain of processes on one worker that ends last, with the build before it).
+- `capabilities/explain` gains `tests.parts`, `tests.live` and `tests.metrics`.
+
+The store is read from `[test] store` when the manifest sets it, else from `target/mtest`. The backend
+reads it as P12 specifies it. It reads `index.ndjson`, then the last N runs' files, and keeps them by
+run id. A run already read is not read again. A store that is absent is not an error: the history and
+insights are empty and say so.
+
+## Each editor
+
+**VS Code.**
+
+- The Testing view shows each project, then its programs, then the files of a directory program, then
+  the cases.
+- Each case carries the test tags `kind:<kind>` and its own tags, so `@kind:bench` filters the view.
+- Results arrive per case as they happen (`test/caseEvent`), else per part.
+- A failure with `expected` and `actual` is a `TestMessage.diff`, and VS Code's diff view compares them.
+- After a run, a case's line ends with a dim decoration: its duration, and its metrics against their
+  bounds (`ready 4.2 s ≤ 15 s`).
+- "mcpp: Test Insights" is a webview with these sections:
+  - the run's summary;
+  - its failures, each opening its place;
+  - the slowest cases;
+  - the flaky cases;
+  - each metric's trend against its bounds;
+  - the schedule, with its critical path.
+
+  It refreshes when a run ends.
+
+**CLion.**
+
+- The SMRunner tree shows a program as a suite, the files of a directory program as nested suites, and
+  the cases as tests, each located at its line.
+- Live case events start and finish tests as they come.
+- A failure with `expected` and `actual` is a comparison failure, whose "Click to see difference" opens
+  the IDE's diff.
+- Metrics appear in a case's output beside their bounds.
+- The mcpp tool window gains a Tests tab: the same insights in JCEF (`webui/tests.html`), as the graph
+  page is drawn.
+
+**Neovim.**
+
+- `mcpp://tests` gains sections per program, part and case as results arrive, live with
+  `test-case-stream`.
+- A failure's `expected` and `actual` appear as two lines under it, and in its quickfix text.
+- A case's line carries its sign (✓, ✗) and, as virtual text, its duration and its metrics against
+  their bounds.
+- `:Mcpp testInsights` opens the insights as text in a floating window. A metric's trend is drawn with
+  block characters (▁▂▃▅▇), and the schedule as bars.
+
+## Performance
+
+- The suite types of a project's tests are read once and kept with a signature of the files: their
+  count, their largest modification time, and their total size, from a walk that only stats.
+  `source/entries` and `test/cases` no longer read every file of `tests/` on each call.
+- Live results are sent as they come. VS Code and CLion apply them as their test APIs do.
+- Neovim redraws the results buffer at most every 100 ms.
+- The store is read incrementally: only runs not read before.
+- The insights window is bounded: 20 runs by default, 50 at most.
+
+## Tests
+
+- Backend:
+  - the derivation of parts from a 2026.11.9.1 listing;
+  - the attribution of part records to their program;
+  - the mapping of the protocol's statuses;
+  - `expected`/`actual`;
+  - `case` records into `test/caseEvent`;
+  - the store reader, and the insights (the window, flaky, the critical path, the margins);
+  - the schema of the new methods and notifications.
+- The mock mcpp gains the fixtures of a directory program, with and without the #917 shapes. A
+  directory's `.mock-mcpp-tests` selects them, as `.mock-mcpp-features` selects the features.
+- VS Code: the tree of a directory program; part results attributed to it; the diff message; the
+  decorations' text.
+- Neovim: the smoke test covers live results, the virtual text and `:Mcpp testInsights`.
+- CLion: the comparison failure and the nested suites, in the plugin's unit tests.
+- The insights page is rendered by a Chromium test, as the graph page is.

.github/actions/mtest-report/action.yml

@@ -0,0 +1,19 @@
+name: The tests' report
+description: mtest's Markdown report of the last run `mcpp test` kept (target/mtest, mtest spec P12) -- its failures, its slowest cases, its metrics and its critical path -- in the job's summary.
+runs:
+  using: composite
+  steps:
+    - name: The run, as mtest reports it
+      shell: bash
+      run: |
+        set -uo pipefail
+        [ -f target/mtest/index.ndjson ] || { echo "mcpp test kept no run"; exit 0; }
+        xlings install "mtest@$MTEST_VERSION" -y -g > /dev/null 2>&1 || { echo "::warning::mtest@$MTEST_VERSION could not be installed: no report"; exit 0; }
+        exe=""; [ "$RUNNER_OS" = Windows ] && exe=.exe
+        # The package's launcher, not the shim: on Windows the shim runs its alias through cmd.
+        mtest="$HOME/.xlings/data/xpkgs/xim-x-mtest/$MTEST_VERSION/bin/mtest$exe"
+        [ -x "$mtest" ] || { echo "::warning::no mtest at $mtest: no report"; exit 0; }
+        store="$RUNNER_TEMP/mtest-report"
+        mkdir -p "$store" && cd "$store"
+        "$mtest" store import --from "$GITHUB_WORKSPACE/target/mtest" > /dev/null
+        "$mtest" report --format md --run last --title "$GITHUB_JOB" >> "$GITHUB_STEP_SUMMARY" || true

.github/scripts/release-plan.sh

@@ -1,14 +1,16 @@
 #!/usr/bin/env bash
 # release-plan.sh [TAG] -- the components a release tag publishes, and its version (design 2026-11-07, D15).
 #
-# A version is X.Y.Z: X.Y, the series, is shared by the backend and every editor plugin; Z is each
-# component's own patch. Two kinds of tag:
+# A version is X.Y.Z: X.Y, the series, is shared by the backend and every editor plugin; Z is an editor
+# plugin's own patch. The backend is always X.Y.0: the VS Code and CLion packages carry it, so a change
+# of the backend changes every package, and it begins a series. Two kinds of tag:
 #
-#   vX.Y.0              a series release: every component, each at X.Y.0
-#   <component>-vX.Y.Z  one component's patch: backend, vscode, clion or nvim, at X.Y.Z
+#   vX.Y.0              a series release: every component, each at X.Y.0 (a change of the backend)
+#   <plugin>-vX.Y.Z     one editor plugin's patch: vscode, clion or nvim, at X.Y.Z
 #
 # Prints `components=...`, `version=...` and `title=...` for $GITHUB_OUTPUT. Without a tag it only
-# checks that every component states one series and the backend's two statements agree (CI).
+# checks that every component states one series, the backend's two statements agree and the backend
+# is its series' X.Y.0 (CI).
 set -euo pipefail
 root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
 declare -A version
@@ -26,6 +28,8 @@ for c in backend vscode clion nvim; do
 done
 [ "$backend_source" = "${version[backend]}" ] || fail "src/server.cppm says $backend_source and mcpp.toml ${version[backend]}"
 the_series="$(series "${version[backend]}")"
+[ "${version[backend]}" = "$the_series.0" ] \
+    || fail "the backend is ${version[backend]}; it is always its series' X.Y.0 (the editor packages carry it, so a change of it begins a series)"
 [ "$series_source" = "$the_series" ] || fail "src/server.cppm's kSeries is $series_source, the backend's series $the_series"
 for c in vscode clion nvim; do
     [ "$(series "${version[$c]}")" = "$the_series" ] \
@@ -45,13 +49,31 @@ if [[ "$tag" =~ ^v([0-9]+\.[0-9]+\.0)$ ]]; then
     echo "components=backend vscode clion nvim"
     echo "version=$v"
     echo "title=mcpp-ide $v"
-elif [[ "$tag" =~ ^(backend|vscode|clion|nvim)-v([0-9]+\.[0-9]+\.[0-9]+)$ ]]; then
+elif [[ "$tag" =~ ^backend-v ]]; then
+    fail "$tag: the backend has no patch of its own; a change of it is a series release vX.Y.0"
+elif [[ "$tag" =~ ^(vscode|clion|nvim)-v([0-9]+\.[0-9]+\.[0-9]+)$ ]]; then
     c="${BASH_REMATCH[1]}"; v="${BASH_REMATCH[2]}"
     [ "${version[$c]}" = "$v" ] || fail "the tag $tag names $c $v, and $c is ${version[$c]}"
-    declare -A names=([backend]=mcpp-ide [vscode]=mcpp-vscode [clion]=mcpp-clion [nvim]=mcpp-nvim)
+    # A plugin's patch carries its series' backend: no path of the backend (.github/components.tsv)
+    # has changed since the series' tag.
+    if git -C "$root" rev-parse -q --verify "refs/tags/v$the_series.0" > /dev/null; then
+        mapfile -t backend_rows < <(awk -F'\t' '$1 !~ /^(#|@)/ && NF == 2 {print $1 "\t" $2}' "$root/.github/components.tsv")
+        while IFS= read -r path; do
+            for row in "${backend_rows[@]}"; do
+                pattern="${row#*$'\t'}"
+                # shellcheck disable=SC2254
+                case "$path" in $pattern)
+                    [ "${row%%$'\t'*}" != backend ] \
+                        || fail "$tag: the backend changed since v$the_series.0 ($path); a change of the backend is a series release"
+                    break ;;
+                esac
+            done
+        done < <(git -C "$root" diff --name-only "v$the_series.0" HEAD)
+    fi
+    declare -A names=([vscode]=mcpp-vscode [clion]=mcpp-clion [nvim]=mcpp-nvim)
     echo "components=$c"
     echo "version=$v"
     echo "title=${names[$c]} $v"
 else
-    fail "$tag is neither vX.Y.0 nor <backend|vscode|clion|nvim>-vX.Y.Z"
+    fail "$tag is neither vX.Y.0 nor <vscode|clion|nvim>-vX.Y.Z"
 fi

.github/versions.env

@@ -1,12 +1,14 @@
 # The versions CI builds and tests with, in one file; .github/actions/setup-mcpp exports them.
 # The published bootstrap; the project explicitly selects platform 0.24.
-MCPP_VERSION=2026.11.7.2
+MCPP_VERSION=2026.11.9.1
 # A ref of mcpp-community/mcpp CI builds mcpp from and uses instead, for a contract not yet released.
 # Empty: the release above is used.
 MCPP_SOURCE_REF=
 # The LLVM the backend builds with on macOS and Windows (the targets mcpp reaches there with it).
 LLVM_VERSION=23.1.3
-# The xlings mcpp 2026.11.7.2 bundles, which reads the current index.
+# The xlings mcpp 2026.11.9.1 bundles, which reads the current index.
 XLINGS_VERSION=v2026.10.26.1
 # The oldest VS Code the extension supports (engines.vscode), which its tests run in.
 VSCODE_VERSION=1.91.0
+# The mtest whose report of each `mcpp test` run is the job's summary (.github/actions/mtest-report).
+MTEST_VERSION=0.5.2

.github/workflows/ci.yml

@@ -155,6 +155,9 @@ jobs:
         run: mcpp build
       - name: Test
         run: mcpp test
+      - name: The tests' report
+        if: always()
+        uses: ./.github/actions/mtest-report
       - name: Every string has its translation
         run: mcpp run mcpp-ide-dev -- check-i18n
       - name: No header, no preprocessor branch, no compile-time platform question
@@ -218,6 +221,9 @@ jobs:
         run: mcpp build
       - name: The tests, as ok images, through the runner
         run: mcpp test --profile release --target ${{ matrix.arch }}-openkal
+      - name: The tests' report
+        if: always()
+        uses: ./.github/actions/mtest-report
 
   # The VS Code extension (mide-plugins/vscode), mcpp-vscode 0.9.0 over the backend. Its gates
   # (the registry, the translations, the unit suite) on the three desktop systems; the manifest

.github/workflows/clion-stable.yml

@@ -71,7 +71,7 @@ jobs:
           set -euo pipefail
           version="$(sed -n 's/^mcpplsPluginVersion *= *//p' mide-plugins/clion/gradle.properties)"
           gh release download "v$version" --repo Sunrisepeak/mcpp-language-server \
-            --pattern "mcppls-clion-$version-linux-x64.zip" --dir stable-plugin
+            --pattern "mcppls-clion-$version.zip" --dir stable-plugin
       - name: Start the real IDE and complete the plugin startup chain
         env:
           IDE_VERSION: ${{ matrix.version }}

.github/workflows/release.yml

@@ -6,11 +6,12 @@ name: Release
 # independently after that release exists.
 #
 # WHAT A TAG PUBLISHES (design 2026-11-07, D15; .github/scripts/release-plan.sh): a version is X.Y.Z,
-# X.Y the series every component shares and Z each component's own patch.
+# X.Y the series every component shares and Z an editor plugin's own patch. The backend is always
+# X.Y.0: the VS Code and CLion packages carry it, so a change of it is a series release.
 #
-#   vX.Y.0              a series release: the backend, the VS Code extension, the CLion plugin and
-#                       the Neovim plugin (a source tree, which xim takes from the tag's archive)
-#   backend-vX.Y.Z      the backend's ok-image bundle and package (xim's mcpp-ide)
+#   vX.Y.0              a series release: the backend's ok-image bundle and package (xim's mcpp-ide),
+#                       the VS Code extension, the CLion plugin and the Neovim plugin (a source tree,
+#                       which xim takes from the tag's archive)
 #   vscode-vX.Y.Z       the VSIX (Open VSX)
 #   clion-vX.Y.Z        the CLion ZIP (JetBrains Marketplace)
 #   nvim-vX.Y.Z         the Neovim plugin: the tag's source archive (xim's mcpp-ide-nvim)
@@ -20,7 +21,7 @@ name: Release
 on:
   workflow_dispatch:
   push:
-    tags: ['v*', 'backend-v*', 'vscode-v*', 'clion-v*', 'nvim-v*']
+    tags: ['v*', 'vscode-v*', 'clion-v*', 'nvim-v*']
 
 permissions:
   contents: read
@@ -34,6 +35,8 @@ jobs:
       title: ${{ steps.plan.outputs.title }}
     steps:
       - uses: actions/checkout@v5
+        with:
+          fetch-depth: 0   # a plugin's patch is checked against its series' tag
       - id: plan
         shell: bash   # -eo pipefail: a refused tag fails the step through tee
         run: bash .github/scripts/release-plan.sh "$GITHUB_REF_NAME" | tee -a "$GITHUB_OUTPUT"

.xlings.json

@@ -1,5 +1,5 @@
 {
   "workspace": {
-    "mcpp": "2026.11.7.2"
+    "mcpp": "2026.11.9.1"
   }
 }