diff --git a/.agents/docs/2026-11-10-tests-from-mcpp.md b/.agents/docs/2026-11-10-tests-from-mcpp.md new file mode 100644 index 0000000..f4eaadc --- /dev/null +++ b/.agents/docs/2026-11-10-tests-from-mcpp.md @@ -0,0 +1,183 @@ +# Tests as mcpp states them: programs, parts, cases, live results, metrics and history + +Status, 2026-11-10: implemented in mcpp-ide 1.3.0 (mcpp-ide#8), with the client side of mcpp#917; the backend's own tests are mtest's. + +## Scope and principle + +The editors show tests as `mcpp test` states them. mcpp builds and runs the programs. mtest's protocol +is how a program's cases report to mcpp. mtest's store (spec P12) is the form in which mcpp keeps a run +(`target/mtest`). The backend reads three inputs: + +- the records of `mcpp test --message-format json`; +- the listings of `--list` and `--cases`; +- the store. + +It does not drive mtest's engine. A program that only mtest's own runner runs (a `[discover]` script of +`mtest.toml`) is outside `mcpp test`, and the editors do not show it. + +Each editor presents the same data in its own idiom: + +- VS Code uses the Testing view, test messages, and editor decorations. +- CLion uses the SMRunner test tree and its comparison view. +- Neovim uses buffers, signs, virtual text, and the quickfix list. + +An editor shows only what the project's mcpp provides. A missing input removes the view that needs it. +It does not remove the views that do not need it. + +## What mcpp provides, by version + +| mcpp | Provides | The editors show | +|---|---|---| +| before 2026.11.8.1 | one record per program | programs and their results | +| 2026.11.8.1 (`test-protocol`) | `listed` and `cases` of a protocol program | cases, their results at the program's end, their failures' places, `kind`, `tags`, `expect` and `expected`/`actual` | +| 2026.11.9.1 | a protocol program of several files runs as one process per file (`/`); the run is kept in `target/mtest` | a program of several files with its files as children; results of a part attach to its program; history, trends of durations, and the schedule from the store | +| `test-parts` (mcpp#917) | `files`, `parts`, `main: null`, `part_of` | the same tree, from mcpp's statement instead of the backend's derivation | +| `test-case-stream` (mcpp#917) | `case` records as cases begin and end | live results per case | +| `test-metrics` (mcpp#917) | `metrics`, `bounds`, `usage`; the store's P12 fields | metrics beside their bounds, their trends, and a case-level schedule | + +A view that needs an input names the missing feature in `capabilities/explain` (`tests.parts`, +`tests.live`, `tests.metrics`). + +## The model + +A **program** is what `mcpp test --list` names: `unit` for the directory `tests/unit/` under +`[test] programs = "directory"`, and `00-a/0` for one file. Its fields are: + +- `files`: its own sources; +- `main`: the file that defines `main`, if one does; +- `layout`: `file` or `directory`; +- `parts`: the processes a run divides it into, each `{name, file}`. + +With `test-parts`, these fields are mcpp's. Without it, the backend derives them from what mcpp +2026.11.9.1 states (docs/08): + +- A program whose `main` file lies in a directory whose path below the test root is the program's name + is of layout `directory`. +- Its files are the C++ sources directly in that directory. +- Its parts are `/`, one per file. +- `main` is the file whose source defines `main`, found by the scanner; otherwise it is absent. + +A **case** belongs to a program and a file. Its fields are `id`, `file`, `line`, `kind`, `tags`, +`expect` and, with `test-metrics`, `bounds`. Static cases are read from each file with mtest's manifest. +A listing (`--cases`) replaces them case by case. + +A result names the program it belongs to: + +- `part_of` when mcpp states it; +- else the longest listed program whose name is the record's name or a prefix of it followed by `/`; +- else the record's name. + +## The backend's protocol (protocol 2, additions only) + +- `test/list`: each test gains `files`, `layout`, `parts`, and `main` as described above. +- `source/entries`: a file of a directory program answers `test` (the program) and `part` (its own + part). +- `test/cases`: reads every file of the program. Each case gains `file`, `kind`, `tags`, `expect` and + `bounds`. +- `test/result` gains: + - `program` and `part`; + - `cases`, each with `id`, `status`, `durationMs`, `reason`, `failures` (`path`, `line`, `message`, + `expected`, `actual`), `metrics`, `bounds`, `kind` and `tags`. + + A case's status is mapped as follows: + + | mcpp | protocol | + |---|---| + | `pass` | `passed` | + | `fail` | `failed` | + | `skip` | `skipped` | + | `xfail` | `passed` with `expectedFailure` | + | `xpass` | `failed` with `unexpectedPass` | + | `invalid` | `errored` | + + A whole-program run therefore marks every case. +- `test/caseResult`: gains the fields of a case above. It is sent for every record of the job, not + only the last one; a run of a directory program writes one record per part. +- `test/caseEvent` (new): `{jobId, root, program, part, case, event: "begin" | "end", tMs, ...}` with + the fields of a case on `end`. It is sent as mcpp writes a `case` record (`test-case-stream`). +- `test/history {root, program?, case?, runs?}` (new): the store's last runs (20 by default) and each + selected case's results across them. +- `test/insights {root, runs?}` (new): + - the last run: its counts and its counts by kind; + - its failures, each with the run since which it has failed; + - the slowest cases, with their median over the window; + - the cases whose status changed within the window (flaky); + - each metric's series, its bounds, its last value and its margin; + - the schedule: one lane per worker, each process with its start, duration and build, and the + critical path (the chain of processes on one worker that ends last, with the build before it). +- `capabilities/explain` gains `tests.parts`, `tests.live` and `tests.metrics`. + +The store is read from `[test] store` when the manifest sets it, else from `target/mtest`. The backend +reads it as P12 specifies it. It reads `index.ndjson`, then the last N runs' files, and keeps them by +run id. A run already read is not read again. A store that is absent is not an error: the history and +insights are empty and say so. + +## Each editor + +**VS Code.** + +- The Testing view shows each project, then its programs, then the files of a directory program, then + the cases. +- Each case carries the test tags `kind:` and its own tags, so `@kind:bench` filters the view. +- Results arrive per case as they happen (`test/caseEvent`), else per part. +- A failure with `expected` and `actual` is a `TestMessage.diff`, and VS Code's diff view compares them. +- After a run, a case's line ends with a dim decoration: its duration, and its metrics against their + bounds (`ready 4.2 s ≤ 15 s`). +- "mcpp: Test Insights" is a webview with these sections: + - the run's summary; + - its failures, each opening its place; + - the slowest cases; + - the flaky cases; + - each metric's trend against its bounds; + - the schedule, with its critical path. + + It refreshes when a run ends. + +**CLion.** + +- The SMRunner tree shows a program as a suite, the files of a directory program as nested suites, and + the cases as tests, each located at its line. +- Live case events start and finish tests as they come. +- A failure with `expected` and `actual` is a comparison failure, whose "Click to see difference" opens + the IDE's diff. +- Metrics appear in a case's output beside their bounds. +- The mcpp tool window gains a Tests tab: the same insights in JCEF (`webui/tests.html`), as the graph + page is drawn. + +**Neovim.** + +- `mcpp://tests` gains sections per program, part and case as results arrive, live with + `test-case-stream`. +- A failure's `expected` and `actual` appear as two lines under it, and in its quickfix text. +- A case's line carries its sign (✓, ✗) and, as virtual text, its duration and its metrics against + their bounds. +- `:Mcpp testInsights` opens the insights as text in a floating window. A metric's trend is drawn with + block characters (▁▂▃▅▇), and the schedule as bars. + +## Performance + +- The suite types of a project's tests are read once and kept with a signature of the files: their + count, their largest modification time, and their total size, from a walk that only stats. + `source/entries` and `test/cases` no longer read every file of `tests/` on each call. +- Live results are sent as they come. VS Code and CLion apply them as their test APIs do. +- Neovim redraws the results buffer at most every 100 ms. +- The store is read incrementally: only runs not read before. +- The insights window is bounded: 20 runs by default, 50 at most. + +## Tests + +- Backend: + - the derivation of parts from a 2026.11.9.1 listing; + - the attribution of part records to their program; + - the mapping of the protocol's statuses; + - `expected`/`actual`; + - `case` records into `test/caseEvent`; + - the store reader, and the insights (the window, flaky, the critical path, the margins); + - the schema of the new methods and notifications. +- The mock mcpp gains the fixtures of a directory program, with and without the #917 shapes. A + directory's `.mock-mcpp-tests` selects them, as `.mock-mcpp-features` selects the features. +- VS Code: the tree of a directory program; part results attributed to it; the diff message; the + decorations' text. +- Neovim: the smoke test covers live results, the virtual text and `:Mcpp testInsights`. +- CLion: the comparison failure and the nested suites, in the plugin's unit tests. +- The insights page is rendered by a Chromium test, as the graph page is. diff --git a/.github/actions/mtest-report/action.yml b/.github/actions/mtest-report/action.yml new file mode 100644 index 0000000..e77a1e9 --- /dev/null +++ b/.github/actions/mtest-report/action.yml @@ -0,0 +1,19 @@ +name: The tests' report +description: mtest's Markdown report of the last run `mcpp test` kept (target/mtest, mtest spec P12) -- its failures, its slowest cases, its metrics and its critical path -- in the job's summary. +runs: + using: composite + steps: + - name: The run, as mtest reports it + shell: bash + run: | + set -uo pipefail + [ -f target/mtest/index.ndjson ] || { echo "mcpp test kept no run"; exit 0; } + xlings install "mtest@$MTEST_VERSION" -y -g > /dev/null 2>&1 || { echo "::warning::mtest@$MTEST_VERSION could not be installed: no report"; exit 0; } + exe=""; [ "$RUNNER_OS" = Windows ] && exe=.exe + # The package's launcher, not the shim: on Windows the shim runs its alias through cmd. + mtest="$HOME/.xlings/data/xpkgs/xim-x-mtest/$MTEST_VERSION/bin/mtest$exe" + [ -x "$mtest" ] || { echo "::warning::no mtest at $mtest: no report"; exit 0; } + store="$RUNNER_TEMP/mtest-report" + mkdir -p "$store" && cd "$store" + "$mtest" store import --from "$GITHUB_WORKSPACE/target/mtest" > /dev/null + "$mtest" report --format md --run last --title "$GITHUB_JOB" >> "$GITHUB_STEP_SUMMARY" || true diff --git a/.github/scripts/release-plan.sh b/.github/scripts/release-plan.sh index 83318ce..acbdfba 100755 --- a/.github/scripts/release-plan.sh +++ b/.github/scripts/release-plan.sh @@ -1,14 +1,16 @@ #!/usr/bin/env bash # release-plan.sh [TAG] -- the components a release tag publishes, and its version (design 2026-11-07, D15). # -# A version is X.Y.Z: X.Y, the series, is shared by the backend and every editor plugin; Z is each -# component's own patch. Two kinds of tag: +# A version is X.Y.Z: X.Y, the series, is shared by the backend and every editor plugin; Z is an editor +# plugin's own patch. The backend is always X.Y.0: the VS Code and CLion packages carry it, so a change +# of the backend changes every package, and it begins a series. Two kinds of tag: # -# vX.Y.0 a series release: every component, each at X.Y.0 -# -vX.Y.Z one component's patch: backend, vscode, clion or nvim, at X.Y.Z +# vX.Y.0 a series release: every component, each at X.Y.0 (a change of the backend) +# -vX.Y.Z one editor plugin's patch: vscode, clion or nvim, at X.Y.Z # # Prints `components=...`, `version=...` and `title=...` for $GITHUB_OUTPUT. Without a tag it only -# checks that every component states one series and the backend's two statements agree (CI). +# checks that every component states one series, the backend's two statements agree and the backend +# is its series' X.Y.0 (CI). set -euo pipefail root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" declare -A version @@ -26,6 +28,8 @@ for c in backend vscode clion nvim; do done [ "$backend_source" = "${version[backend]}" ] || fail "src/server.cppm says $backend_source and mcpp.toml ${version[backend]}" the_series="$(series "${version[backend]}")" +[ "${version[backend]}" = "$the_series.0" ] \ + || fail "the backend is ${version[backend]}; it is always its series' X.Y.0 (the editor packages carry it, so a change of it begins a series)" [ "$series_source" = "$the_series" ] || fail "src/server.cppm's kSeries is $series_source, the backend's series $the_series" for c in vscode clion nvim; do [ "$(series "${version[$c]}")" = "$the_series" ] \ @@ -45,13 +49,31 @@ if [[ "$tag" =~ ^v([0-9]+\.[0-9]+\.0)$ ]]; then echo "components=backend vscode clion nvim" echo "version=$v" echo "title=mcpp-ide $v" -elif [[ "$tag" =~ ^(backend|vscode|clion|nvim)-v([0-9]+\.[0-9]+\.[0-9]+)$ ]]; then +elif [[ "$tag" =~ ^backend-v ]]; then + fail "$tag: the backend has no patch of its own; a change of it is a series release vX.Y.0" +elif [[ "$tag" =~ ^(vscode|clion|nvim)-v([0-9]+\.[0-9]+\.[0-9]+)$ ]]; then c="${BASH_REMATCH[1]}"; v="${BASH_REMATCH[2]}" [ "${version[$c]}" = "$v" ] || fail "the tag $tag names $c $v, and $c is ${version[$c]}" - declare -A names=([backend]=mcpp-ide [vscode]=mcpp-vscode [clion]=mcpp-clion [nvim]=mcpp-nvim) + # A plugin's patch carries its series' backend: no path of the backend (.github/components.tsv) + # has changed since the series' tag. + if git -C "$root" rev-parse -q --verify "refs/tags/v$the_series.0" > /dev/null; then + mapfile -t backend_rows < <(awk -F'\t' '$1 !~ /^(#|@)/ && NF == 2 {print $1 "\t" $2}' "$root/.github/components.tsv") + while IFS= read -r path; do + for row in "${backend_rows[@]}"; do + pattern="${row#*$'\t'}" + # shellcheck disable=SC2254 + case "$path" in $pattern) + [ "${row%%$'\t'*}" != backend ] \ + || fail "$tag: the backend changed since v$the_series.0 ($path); a change of the backend is a series release" + break ;; + esac + done + done < <(git -C "$root" diff --name-only "v$the_series.0" HEAD) + fi + declare -A names=([vscode]=mcpp-vscode [clion]=mcpp-clion [nvim]=mcpp-nvim) echo "components=$c" echo "version=$v" echo "title=${names[$c]} $v" else - fail "$tag is neither vX.Y.0 nor -vX.Y.Z" + fail "$tag is neither vX.Y.0 nor -vX.Y.Z" fi diff --git a/.github/versions.env b/.github/versions.env index d956dab..2ab092b 100644 --- a/.github/versions.env +++ b/.github/versions.env @@ -1,12 +1,14 @@ # The versions CI builds and tests with, in one file; .github/actions/setup-mcpp exports them. # The published bootstrap; the project explicitly selects platform 0.24. -MCPP_VERSION=2026.11.7.2 +MCPP_VERSION=2026.11.9.1 # A ref of mcpp-community/mcpp CI builds mcpp from and uses instead, for a contract not yet released. # Empty: the release above is used. MCPP_SOURCE_REF= # The LLVM the backend builds with on macOS and Windows (the targets mcpp reaches there with it). LLVM_VERSION=23.1.3 -# The xlings mcpp 2026.11.7.2 bundles, which reads the current index. +# The xlings mcpp 2026.11.9.1 bundles, which reads the current index. XLINGS_VERSION=v2026.10.26.1 # The oldest VS Code the extension supports (engines.vscode), which its tests run in. VSCODE_VERSION=1.91.0 +# The mtest whose report of each `mcpp test` run is the job's summary (.github/actions/mtest-report). +MTEST_VERSION=0.5.2 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4fb7b47..01b7fa6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -155,6 +155,9 @@ jobs: run: mcpp build - name: Test run: mcpp test + - name: The tests' report + if: always() + uses: ./.github/actions/mtest-report - name: Every string has its translation run: mcpp run mcpp-ide-dev -- check-i18n - name: No header, no preprocessor branch, no compile-time platform question @@ -218,6 +221,9 @@ jobs: run: mcpp build - name: The tests, as ok images, through the runner run: mcpp test --profile release --target ${{ matrix.arch }}-openkal + - name: The tests' report + if: always() + uses: ./.github/actions/mtest-report # The VS Code extension (mide-plugins/vscode), mcpp-vscode 0.9.0 over the backend. Its gates # (the registry, the translations, the unit suite) on the three desktop systems; the manifest diff --git a/.github/workflows/clion-stable.yml b/.github/workflows/clion-stable.yml index 8158610..44afa71 100644 --- a/.github/workflows/clion-stable.yml +++ b/.github/workflows/clion-stable.yml @@ -71,7 +71,7 @@ jobs: set -euo pipefail version="$(sed -n 's/^mcpplsPluginVersion *= *//p' mide-plugins/clion/gradle.properties)" gh release download "v$version" --repo Sunrisepeak/mcpp-language-server \ - --pattern "mcppls-clion-$version-linux-x64.zip" --dir stable-plugin + --pattern "mcppls-clion-$version.zip" --dir stable-plugin - name: Start the real IDE and complete the plugin startup chain env: IDE_VERSION: ${{ matrix.version }} diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index df2261e..9a9cd9a 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -6,11 +6,12 @@ name: Release # independently after that release exists. # # WHAT A TAG PUBLISHES (design 2026-11-07, D15; .github/scripts/release-plan.sh): a version is X.Y.Z, -# X.Y the series every component shares and Z each component's own patch. +# X.Y the series every component shares and Z an editor plugin's own patch. The backend is always +# X.Y.0: the VS Code and CLion packages carry it, so a change of it is a series release. # -# vX.Y.0 a series release: the backend, the VS Code extension, the CLion plugin and -# the Neovim plugin (a source tree, which xim takes from the tag's archive) -# backend-vX.Y.Z the backend's ok-image bundle and package (xim's mcpp-ide) +# vX.Y.0 a series release: the backend's ok-image bundle and package (xim's mcpp-ide), +# the VS Code extension, the CLion plugin and the Neovim plugin (a source tree, +# which xim takes from the tag's archive) # vscode-vX.Y.Z the VSIX (Open VSX) # clion-vX.Y.Z the CLion ZIP (JetBrains Marketplace) # nvim-vX.Y.Z the Neovim plugin: the tag's source archive (xim's mcpp-ide-nvim) @@ -20,7 +21,7 @@ name: Release on: workflow_dispatch: push: - tags: ['v*', 'backend-v*', 'vscode-v*', 'clion-v*', 'nvim-v*'] + tags: ['v*', 'vscode-v*', 'clion-v*', 'nvim-v*'] permissions: contents: read @@ -34,6 +35,8 @@ jobs: title: ${{ steps.plan.outputs.title }} steps: - uses: actions/checkout@v5 + with: + fetch-depth: 0 # a plugin's patch is checked against its series' tag - id: plan shell: bash # -eo pipefail: a refused tag fails the step through tee run: bash .github/scripts/release-plan.sh "$GITHUB_REF_NAME" | tee -a "$GITHUB_OUTPUT" diff --git a/.xlings.json b/.xlings.json index 4c1f3ff..c0a15fe 100644 --- a/.xlings.json +++ b/.xlings.json @@ -1,5 +1,5 @@ { "workspace": { - "mcpp": "2026.11.7.2" + "mcpp": "2026.11.9.1" } } diff --git a/CHANGELOG.md b/CHANGELOG.md index 7df5826..8761bba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,60 @@ The changelog is kept by series (`X.Y`); each section lists its components' releases. +## 1.3 + +### 1.3.0 + +Every component is 1.3.0: the backend, mcpp-vscode, mcpp-clion and mcpp-nvim (from 1.2.1, 1.2.0, 1.2.0 +and 1.2.0). + +- **A change of the backend begins a series.** The VS Code and CLion packages carry the backend, so a + change of it is a change of every package: it is released as `vX.Y.0`, every component at `X.Y.0`, + and the backend is always its series' `X.Y.0`. `Z` is an editor plugin's own patch (`vscode-`, + `clion-`, `nvim-vX.Y.Z`), which carries its series' backend. The release refuses a `backend-v` tag, + a backend that is not `X.Y.0`, and a plugin's patch when a path of the backend has changed since its + series' tag. Backend 1.2.1 reached `xlings install mcpp-ide` alone; its change, mtest's cases read + with mtest's own static manifest, is in every package of 1.3.0. +- **Tests as mcpp states them** (design 2026-11-10). A program of several files (`[test] programs = + "directory"`, mcpp 2026.11.9.1) is shown with its files as children, each the part of the program a + run gives a process (`unit/a` of `unit`), and a part's record is attributed to its program; 1.2 listed + `unit` and lost the results of `unit/a` and `unit/b`. A case carries its file, kind, tags, expectation + and the bounds on its metrics, from mtest's manifest and the program's listing. A whole run marks every + case of a protocol program, and a run of cases reports every part's record, not only the last. + `xfail` passes, `xpass` fails as an unexpected pass, and `invalid` is errored. A failure that states + both sides is a diff in VS Code, a comparison failure in CLion, and two lines in Neovim's buffer and + quickfix item. + - **Live results.** With mcpp's `test-case-stream` (mcpp#917), each case is shown as it begins and + ends (`test/caseEvent`). + - **Metrics.** With `test-metrics`, a case's metrics are shown against their bounds: at the end of its + line in VS Code and Neovim, and in its output in CLion. + - **Test Insights** (`mcpp.test.insights`; `test/insights`, `test/history`). It reads the runs mcpp + keeps in `target/mtest`: the last run and its counts by kind; its failures and since when; each + metric's trend against its bound; the slowest and the flaky cases; the schedule with its critical + path. VS Code shows it in a webview, CLion in the mcpp tool window (both draw `webui/tests.html`), + and Neovim as text. + - **Older mcpp.** A view that needs a feature the project's mcpp does not state is left out, and + `capabilities/explain` names the feature (`tests.cases`, `tests.parts`, `tests.live`, + `tests.metrics`). + - **Speed.** The suite types of a project's tests are read again only when its test files change. + - **CLion.** The test tree is ID-based, so parts that run at once report into their own suites. +- **The backend's tests are mtest's** (mtest 0.5.2). `tests/unit`, `tests/e2e`, `tests/cts` and + `tests/bench` are each one program whose files declare their cases, with no `main`; the rules + goldens and the published payloads sit beside the cases that read them. `tests/bench` bounds the + latency of a keystroke in a manifest of 400 dependencies, of reading a test file of 2000 cases, and + of the insights over 50 runs of 400 cases. testsupport's `check` is mtest's fatal expectation, and + CI reports each run with `mtest report` in the job's summary. +- **A deprecated command is VS Code's alone.** `commands/list` offered CLion the deprecated ids VS Code keeps + for earlier releases (`mcpp.checkModuleSupport`, `mcpp.configureClangd`, ...) once mcppls 0.21 named a + CLion action for what they forward; CLion declares none of them, so they were offered and not registered. +- **Dependencies.** The backend builds with mcpp 2026.11.9.1, mcpplibs.os 0.4.2 and + mcpplibs.mtest.manifest 0.5.2. +- **mcppls 0.21.** mcpp-clion is built and tested against the mcppls CLion plugin 0.21.0, whose one + package serves every system (`mcppls-clion-.zip`, since mcppls 0.0.20); the build fetched + 0.0.19's package of this machine. mcppls 0.21 numbers its versions as a series and a patch (0.0.21 + is released as 0.21.0); the backend reads that version, its payloads (version 5) and `mcppls status` + as before. + ## 1.2 ### backend 1.2.1 diff --git a/README.md b/README.md index 1c759cc..1fe0ef8 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,9 @@ those of mcpp.toml and build.mcpp come from the backend ([docs/rules.md](docs/ru | | VS Code (mcpp-vscode) | CLion (mcpp-clion) | Neovim (mcpp-nvim) | |---|---|---|---| | Build, run, test | Tasks in a pseudoterminal showing mcpp's output | "mcpp" run configurations, PTY console, ▶ beside `main` | `:Mcpp build`, `test`: jobs streamed into a buffer; `:Mcpp run` in a terminal buffer | -| Test tree, re-run one test | Testing view | IDE test runner | `:Mcpp test ` | +| Test tree, re-run one test | Testing view: program, file, case; `@kind:bench` and a case's tags filter it | IDE test runner: a suite per program and per file | `:Mcpp test `: `mcpp://tests`, signs, the quickfix list | +| Test results | Live per case; a failure with both sides opens VS Code's diff; a case's line ends with its duration and metrics | Live per case; a comparison failure opens the IDE's diff; metrics in the case's output | Live per case; both sides in the buffer and the quickfix list; duration and metrics as virtual text | +| Test insights (the runs mcpp keeps) | "mcpp: Show Test Insights", a webview | The mcpp tool window's Test Insights tab (JCEF) | `:Mcpp test.insights`, a floating window | | Debug | CodeLLDB, C/C++ (gdb, lldb), lldb-dap | CLion's GDB or LLDB | nvim-dap (codelldb, lldb-dap, cppdbg, gdb) | | Diagnostics | Problems, from mcpp's located diagnostics | Console links to the reported location | `vim.diagnostic`, `:Mcpp problems` in the quickfix list | | mcpp.toml, build.mcpp | Completion, hover, jumps and the rules' diagnostics from the backend (language client); Problems from mcpp | The same, through CLion's completion, documentation, annotations and go-to | The same, through `vim.lsp`; mcpp's problems in `vim.diagnostic` | @@ -42,6 +44,23 @@ Entering a setting's default removes its stored override. Graph queries report build-database errors instead of showing an empty graph. After correcting the reported problem, reopen the graph to retry. +**Tests.** The editors show tests as `mcpp test` states them ([docs/architecture.md](docs/architecture.md), +"A test run, end to end"). A program of several files (`[test] programs = "directory"`) has its files as +children, each the part of the program a run gives a process. A case of the mtest test protocol carries its +kind, tags and the bounds on its metrics. Test Insights reads the runs mcpp keeps in `target/mtest`: the last +run, its failures and since when, each metric against its bound, the slowest and the flaky cases, and the +schedule with its critical path. What an editor shows follows the project's mcpp: + +| The project's mcpp | Adds | +|---|---| +| any | programs and their results | +| 2026.11.8.1 (`test-protocol`) | cases, their places and results, both sides of a failure | +| 2026.11.9.1 | the files of a program of several files; Test Insights over the kept runs | +| `test-parts`, `test-case-stream`, `test-metrics` ([mcpp#917](https://github.com/mcpp-community/mcpp/issues/917)) | the parts as mcpp states them, results as each case ends, metrics and their trends | + +A view that needs a feature the project's mcpp does not state is left out. The self-check +(`capabilities/explain`) says which mcpp states it. + - **One answer everywhere** -- what a build runs, how a test is reported, what a cleanup costs: the backend's - **Only mcpp's contract** -- no manifest parsing, no dependency resolution of its own - **Jobs in order** -- one at a time per project, a newer build replaces a waiting one, cancelling ends cleanly @@ -76,9 +95,12 @@ VS Code and CLion check for the local build tool only when a user starts an acti **Versions.** A version is `X.Y.Z`. `X.Y`, the series, is shared by the backend and every plugin: a plugin works with any backend of its series and refuses another, naming both versions and the -install that matches. `Z` is each component's own patch, so VS Code 1.2.3 and CLion 1.2.1 both work -with backend 1.2.0. A series release is tagged `vX.Y.0`; a component's patch is tagged -`backend-`, `vscode-`, `clion-` or `nvim-vX.Y.Z` and publishes that component alone. A pull request runs the jobs of +install that matches. The VS Code and CLion packages carry the backend, so a change of the backend is a +change of every package: it begins a series, every component at `X.Y.0`, tagged `vX.Y.0`, and the backend +is always its series' `X.Y.0`. `Z` is an editor plugin's own patch, tagged `vscode-`, `clion-` or +`nvim-vX.Y.Z`, which publishes that plugin alone over its series' backend: VS Code 1.3.2 and CLion 1.3.1 +both carry backend 1.3.0. A plugin's patch is refused when a path of the backend has changed since its +series' tag. A pull request runs the jobs of the components it changes ([`.github/components.tsv`](.github/components.tsv)): a change of the backend runs everything, a change of one plugin runs that plugin's jobs over the backend main built from the same inputs, a change of the documentation its checks; `.github/` and any path the table does not name run everything. @@ -90,9 +112,9 @@ If `mcpp` is absent, they show one notification linking to the ```bash mcpp build # the backend, the mock mcpp, mcpp-ide-dev -mcpp test # the backend's tests +mcpp test # the backend's tests (mtest: tests/unit, e2e, cts, bench) cd mide-plugins/vscode && npm ci && npm test # the extension's tests, the parity gates against 0.9.0 among them -npm run test:webui # CLion's graph page in Chromium, over the backend +npm run test:webui # the graph and tests pages in Chromium, over the backend npm run package # dist/mcpp-vscode-.vsix cd ../clion && gradle buildPlugin verifyPlugin # build/distributions/mcpp-clion-.zip cd ../.. && nvim --headless --clean -l mide-plugins/nvim/tests/smoke.lua # the Neovim plugin's tests diff --git a/README.zh-CN.md b/README.zh-CN.md index 23e4919..cfbd485 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -23,7 +23,9 @@ quickfix 列表。各编辑器插件位于本仓库的 `mide-plugins/`。C++ 文 | | VS Code（mcpp-vscode） | CLion（mcpp-clion） | Neovim（mcpp-nvim） | |---|---|---|---| | 构建、运行、测试 | 伪终端中的任务，显示 mcpp 自身的输出 | "mcpp" 运行配置、PTY 控制台、`main` 旁的 ▶ | `:Mcpp build`、`test`：任务输出流入缓冲区；`:Mcpp run` 在终端缓冲区中运行 | -| 测试树、单个测试重跑 | 测试视图 | IDE 测试运行器 | `:Mcpp test ` | +| 测试树、单个测试重跑 | 测试视图：程序、文件、用例；`@kind:bench` 与用例的标签可筛选 | IDE 测试运行器：每个程序、每个文件一个套件 | `:Mcpp test `：`mcpp://tests`、标记、quickfix 列表 | +| 测试结果 | 逐用例实时显示；陈述两侧的失败可打开 VS Code 的差异视图；用例所在行末显示耗时与指标 | 逐用例实时显示；比较型失败可打开 IDE 的差异视图；指标在用例输出中 | 逐用例实时显示；两侧写入缓冲区与 quickfix 列表；耗时与指标以虚拟文本显示 | +| 测试洞察（mcpp 保留的运行） | "mcpp: Show Test Insights"，一个 webview | mcpp 工具窗口的 Test Insights 页（JCEF） | `:Mcpp test.insights`，浮动窗口 | | 调试 | CodeLLDB、C/C++（gdb、lldb）、lldb-dap | CLion 的 GDB 或 LLDB | nvim-dap（codelldb、lldb-dap、cppdbg、gdb） | | 诊断 | 问题面板，来自 mcpp 带位置的诊断 | 控制台链接到报告的位置 | `vim.diagnostic`，`:Mcpp problems` 填入 quickfix 列表 | | mcpp.toml、build.mcpp | 补全、悬停、跳转与规则诊断来自后端（语言客户端）；问题来自 mcpp | 同上，经 CLion 的补全、文档、标注与跳转 | 同上，经 `vim.lsp`；mcpp 的问题在 `vim.diagnostic` | @@ -36,6 +38,21 @@ CLion 的 **Settings | Tools | mcpp** 在保存表单前检查数值是否有限 图查询失败时显示构建数据库错误。修正所报告的问题后，重新打开图即可重试。 +**测试。** 编辑器按 `mcpp test` 的陈述显示测试（[docs/architecture.md](docs/architecture.md)， +"A test run, end to end"）。多文件程序（`[test] programs = "directory"`）以其文件为子项，每个文件是一次运行中 +单独成进程的那一部分。mtest 测试协议的用例带有其类别、标签与指标上下限。测试洞察读取 mcpp 保存在 +`target/mtest` 的运行：最近一次运行、其失败及起始时间、各指标与上下限、最慢与不稳定的用例、调度与关键路径。 +编辑器所显示的内容取决于工程的 mcpp： + +| 工程的 mcpp | 增加 | +|---|---| +| 任意版本 | 程序及其结果 | +| 2026.11.8.1（`test-protocol`） | 用例、其位置与结果、失败的两侧 | +| 2026.11.9.1 | 多文件程序的文件；基于已保留运行的测试洞察 | +| `test-parts`、`test-case-stream`、`test-metrics`（[mcpp#917](https://github.com/mcpp-community/mcpp/issues/917)） | 由 mcpp 陈述的部分、逐用例结束时的结果、指标及其趋势 | + +工程的 mcpp 未声明某视图所需的能力时，该视图不显示；自检（`capabilities/explain`）说明哪个版本的 mcpp 声明了它。 + - **处处同一答案**——构建运行什么、测试如何报告、清理的代价：都由后端决定 - **只读 mcpp 的契约**——不自行解析清单，不自行解析依赖 - **任务有序**——每个工程同一时间一个任务，新构建替换等待中的构建，取消会正常结束 @@ -68,8 +85,10 @@ VS Code 和 CLion 仅在用户主动执行依赖本地构建工具的操作时 [mcpp 开源仓库](https://github.com/mcpp-community/mcpp)链接。 **版本。** 版本为 `X.Y.Z`。`X.Y` 是系列，由后端与全部插件共享：插件与同系列的任何后端配合，遇到其他系列时拒绝， -并写出双方的版本与对应的安装方法。`Z` 是各组件自己的补丁，所以 VS Code 1.2.3 与 CLion 1.2.1 都与后端 1.2.0 -兼容。系列发布的标签为 `vX.Y.0`；组件补丁的标签为 `backend-`、`vscode-`、`clion-` 或 `nvim-vX.Y.Z`，只发布该组件。 +并写出双方的版本与对应的安装方法。VS Code 与 CLion 的包内含后端，因此后端的改动即是全部包的改动：它开启一个新系列， +全部组件为 `X.Y.0`，标签为 `vX.Y.0`；后端始终是所属系列的 `X.Y.0`。`Z` 是编辑器插件自己的补丁，标签为 `vscode-`、 +`clion-` 或 `nvim-vX.Y.Z`，只发布该插件，所带的后端仍是本系列的那一份：VS Code 1.3.2 与 CLion 1.3.1 都带后端 1.3.0。 +自本系列的标签以来后端的任何路径有改动时，插件补丁被拒绝。 拉取请求只运行它所改动的组件的作业（[`.github/components.tsv`](.github/components.tsv)）：后端的改动运行全部作业； 一个插件的改动只运行该插件的作业，所用的后端是 main 以相同输入构建的那一份；文档的改动只运行文档的检查；`.github/` 与表中没有列出的路径运行全部作业。唯一必需的检查是 `ci`。 @@ -78,9 +97,9 @@ VS Code 和 CLion 仅在用户主动执行依赖本地构建工具的操作时 ```bash mcpp build # 后端、mock mcpp、mcpp-ide-dev -mcpp test # 后端测试 +mcpp test # 后端测试（mtest：tests/unit、e2e、cts、bench） cd mide-plugins/vscode && npm ci && npm test # 扩展的测试，含对 0.9.0 的一致性门 -npm run test:webui # CLion 的图页面在 Chromium 中、经后端运行 +npm run test:webui # 图页面与测试页面在 Chromium 中、经后端运行 npm run package # dist/mcpp-vscode-.vsix cd ../clion && gradle buildPlugin verifyPlugin # build/distributions/mcpp-clion-.zip cd ../.. && nvim --headless --clean -l mide-plugins/nvim/tests/smoke.lua # Neovim 插件的测试 diff --git a/agents/skills/mcpp-ide/SKILL.md b/agents/skills/mcpp-ide/SKILL.md index ab8900b..b2ef8bb 100644 --- a/agents/skills/mcpp-ide/SKILL.md +++ b/agents/skills/mcpp-ide/SKILL.md @@ -19,8 +19,10 @@ dependency. Its answers are the editors' answers. | Method | Answers | |---|---| -| `test/list {root}` | every test, its workspace member and the file holding its `main`; no build | -| `job/start {kind: build\|test, root, target?, profile?, filter?}` | a job id; then `job/output`, `job/plan`, `job/diagnostic`, `test/result`, `test/summary`, `job/finished` | +| `test/list {root}` | every test program, its workspace member, its files, the file holding its `main`, and the parts a run divides a program of several files into; no build | +| `test/cases {root, test, list?}` | a program's cases, each with its file, line, kind, tags and the bounds on its metrics | +| `job/start {kind: build\|test, root, target?, profile?, filter?}` | a job id; then `job/output`, `job/plan`, `job/diagnostic`, `test/result` (with the program, the part and its cases), `test/caseEvent` (as each case begins and ends, when mcpp states it), `test/summary`, `job/finished` | +| `test/insights {root, runs?}`, `test/history {root, program?, case?}` | the runs mcpp keeps in `target/mtest`: failures and since when, metrics against their bounds, the slowest and the flaky cases, the schedule; a case across the runs | | `job/cancel {jobId}` | mcpp ends the steps it started; the result record says `cancelled` | | `targets/list {root}` | each target, its toolchains, `installed`, and `platform` (name, version, source, fields) | | `project/summary {root}` | executables, tests, libraries, last build, the platforms the last plan named | @@ -34,7 +36,8 @@ dependency. Its answers are the editors' answers. - `job/finished.status`: `ok`, `failed`, `cancelled`, `denied` (an effect was denied: exit code 5), `superseded` (a newer job of the same kind replaced it). - `test/result.status`: `passed`, `failed`, `timedOut`, `skipped`, `errored`; `output` is the - test's compile and run output. + test's compile and run output. A case's status is `passed` (also `expectedFailure`), `failed` (also + `unexpectedPass`), `skipped` or `errored`; a failure's `expected` and `actual` are both sides. - Diagnostics are 1-based `{line, column}`, with `path` absolute. - A cleanup at level > 0 is applied only with `cache/apply {action, confirmation: {confirmed, previewShown, acknowledged}}` -- ask the person before you set those. diff --git a/docs/architecture.md b/docs/architecture.md index ecbc3f2..b02eac0 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -73,6 +73,41 @@ mcppls what a compiler answers better. 5. mcpp writes the result record; the backend's watcher sends `result/changed`; mcppls reads the same record and refreshes. +## A test run, end to end + +The editors show tests as `mcpp test` states them (design 2026-11-10). mcpp builds and runs the +programs; mtest's protocol is how a program's cases report to mcpp; mtest's store (spec P12) is the form +in which mcpp keeps a run (`target/mtest`, or `[test] store`). The backend reads those three; it runs no +other test runner. + +1. `test/list` runs `mcpp test --list` and answers each program with its files, its `main` and its parts. + A protocol program of several files (`[test] programs = "directory"`) runs as one process per file, + `/`. mcpp states the parts with `test-parts` (mcpp#917); before it, the backend derives + them as mcpp 2026.11.9.1 forms them. The programs are kept per project. +2. `test/cases` reads the cases of every file of a program with mtest's static manifest (their kinds, + tags and the bounds on their metrics), and with `list` the program's own listing (`--cases`). +3. A test job runs `mcpp test --message-format json`. Each record becomes `test/result`, with the + program it belongs to, the part it is and its cases, each status mapped (`xfail` passes, `xpass` + fails, `invalid` is errored). With `test-case-stream` (mcpp#917) each case is sent as it begins and + ends (`test/caseEvent`). +4. VS Code marks the Testing view's items, a failure with both sides as a diff, and ends a case's line + with its duration and metrics. CLion fills an ID-based test tree: a suite per program and per file, a + comparison failure when both sides are stated. Neovim writes `mcpp://tests`, the quickfix list and + the signs. +5. `test/insights` reads the store's last runs (20 by default) and answers the last run, its failures + and since when, the slowest and the flaky cases, each metric against its bound, and the schedule + with its critical path. VS Code and CLion draw it with one page (`webui/tests.html`); Neovim prints + it. + +Every step reads what the project's mcpp states and nothing more: an older mcpp gives programs and +their results, and a view that needs a later feature is left out (`capabilities/explain`). + +The backend's own tests are mtest's (mtest docs/layout.md): `tests/unit`, `tests/e2e` (the wire and a +session over the mock mcpp), `tests/cts` (the contracts against their schemas, the rules layer against +its goldens) and `tests/bench` (the latency of a keystroke in a large manifest, of reading a large test +file, of the insights over a full window), each directory one program. CI reports each run with +`mtest report` in the job's summary. + ## Distribution The backend is one program for every editor, and one file for every system: an ok image per diff --git a/docs/rules.md b/docs/rules.md index 8620059..c4d6fed 100644 --- a/docs/rules.md +++ b/docs/rules.md @@ -90,8 +90,8 @@ The rules read two snapshots of mcpp: `spec/toml-schema.json` (every table and k release that added it) and `spec/buildscript-api.json` (the directives, roles and protocol), each naming the mcpp release and commit it was taken from. -The layer is a port of mcpp-vscode 0.9's TypeScript implementation. `tests/rules/golden` holds that -implementation's answers over `tests/rules/corpus` -- the scanner's structure and the cursor's context +The layer is a port of mcpp-vscode 0.9's TypeScript implementation. `tests/cts/rules/golden` holds that +implementation's answers over `tests/cts/rules/corpus` -- the scanner's structure and the cursor's context at every position of every line, completion, hover, the jumps, the version contexts and the diagnostics under two sets of severities -- and over the documents of its parser tests -(`golden/snippets.json`); `test_rules` compares the backend's answers with them. +(`golden/snippets.json`); the conformance case `rules` (tests/cts/rules.cpp) compares the backend's answers with them. diff --git a/mcpp.lock b/mcpp.lock index 6b1ee90..477ed06 100644 --- a/mcpp.lock +++ b/mcpp.lock @@ -5,6 +5,14 @@ # file must not change depending on which command ran last. version = 2 +[target."aarch64-openkal"] +identity = "aarch64-openkal" +provider = "openkal" + +[target."x86_64-openkal"] +identity = "x86_64-openkal" +provider = "openkal" + [package."compat.mimalloc"] namespace = "compat" version = "3.6.1" @@ -76,16 +84,17 @@ targets = ["aarch64-openkal", "x86_64-openkal"] [package."mcpplibs.os"] namespace = "mcpplibs" -version = "0.4.0" -source = "index+mcpplibs@0.4.0" -hash = "fnv1a:c06e4aa4f73f1f27" +version = "0.4.2" +source = "index+mcpplibs@0.4.2" +hash = "fnv1a:c06e4ca4f73f228d" targets = ["aarch64-openkal", "x86_64-linux-gnu", "x86_64-openkal"] [package."mcpplibs.mtest.manifest"] namespace = "mcpplibs.mtest" -version = "0.4.1" -source = "index+mcpplibs.mtest@0.4.1" -hash = "fnv1a:0339d37dd3c40581" +version = "0.5.2" +source = "index+mcpplibs.mtest@0.5.2" +hash = "fnv1a:0caef37dd958cfef" +targets = ["aarch64-openkal", "x86_64-linux-gnu", "x86_64-openkal"] [package."nlohmann.json"] namespace = "nlohmann" diff --git a/mcpp.toml b/mcpp.toml index 8627ffd..5c31729 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -9,12 +9,12 @@ # M2/M3), where the system it runs upon is answered when the image is loaded. [package] name = "mcpp-ide" -version = "1.2.1" +version = "1.3.0" description = "The IDE backend for mcpp: sessions, jobs, records and view models over mcpp's control contract" license = "Apache-2.0" authors = ["mcpp-community"] repo = "https://github.com/mcpp-community/mcpp-ide" -mcpp = ">=2026.11.7.1" +mcpp = ">=2026.11.9.1" [language] standard = "c++23" @@ -78,11 +78,17 @@ main = "src/bin/dev.cpp" [dependencies] # 0.3.1 reads a name in Windows' long spelling as the name it spells. The image C library, C++ # runtime and loader remain the platform's. -mcpplibs.os = "0.4.0" +mcpplibs.os = "0.4.2" nlohmann.json = "3.12.0" # mtest's static case manifest: the cases a test file declares, read as mtest reads them. -mcpplibs.mtest.manifest = "0.4.1" +mcpplibs.mtest.manifest = "0.5.2" -# What every backend test shares (testsupport/): a module, as the backend's units are. +# The tests are mtest's (mtest docs/layout.md): tests/unit, tests/e2e, tests/cts and tests/bench, each +# directory one program whose files declare their cases; what every test shares is testsupport/, a +# module, as the backend's units are. [dev-dependencies] +mcpplibs.mtest = "0.5.2" mcpp-ide-testsupport = { path = "testsupport" } + +[test] +programs = "directory" diff --git a/mide-plugins/clion/README.md b/mide-plugins/clion/README.md index 854503e..79dfc74 100644 --- a/mide-plugins/clion/README.md +++ b/mide-plugins/clion/README.md @@ -99,8 +99,8 @@ An individual packaged plugin can be checked without rebuilding the backend: ```bash xvfb-run -a gradle --no-daemon -p integration-tests test \ - -PpluginZip=/absolute/path/mcpp-clion-0.2.1.zip \ - -PmcpplsZip=/absolute/path/mcppls-clion-0.0.19-linux-x64.zip \ + -PpluginZip=/absolute/path/mcpp-clion-1.3.0.zip \ + -PmcpplsZip=/absolute/path/mcppls-clion-0.21.0.zip \ -PideVersion=2026.1.5.2 ``` diff --git a/mide-plugins/clion/build.gradle.kts b/mide-plugins/clion/build.gradle.kts index f317d57..d8cc233 100644 --- a/mide-plugins/clion/build.gradle.kts +++ b/mide-plugins/clion/build.gradle.kts @@ -30,23 +30,13 @@ repositories { // The mcppls CLion plugin, from its release: the dependency the sandbox, the tests and the verifier // load beside this plugin. Fetched once, when the build is configured, because the platform's -// dependency resolution reads it then. mcppls ships one package per platform, each carrying that -// platform's server (mcppls 0.0.19); this machine's is the one loaded here. +// dependency resolution reads it then. Since mcppls 0.0.20 one package serves every system: its +// payload holds the server's ok images and each system's runner. val mcpplsVersion = providers.gradleProperty("mcpplsPluginVersion").get() -val hostPlatform: String = run { - val os = System.getProperty("os.name").lowercase() - val system = when { - os.startsWith("windows") -> "win32" - os.startsWith("mac") -> "darwin" - else -> "linux" - } - val arch = System.getProperty("os.arch").lowercase() - "$system-${if (arch == "aarch64" || arch == "arm64") "arm64" else "x64"}" -} -val mcpplsArchive: File = layout.buildDirectory.file("mcppls/mcppls-clion-$mcpplsVersion-$hostPlatform.zip").get().asFile +val mcpplsArchive: File = layout.buildDirectory.file("mcppls/mcppls-clion-$mcpplsVersion.zip").get().asFile if (!mcpplsArchive.exists()) { mcpplsArchive.parentFile.mkdirs() - val url = "https://github.com/Sunrisepeak/mcpp-language-server/releases/download/v$mcpplsVersion/mcppls-clion-$mcpplsVersion-$hostPlatform.zip" + val url = "https://github.com/Sunrisepeak/mcpp-language-server/releases/download/v$mcpplsVersion/mcppls-clion-$mcpplsVersion.zip" val partial = File(mcpplsArchive.path + ".part") uri(url).toURL().openStream().use { input -> partial.outputStream().use { input.copyTo(it) } } partial.renameTo(mcpplsArchive) @@ -89,7 +79,7 @@ val generateDeclarations = tasks.register("generateDeclarations") { if (dev != null) commandLine(dev, "gen", "jetbrains", "--out", out.absolutePath) else commandLine("mcpp", "run", "mcpp-ide-dev", "--", "gen", "jetbrains", "--out", out.absolutePath) } -// The graph page JCEF loads (../../webui, mcpp design 2026-10-30 part 2 §9): webui/src/graph.ts and +// The pages JCEF loads (../../webui, mcpp design 2026-10-30 part 2 §9): webui/src/graph.ts, tests.ts and // mermaid, bundled by the extension's tools/bundle-webui.mjs (Node; `npm ci` in ../vscode first), // with webui/files.txt naming its files. val webuiOut = layout.buildDirectory.dir("generated/webui/webui") @@ -97,6 +87,7 @@ val bundleWebui = tasks.register("bundleWebui") { val extension = repositoryRoot.resolve("mide-plugins/vscode") inputs.dir(repositoryRoot.resolve("webui/src")) inputs.file(repositoryRoot.resolve("webui/graph.html")) + inputs.file(repositoryRoot.resolve("webui/tests.html")) inputs.file(extension.resolve("tools/bundle-webui.mjs")) outputs.dir(webuiOut) workingDir = extension diff --git a/mide-plugins/clion/gradle.properties b/mide-plugins/clion/gradle.properties index fc9d16f..d53b182 100644 --- a/mide-plugins/clion/gradle.properties +++ b/mide-plugins/clion/gradle.properties @@ -2,10 +2,10 @@ # where it installs (2026.1 and later); the verifier checks both ends. platformType = CL platformVersion = 2026.2.3 -pluginVersion = 1.2.0 +pluginVersion = 1.3.0 # The mcppls CLion plugin this one depends on (io.github.sunrisepeak.mcppls): -# this machine's package of the release, fetched for the sandbox, the tests and the verifier. -mcpplsPluginVersion = 0.0.19 +# its release's one package for every system, fetched for the sandbox, the tests and the verifier. +mcpplsPluginVersion = 0.21.0 org.gradle.jvmargs = -Xmx2g # The IDE ships the Kotlin standard library; a plugin bundles no second copy. kotlin.stdlib.default.dependency = false diff --git a/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/run/McppStates.kt b/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/run/McppStates.kt index d9395a4..24aeafe 100644 --- a/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/run/McppStates.kt +++ b/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/run/McppStates.kt @@ -118,94 +118,31 @@ class McppJobState(private val c: McppRunConfiguration, env: ExecutionEnvironmen } /** - * A test job whose results fill the IDE's test tree: each `test/result` becomes the TeamCity - * service messages the test runner reads, located at the file that holds the test's `main`. + * A test job whose results fill the IDE's test tree (design 2026-11-10), as TeamCity service messages + * of an ID-based tree, so that the parts of a program that run at once report into their own nodes: a + * program is a suite, a file of a program of several files (`unit/a` of `unit`) a suite inside it, and + * each case a test, located at its line. A case starts and finishes as the program writes it when the + * project's mcpp states it (`test/caseEvent`), else with its part's record (`test/result`); a program + * without cases is one test. A failure that states both sides is a comparison failure, whose "Click to + * see difference" opens the IDE's diff; a case's metrics are said in its output beside their bounds. * * EVERY TEST AND CASE HAS ITS OWN OUTPUT (design 2026-11-07, D16): `testStdOut` carries a program's * whole output and a case's own (GoogleTest's console between `[ RUN ]` and its result), passed or * not; a failure's `details` start with the place its output names (`file:line`, which the console * filter links), then the output. */ -class TestJobProcessHandler(session: McppSession, params: JsonObject, private val mains: Map, method: String = "job/start") : +class TestJobProcessHandler(session: McppSession, params: JsonObject, places: Map, method: String = "job/start") : JobProcessHandler(session, params, method) { - private fun escape(s: String) = s.replace("|", "||").replace("'", "|'").replace("\n", "|n").replace("\r", "|r").replace("[", "|[").replace("]", "|]") - - /** `path:line` of a result's `location` (its line 0-based), or "". */ - private fun place(p: JsonObject): String = runCatching { - val l = p.getAsJsonObject("location") - "${l["path"].asString}:${l["line"].asInt + 1}" - }.getOrDefault("") - - /** The output a result carries, as its test's standard output, with a note when it was cut short. */ - private fun stdout(out: StringBuilder, name: String, p: JsonObject) { - val output = p["output"]?.takeIf { it.isJsonPrimitive }?.asString ?: "" - if (output.isEmpty()) return - val note = if (p["outputTruncated"]?.asBoolean == true) "\n(the output was longer; its head and tail are shown)\n" else "" - out.append("##teamcity[testStdOut name='$name' out='${escape(output + note)}']\n") - } - - /** `testFailed` with the failure's place first in its details, then what the result says. */ - private fun failed(out: StringBuilder, name: String, message: String, p: JsonObject, details: String) { - val at = place(p) - val text = listOf(at, details).filter { it.isNotBlank() }.joinToString("\n") - out.append("##teamcity[testFailed name='$name' message='${escape(message)}' details='${escape(text)}']\n") - } - /** The program whose cases are reported, once its first case is (test/runCases). */ - private var suite: String? = null + private val tree = TestTree(places, { session.t(it) }) { root, program, case, file -> caseUrl(session, root, program, case, file) } override fun handle(method: String, p: JsonObject) { when (method) { - // A case of a framework (§7.4): the program is a suite, each case a test in it. - "test/caseResult" -> { - val program = p["test"].asString - val out = StringBuilder() - if (suite == null) { - suite = program - val hint = mains[program]?.let { " locationHint='file://${escape(it)}'" } ?: "" - out.append("##teamcity[testSuiteStarted name='${escape(program)}'$hint]\n") - } - val name = escape(p["case"].asString) - // The case's place in its source, for the editor's ▶ to show the last outcome (TestStateStorage). - val hint = caseUrl(session, p["root"]?.asString ?: "", program, p["case"].asString, mains[program]) - ?.let { " locationHint='${escape(it)}'" } ?: "" - out.append("##teamcity[testStarted name='$name'$hint]\n") - stdout(out, name, p) - when (p["status"]?.asString) { - "passed" -> {} - "skipped" -> out.append("##teamcity[testIgnored name='$name']\n") - else -> { - val message = p["message"]?.asString ?: "" - failed(out, name, message.lineSequence().firstOrNull() ?: "", p, message) - } - } - out.append("##teamcity[testFinished name='$name' duration='${p["durationMs"]?.asLong ?: 0}']\n") - notifyTextAvailable(out.toString(), ProcessOutputTypes.STDOUT) - } + "test/caseEvent", "test/caseResult", "test/result" -> tree.messages(method, p).takeIf { it.isNotEmpty() } + ?.let { notifyTextAvailable(it, ProcessOutputTypes.STDOUT) } "job/finished" -> { - suite?.let { notifyTextAvailable("##teamcity[testSuiteFinished name='${escape(it)}']\n", ProcessOutputTypes.STDOUT) } + tree.close().takeIf { it.isNotEmpty() }?.let { notifyTextAvailable(it, ProcessOutputTypes.STDOUT) } super.handle(method, p) } - "test/result" -> { - val name = escape(p["test"].asString) - val hint = mains[p["test"].asString]?.let { " locationHint='file://${escape(it)}'" } ?: "" - val out = StringBuilder("##teamcity[testStarted name='$name'$hint]\n") - stdout(out, name, p) - val reason = p["reason"]?.asString?.ifBlank { null } - when (p["status"]?.asString) { - "passed" -> {} - "skipped" -> out.append("##teamcity[testIgnored name='$name'${reason?.let { " message='${escape(it)}'" } ?: ""}]\n") - else -> { - val message = when { - p["timedOut"]?.asBoolean == true -> "The test timed out" - p["compileFailed"]?.asBoolean == true -> "The test did not compile" - else -> reason ?: "The test failed (${p["status"].asString})" - } - failed(out, name, message, p, "") - } - } - out.append("##teamcity[testFinished name='$name' duration='${p["durationMs"]?.asLong ?: 0}']\n") - notifyTextAvailable(out.toString(), ProcessOutputTypes.STDOUT) - } else -> super.handle(method, p) } } @@ -214,9 +151,17 @@ class TestJobProcessHandler(session: McppSession, params: JsonObject, private va class McppTestState(private val c: McppRunConfiguration, env: ExecutionEnvironment) : CommandLineState(env) { override fun startProcess(): ProcessHandler { val session = McppSession.get(c.project) - val mains = runCatching { - session.request("test/list", JsonObject().apply { addProperty("root", c.root) }).get() - .asJsonObject.getAsJsonArray("tests").associate { it.asJsonObject["test"].asString to it.asJsonObject["main"].asString } + // Where each program and each part of one is: its `main` (else its first file), and each part's file. + val places = runCatching { + val out = mutableMapOf() + for (t in session.request("test/list", JsonObject().apply { addProperty("root", c.root) }).get().asJsonObject.getAsJsonArray("tests")) { + val o = t.asJsonObject + val name = o["test"].asString + out[name] = o["main"]?.takeIf { it.isJsonPrimitive }?.asString?.ifEmpty { null } + ?: o.getAsJsonArray("files")?.firstOrNull()?.asString ?: "" + o.getAsJsonArray("parts")?.forEach { part -> out[part.asJsonObject["name"].asString] = part.asJsonObject["file"].asString } + } + out.toMap() }.getOrDefault(emptyMap()) // A framework's cases (test/runCases): the program receives the framework's filter. if (c.options.framework.isNotBlank() && c.options.filter.isNotBlank()) @@ -225,16 +170,16 @@ class McppTestState(private val c: McppRunConfiguration, env: ExecutionEnvironme addProperty("test", c.options.filter) addProperty("framework", c.options.framework) add("cases", JsonArray().apply { c.options.cases.lines().filter { it.isNotBlank() }.forEach { add(it) } }) - }, mains, "test/runCases") + }, places, "test/runCases") return TestJobProcessHandler(session, jobParams(c, "test").apply { if (c.options.filter.isNotBlank()) addProperty("filter", c.options.filter) if (c.programArgs.isNotEmpty()) add("args", JsonArray().apply { add("--"); c.programArgs.forEach { add(it) } }) - }, mains) + }, places) } override fun execute(executor: Executor, runner: ProgramRunner<*>): ExecutionResult { val handler = startProcess() - val properties = SMTRunnerConsoleProperties(c, "mcpp", executor) + val properties = SMTRunnerConsoleProperties(c, "mcpp", executor).apply { isIdBasedTestTree = true } val console = SMTestRunnerConnectionUtil.createAndAttachConsole("mcpp", handler, properties) return DefaultExecutionResult(console, handler) } @@ -404,12 +349,12 @@ class McppProgramState(private val c: McppRunConfiguration, env: ExecutionEnviro override fun startProcess(): ProcessHandler = LaunchingProcessHandler(c) } -/** The URL of a case: its program's source and the line of its macro (`file://:`, 1-based). */ +/** The URL of a case: the file that declares it (else its program's) and its line (`file://:`, 1-based). */ internal fun caseUrl(session: McppSession, root: String, test: String, case: String, main: String?): String? { - val file = main?.takeIf { it.isNotEmpty() } ?: return null val cases = session.casesFor(root, test) ?: return null - val line = cases.getAsJsonArray("cases")?.firstOrNull { it.asJsonObject["name"]?.asString == case } - ?.asJsonObject?.get("line")?.asInt ?: return null + val row = cases.getAsJsonArray("cases")?.firstOrNull { it.asJsonObject["name"]?.asString == case }?.asJsonObject ?: return null + val file = row["file"]?.asString?.takeIf { it.isNotEmpty() } ?: main?.takeIf { it.isNotEmpty() } ?: return null + val line = row["line"]?.asInt ?: return null return "file://$file:${line + 1}" } diff --git a/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/run/TestTree.kt b/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/run/TestTree.kt new file mode 100644 index 0000000..3acfb10 --- /dev/null +++ b/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/run/TestTree.kt @@ -0,0 +1,161 @@ +// The test tree of a test job (design 2026-11-10): the TeamCity service messages of an ID-based tree +// that the IDE's test runner reads, made from the backend's notifications. Kept apart from the process +// handler that sends them, so that it is tested without an IDE. +package io.github.mcppcommunity.mcppclion.run + +import com.google.gson.JsonObject + +class TestTree( + private val places: Map, + private val t: (String) -> String, + private val locate: (root: String, program: String, case: String, file: String?) -> String?, +) { + private fun escape(s: String) = s.replace("|", "||").replace("'", "|'").replace("\n", "|n").replace("\r", "|r").replace("[", "|[").replace("]", "|]") + + /** `path:line` of a result's `location` (its line 0-based), or "". */ + private fun place(p: JsonObject): String = runCatching { + val l = p.getAsJsonObject("location") + "${l["path"].asString}:${l["line"].asInt + 1}" + }.getOrDefault("") + + // The suites opened, in order (closed at the end, innermost first), and the tests already reported. + private val suites = mutableListOf() + private val started = mutableSetOf() + private val finished = mutableSetOf() + + private val out = StringBuilder() + private fun send(text: String) { out.append(text) } + + private fun hint(file: String?) = file?.takeIf { it.isNotEmpty() }?.let { " locationHint='file://${escape(it)}'" } ?: "" + + /** The suite of a program, and of the part of it a record or a case belongs to; opened once. */ + private fun parentOf(program: String, part: String?): String { + val programNode = "p:$program" + if (programNode !in suites) { + send("##teamcity[testSuiteStarted nodeId='${escape(programNode)}' parentNodeId='0' name='${escape(program)}'${hint(places[program])}]\n") + suites += programNode + } + if (part.isNullOrEmpty() || part == program) return programNode + val partNode = "f:$part" + if (partNode !in suites) { + val name = part.removePrefix("$program/") + send("##teamcity[testSuiteStarted nodeId='${escape(partNode)}' parentNodeId='${escape(programNode)}' name='${escape(name)}'${hint(places[part])}]\n") + suites += partNode + } + return partNode + } + + private fun start(node: String, parent: String, name: String, hint: String) { + if (!started.add(node)) return + send("##teamcity[testStarted nodeId='${escape(node)}' parentNodeId='${escape(parent)}' name='${escape(name)}'$hint]\n") + } + + /** A number as a person writes it: `15`, `4.2`. */ + private fun number(d: Double): String = if (d % 1.0 == 0.0 && kotlin.math.abs(d) < 1e15) d.toLong().toString() else d.toString() + + /** A metric against its bound: `ready 4.2 s ≤ 15 s`. */ + private fun metrics(p: JsonObject): String { + val bounds = p.getAsJsonArray("bounds")?.mapNotNull { it.takeIf { b -> b.isJsonObject }?.asJsonObject } ?: emptyList() + return (p.getAsJsonArray("metrics") ?: return "").mapNotNull { m -> + val o = m.takeIf { it.isJsonObject }?.asJsonObject ?: return@mapNotNull null + val name = o["name"]?.asString ?: return@mapNotNull null + val unit = o["unit"]?.asString?.takeIf { it != "1" }?.let { " $it" } ?: "" + val b = bounds.firstOrNull { it["metric"]?.asString == name } + val limit = b?.get("max")?.let { " ≤ ${number(it.asDouble)}$unit" } ?: b?.get("min")?.let { " ≥ ${number(it.asDouble)}$unit" } ?: "" + "$name ${o["value"]?.asDouble?.let(::number) ?: ""}$unit$limit" + }.joinToString("\n") + } + + /** A case as it ended: its output, its verdict, its duration. */ + private fun outcome(node: String, p: JsonObject, output: String) { + if (!finished.add(node)) return + val text = listOf(output, metrics(p)).filter { it.isNotBlank() }.joinToString("\n") + if (text.isNotEmpty()) send("##teamcity[testStdOut nodeId='${escape(node)}' out='${escape(text + "\n")}']\n") + when (p["status"]?.asString) { + "passed" -> {} + "skipped" -> send("##teamcity[testIgnored nodeId='${escape(node)}' message='${escape(p["reason"]?.asString ?: "")}']\n") + else -> { + val failure = p.getAsJsonArray("failures")?.firstOrNull()?.takeIf { it.isJsonObject }?.asJsonObject + val message = when { + p["unexpectedPass"]?.asBoolean == true -> t("The case passed, though it is expected to fail.") + else -> failure?.get("message")?.asString ?: p["message"]?.asString ?: "" + } + val details = listOf(place(p), p["message"]?.asString ?: "").filter { it.isNotBlank() }.joinToString("\n") + val sides = if (failure != null && failure.has("expected") && failure.has("actual")) + " type='comparisonFailure' expected='${escape(failure["expected"].asString)}' actual='${escape(failure["actual"].asString)}'" else "" + send("##teamcity[testFailed nodeId='${escape(node)}' message='${escape(message)}' details='${escape(details)}'$sides]\n") + } + } + send("##teamcity[testFinished nodeId='${escape(node)}' duration='${p["durationMs"]?.asLong ?: 0}']\n") + } + + /** A case of a program, under its part's suite, at its line. */ + private fun case(program: String, part: String?, p: JsonObject, output: String?) { + val name = p["case"]?.asString ?: return + val node = "c:$program\u001f$name" + val parent = parentOf(program, part) + val url = locate(p["root"]?.asString ?: "", program, name, places[part ?: program] ?: places[program]) + start(node, parent, name, url?.let { " locationHint='${escape(it)}'" } ?: "") + if (output != null) outcome(node, p, output) + } + + /** The service messages a test job's notification becomes. */ + fun messages(method: String, p: JsonObject): String { + when (method) { + // A case as the program writes it (test-case-stream, mcpp#917). + "test/caseEvent" -> { + val program = p["program"]?.asString ?: return "" + case(program, p["part"]?.asString, p, if (p["event"]?.asString == "end") "" else null) + } + // A case of a framework or of the test protocol (test/runCases). + "test/caseResult" -> { + val program = p["program"]?.asString ?: p["test"].asString + case(program, p["part"]?.asString, p, p["output"]?.takeIf { it.isJsonPrimitive }?.asString ?: "") + } + "test/result" -> { + val program = p["program"]?.asString ?: p["test"].asString + val part = p["part"]?.asString + val cases = p.getAsJsonArray("cases")?.mapNotNull { it.takeIf { c -> c.isJsonObject }?.asJsonObject } ?: emptyList() + if (cases.isEmpty()) result(program, part, p) + for (c in cases) case(program, part, c.deepCopy().apply { add("root", p["root"]) }, if (c["status"]?.asString == "failed") c["message"]?.asString ?: "" else "") + } + } + return drain() + } + + /** The job ended: every suite opened closes, innermost first. */ + fun close(): String { + for (node in suites.asReversed()) send("##teamcity[testSuiteFinished nodeId='${escape(node)}']\n") + suites.clear() + return drain() + } + + private fun drain(): String = out.toString().also { out.setLength(0) } + + /** A program, or a part of one, without cases: one test, judged by mcpp's status. */ + private fun result(program: String, part: String?, p: JsonObject) { + val name = part ?: program + val node = "t:$name" + val parent = if (part != null) parentOf(program, null) else "0" + start(node, parent, if (part != null) part.removePrefix("$program/") else program, hint(places[name])) + val output = p["output"]?.takeIf { it.isJsonPrimitive }?.asString ?: "" + val note = if (p["outputTruncated"]?.asBoolean == true) "\n(the output was longer; its head and tail are shown)\n" else "" + if (output.isNotEmpty()) send("##teamcity[testStdOut nodeId='${escape(node)}' out='${escape(output + note)}']\n") + val reason = p["reason"]?.asString?.ifBlank { null } + when (p["status"]?.asString) { + "passed" -> {} + "skipped" -> send("##teamcity[testIgnored nodeId='${escape(node)}'${reason?.let { " message='${escape(it)}'" } ?: ""}]\n") + else -> { + val message = when { + p["timedOut"]?.asBoolean == true -> "The test timed out" + p["compileFailed"]?.asBoolean == true -> "The test did not compile" + else -> reason ?: "The test failed (${p["status"].asString})" + } + send("##teamcity[testFailed nodeId='${escape(node)}' message='${escape(message)}' details='${escape(place(p))}']\n") + } + } + finished += node + send("##teamcity[testFinished nodeId='${escape(node)}' duration='${p["durationMs"]?.asLong ?: 0}']\n") + } +} + diff --git a/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/ui/GraphPage.kt b/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/ui/GraphPage.kt index 6ebd527..bc23453 100644 --- a/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/ui/GraphPage.kt +++ b/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/ui/GraphPage.kt @@ -4,115 +4,34 @@ // nothing else. Without JCEF the tab says so. package io.github.mcppcommunity.mcppclion.ui +import com.google.gson.JsonArray import com.google.gson.JsonObject import com.google.gson.JsonParser -import com.intellij.openapi.Disposable -import com.intellij.openapi.application.ApplicationManager -import com.intellij.openapi.application.PathManager -import com.intellij.openapi.fileEditor.OpenFileDescriptor import com.intellij.openapi.project.Project -import com.intellij.openapi.util.Disposer -import com.intellij.openapi.vfs.LocalFileSystem import com.intellij.ui.JBColor -import com.intellij.ui.components.JBTextArea -import com.intellij.ui.jcef.JBCefApp -import com.intellij.ui.jcef.JBCefBrowser -import com.intellij.ui.jcef.JBCefBrowserBase -import com.intellij.ui.jcef.JBCefJSQuery -import com.intellij.util.ui.JBUI import io.github.mcppcommunity.mcppclion.McppSession -import org.cef.browser.CefBrowser -import org.cef.browser.CefFrame -import org.cef.handler.CefLoadHandlerAdapter -import java.io.File -import javax.swing.JComponent -class GraphPage(private val project: Project, private val kind: String) : McppView, Disposable { +class GraphPage(project: Project, private val kind: String) : + JcefPage(project, "graph.html", McppSession.get(project).t("This IDE runs without JCEF, which draws the graph.")) { override val id = "graph:$kind" - private val session = McppSession.get(project) - private val browser: JBCefBrowser? = if (JBCefApp.isSupported()) JBCefBrowser() else null - private val query: JBCefJSQuery? = browser?.let { JBCefJSQuery.create(it as JBCefBrowserBase) } - @Volatile private var loaded = false - private var pending: String? = null - // Without JCEF the page says why. The tool window sits on the left beside Project, narrower than - // the sentence, so the text wraps; its accessible name is the whole sentence. - val component: JComponent = browser?.component ?: unavailable(session.t("This IDE runs without JCEF, which draws the graph.")) - - init { - if (browser != null && query != null) { - Disposer.register(this, browser) - Disposer.register(this, query) - query.addHandler { text -> open(text); null } - browser.jbCefClient.addLoadHandler(object : CefLoadHandlerAdapter() { - override fun onLoadEnd(b: CefBrowser, frame: CefFrame, status: Int) { - if (!frame.isMain) return - b.executeJavaScript("window.mcppHostSend = function (text) { ${query.inject("text")} };", b.url, 0) - ApplicationManager.getApplication().invokeLater { - loaded = true - pending?.let { render(it) } - pending = null - } - } - }, browser.cefBrowser) - browser.loadURL(page().toURI().toString()) - } - } - - /** The page's files, written once per build of the plugin where the browser can load them. */ - private fun page(): File { - val jar = runCatching { File(GraphPage::class.java.protectionDomain.codeSource.location.toURI()) }.getOrNull() - val version = jar?.takeIf { it.isFile }?.let { "${it.nameWithoutExtension}-${it.lastModified()}" } ?: "dev" - val dir = File(PathManager.getSystemPath(), "mcpp-clion/webui-$version") - val index = javaClass.getResource("/webui/files.txt")?.readText() ?: error("the graph page is not in the plugin (webui/files.txt)") - for (f in index.lines().filter { it.isNotBlank() }) { - val target = File(dir, f) - if (target.isFile && version != "dev") continue - target.parentFile.mkdirs() - javaClass.getResourceAsStream("/webui/$f")?.use { input -> target.outputStream().use { input.copyTo(it) } } - } - return File(dir, "graph.html") - } override fun refresh() { val root = session.root ?: return session.request("graph/$kind", JsonObject().apply { addProperty("root", root) }).whenComplete { graph, error -> - val page = JsonObject().apply { + show(JsonObject().apply { addProperty("kind", kind) - add("graph", graph?.takeIf { it.isJsonObject } ?: JsonObject().apply { add("nodes", com.google.gson.JsonArray()); add("edges", com.google.gson.JsonArray()) }) + add("graph", graph?.takeIf { it.isJsonObject } ?: JsonObject().apply { add("nodes", JsonArray()); add("edges", JsonArray()) }) addProperty("title", if (kind == "modules") session.t("Module Graph") else session.t("Dependency Graph")) addProperty("empty", error?.let { session.t("mcpp: {0}", (it.cause ?: it).message ?: "") } ?: session.t("Nothing to draw yet: build the project first.")) addProperty("truncated", session.t("Only the first {0} nodes are drawn.")) addProperty("dark", !JBColor.isBright()) - } - ApplicationManager.getApplication().invokeLater { - if (loaded) render(page.toString()) else pending = page.toString() - } + }) } } - private fun render(page: String) { - val b = browser ?: return - b.cefBrowser.executeJavaScript("window.mcppRender && window.mcppRender($page);", b.cefBrowser.url, 0) - } - /** `{"open": ""}`: the module's source in an editor. */ - private fun open(text: String) { + override fun received(text: String) { val source = runCatching { JsonParser.parseString(text).asJsonObject["open"]?.asString }.getOrNull() ?: return - val root = session.root ?: return - val path = File(source).let { if (it.isAbsolute) it else File(root, source) }.path - ApplicationManager.getApplication().invokeLater { - LocalFileSystem.getInstance().refreshAndFindFileByPath(path)?.let { OpenFileDescriptor(project, it).navigate(true) } - } + openSource(source) } - - override fun dispose() {} -} - -private fun unavailable(text: String): JComponent = JBTextArea(text).apply { - isEditable = false - lineWrap = true - wrapStyleWord = true - isOpaque = false - border = JBUI.Borders.empty(12) - accessibleContext.accessibleName = text } diff --git a/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/ui/JcefPage.kt b/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/ui/JcefPage.kt new file mode 100644 index 0000000..2b19a71 --- /dev/null +++ b/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/ui/JcefPage.kt @@ -0,0 +1,108 @@ +// A page of webui/ in JCEF (mcpp design 2026-10-30 part 2 §9): the plugin's copy of the page's files, +// a browser that loads one of them, `mcppRender` to hand the page the backend's answer and its strings, +// and `mcppHostSend` for what the page asks of the IDE. A page reaches nothing else. Without JCEF the +// tab says why. +package io.github.mcppcommunity.mcppclion.ui + +import com.google.gson.JsonObject +import com.intellij.openapi.Disposable +import com.intellij.openapi.application.ApplicationManager +import com.intellij.openapi.application.PathManager +import com.intellij.openapi.fileEditor.OpenFileDescriptor +import com.intellij.openapi.project.Project +import com.intellij.openapi.util.Disposer +import com.intellij.openapi.vfs.LocalFileSystem +import com.intellij.ui.components.JBTextArea +import com.intellij.ui.jcef.JBCefApp +import com.intellij.ui.jcef.JBCefBrowser +import com.intellij.ui.jcef.JBCefBrowserBase +import com.intellij.ui.jcef.JBCefJSQuery +import com.intellij.util.ui.JBUI +import io.github.mcppcommunity.mcppclion.McppSession +import org.cef.browser.CefBrowser +import org.cef.browser.CefFrame +import org.cef.handler.CefLoadHandlerAdapter +import java.io.File +import javax.swing.JComponent + +abstract class JcefPage(protected val project: Project, private val file: String, withoutJcef: String) : McppView, Disposable { + protected val session = McppSession.get(project) + private val browser: JBCefBrowser? = if (JBCefApp.isSupported()) JBCefBrowser() else null + private val query: JBCefJSQuery? = browser?.let { JBCefJSQuery.create(it as JBCefBrowserBase) } + @Volatile private var loaded = false + private var pending: String? = null + // Without JCEF the page says why. The tool window sits on the left beside Project, narrower than + // the sentence, so the text wraps; its accessible name is the whole sentence. + val component: JComponent = browser?.component ?: unavailable(withoutJcef) + + init { + if (browser != null && query != null) { + Disposer.register(this, browser) + Disposer.register(this, query) + query.addHandler { text -> received(text); null } + browser.jbCefClient.addLoadHandler(object : CefLoadHandlerAdapter() { + override fun onLoadEnd(b: CefBrowser, frame: CefFrame, status: Int) { + if (!frame.isMain) return + b.executeJavaScript("window.mcppHostSend = function (text) { ${query.inject("text")} };", b.url, 0) + ApplicationManager.getApplication().invokeLater { + loaded = true + pending?.let { render(it) } + pending = null + } + } + }, browser.cefBrowser) + browser.loadURL(page().toURI().toString()) + } + } + + /** What the page sent (`mcppHostSend`). */ + protected abstract fun received(text: String) + + /** The page's files, written once per build of the plugin where the browser can load them. */ + private fun page(): File { + val jar = runCatching { File(JcefPage::class.java.protectionDomain.codeSource.location.toURI()) }.getOrNull() + val version = jar?.takeIf { it.isFile }?.let { "${it.nameWithoutExtension}-${it.lastModified()}" } ?: "dev" + val dir = File(PathManager.getSystemPath(), "mcpp-clion/webui-$version") + val index = javaClass.getResource("/webui/files.txt")?.readText() ?: error("the pages are not in the plugin (webui/files.txt)") + for (f in index.lines().filter { it.isNotBlank() }) { + val target = File(dir, f) + if (target.isFile && version != "dev") continue + target.parentFile.mkdirs() + javaClass.getResourceAsStream("/webui/$f")?.use { input -> target.outputStream().use { input.copyTo(it) } } + } + return File(dir, file) + } + + /** Hands the page what it draws, once it has loaded. */ + protected fun show(page: JsonObject) { + val text = page.toString() + ApplicationManager.getApplication().invokeLater { + if (loaded) render(text) else pending = text + } + } + + private fun render(page: String) { + val b = browser ?: return + b.cefBrowser.executeJavaScript("window.mcppRender && window.mcppRender($page);", b.cefBrowser.url, 0) + } + + /** A file of the project in an editor, at a 0-based line. */ + protected fun openSource(source: String, line: Int = 0) { + val root = session.root ?: return + val path = File(source).let { if (it.isAbsolute) it else File(root, source) }.path + ApplicationManager.getApplication().invokeLater { + LocalFileSystem.getInstance().refreshAndFindFileByPath(path)?.let { OpenFileDescriptor(project, it, line, 0).navigate(true) } + } + } + + override fun dispose() {} +} + +private fun unavailable(text: String): JComponent = JBTextArea(text).apply { + isEditable = false + lineWrap = true + wrapStyleWord = true + isOpaque = false + border = JBUI.Borders.empty(12) + accessibleContext.accessibleName = text +} diff --git a/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/ui/McppToolWindow.kt b/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/ui/McppToolWindow.kt index f02efe3..36f1b35 100644 --- a/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/ui/McppToolWindow.kt +++ b/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/ui/McppToolWindow.kt @@ -52,6 +52,7 @@ object McppPanels { window.activate { when (panel) { "graph" -> openGraph(project, window, params["graph"]?.asString ?: "packages") + "tests" -> openTests(project, window) "project" -> select(window, "project", params) "cache" -> select(window, "cache", params) "libraries", "library" -> select(window, "libraries", params) @@ -88,6 +89,21 @@ object McppPanels { select(window, content, JsonObject()) } + /** The test insights are a tab of their own, closeable, drawn again when asked again. */ + private fun openTests(project: Project, window: ToolWindow) { + window.contentManager.contents.firstOrNull { it.getUserData(VIEW)?.id == "tests" }?.let { + select(window, it, JsonObject()) + return + } + val page = TestsPage(project) + val content = ContentFactory.getInstance().createContent(page.component, McppSession.get(project).t("Test Insights"), false) + content.isCloseable = true + content.putUserData(VIEW, page) + Disposer.register(content, page) + window.contentManager.addContent(content) + select(window, content, JsonObject()) + } + class Factory : ToolWindowFactory, DumbAware { override suspend fun isApplicableAsync(project: Project): Boolean = project.basePath?.let { File(it, "mcpp.toml").isFile } ?: false diff --git a/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/ui/TestsPage.kt b/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/ui/TestsPage.kt new file mode 100644 index 0000000..24a6e38 --- /dev/null +++ b/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/ui/TestsPage.kt @@ -0,0 +1,61 @@ +// What the last runs of the project's tests say (design 2026-11-10) -- the run, its failures and since +// when, each metric against its bound, the slowest and the flaky cases, the schedule with its critical +// path -- as the page webui/tests.html draws the backend's `test/insights` in JCEF, the store mcpp keeps +// read by the backend. It is drawn again when a test job ends; a click on a failure opens its place. +package io.github.mcppcommunity.mcppclion.ui + +import com.google.gson.JsonObject +import com.google.gson.JsonParser +import com.intellij.openapi.project.Project +import com.intellij.ui.JBColor +import io.github.mcppcommunity.mcppclion.McppSession + +class TestsPage(project: Project) : + JcefPage(project, "tests.html", McppSession.get(project).t("This IDE runs without JCEF, which draws the test insights.")) { + override val id = "tests" + private val unsubscribe = session.on("job/finished") { p -> + if (p.isJsonObject && p.asJsonObject["kind"]?.asString == "test") refresh() + } + + private fun labels() = JsonObject().apply { + addProperty("title", session.t("Test Insights")) + addProperty("noRun", session.t("No run is kept yet: mcpp test keeps each run in {0} (mcpp 2026.11.9.1 and later).")) + addProperty("window", session.t("{0} runs kept · the last began {1}")) + addProperty("failures", session.t("Failures")) + addProperty("fresh", session.t("new")) + addProperty("since", session.t("since {0}")) + addProperty("expected", session.t("expected")) + addProperty("actual", session.t("actual")) + addProperty("metrics", session.t("Metrics")) + addProperty("slowest", session.t("Slowest")) + addProperty("median", session.t("median {0}")) + addProperty("flaky", session.t("Flaky")) + addProperty("flakyCounts", session.t("{0} passed · {1} failed · {2} changes")) + addProperty("schedule", session.t("Schedule")) + addProperty("worker", session.t("worker {0}")) + addProperty("criticalPath", session.t("Critical path")) + addProperty("noMetrics", session.t("Metrics appear when the project's mcpp states them (test-metrics, mcpp#917).")) + } + + override fun refresh() { + val root = session.root ?: return + session.request("test/insights", JsonObject().apply { addProperty("root", root) }).whenComplete { insights, _ -> + if (insights == null || !insights.isJsonObject) return@whenComplete + show(JsonObject().apply { + add("insights", insights) + add("labels", labels()) + addProperty("dark", !JBColor.isBright()) + }) + } + } + + /** `{"open": {"path": ..., "line": ...}}`: a failure's place in an editor. */ + override fun received(text: String) { + val at = runCatching { JsonParser.parseString(text).asJsonObject["open"]?.asJsonObject }.getOrNull() ?: return + openSource(at["path"]?.asString ?: return, at["line"]?.asInt ?: 0) + } + + override fun dispose() { + unsubscribe() + } +} diff --git a/mide-plugins/clion/src/test/kotlin/io/github/mcppcommunity/mcppclion/TestTreeTest.kt b/mide-plugins/clion/src/test/kotlin/io/github/mcppcommunity/mcppclion/TestTreeTest.kt new file mode 100644 index 0000000..d0e7256 --- /dev/null +++ b/mide-plugins/clion/src/test/kotlin/io/github/mcppcommunity/mcppclion/TestTreeTest.kt @@ -0,0 +1,55 @@ +package io.github.mcppcommunity.mcppclion + +import com.google.gson.JsonObject +import com.google.gson.JsonParser +import io.github.mcppcommunity.mcppclion.run.TestTree +import org.junit.Assert.assertEquals +import org.junit.Assert.assertTrue +import org.junit.Test + +// The test tree of a test job (design 2026-11-10): a program of several files is a suite with a suite +// per file, each case a test inside its file's suite; a case that begins and then ends is reported +// once; a failure with both sides is a comparison failure; the suites close at the end, innermost first. +class TestTreeTest { + private fun json(text: String): JsonObject = JsonParser.parseString(text).asJsonObject + private fun tree() = TestTree(mapOf("unit" to "/p/tests/unit/a.cpp", "unit/b" to "/p/tests/unit/b.cpp"), { it }) { _, _, case, file -> "file://$file:$case" } + + @Test + fun aProgramOfSeveralFilesNestsItsCasesInTheirFiles() { + val tree = tree() + val text = tree.messages("test/result", json("""{"root":"/p","test":"unit/b","program":"unit","part":"unit/b","status":"failed", + "cases":[{"case":"beta","status":"failed","durationMs":2,"message":"b.cpp:5: sum == 4", + "failures":[{"path":"tests/unit/b.cpp","line":4,"message":"sum == 4","expected":"4","actual":"5"}], + "location":{"path":"tests/unit/b.cpp","line":4}}]}""")) + assertTrue(text, text.contains("testSuiteStarted nodeId='p:unit' parentNodeId='0' name='unit' locationHint='file:///p/tests/unit/a.cpp'")) + assertTrue(text, text.contains("testSuiteStarted nodeId='f:unit/b' parentNodeId='p:unit' name='b'")) + assertTrue(text, text.contains("testStarted nodeId='c:unit\u001fbeta' parentNodeId='f:unit/b' name='beta'")) + assertTrue(text, text.contains("type='comparisonFailure' expected='4' actual='5'")) + assertTrue(text, text.contains("details='tests/unit/b.cpp:5")) + assertTrue(text, text.contains("testFinished nodeId='c:unit\u001fbeta' duration='2'")) + assertEquals("##teamcity[testSuiteFinished nodeId='f:unit/b']\n##teamcity[testSuiteFinished nodeId='p:unit']\n", tree.close()) + } + + @Test + fun aLiveCaseIsReportedOnceWithItsMetrics() { + val tree = tree() + val begin = tree.messages("test/caseEvent", json("""{"root":"/p","program":"unit","part":"unit/a","case":"alpha","event":"begin"}""")) + assertTrue(begin, begin.contains("testStarted nodeId='c:unit\u001falpha'") && !begin.contains("testFinished")) + val end = tree.messages("test/caseEvent", json("""{"root":"/p","program":"unit","part":"unit/a","case":"alpha","event":"end","status":"passed", + "durationMs":1,"metrics":[{"name":"ready","value":4.2,"unit":"s"}],"bounds":[{"metric":"ready","max":15}]}""")) + assertTrue(end, !end.contains("testStarted") && end.contains("ready 4.2 s ≤ 15 s") && end.contains("testFinished nodeId='c:unit\u001falpha' duration='1'")) + // The part's record comes after: its cases were already reported. + val record = tree.messages("test/result", json("""{"root":"/p","test":"unit/a","program":"unit","part":"unit/a","status":"passed", + "cases":[{"case":"alpha","status":"passed","durationMs":1}]}""")) + assertEquals("", record) + } + + @Test + fun aProgramWithoutCasesIsOneTest() { + val tree = tree() + val text = tree.messages("test/result", json("""{"root":"/p","test":"smoke","status":"timedOut","timedOut":true,"durationMs":9,"output":"x","reason":""}""")) + assertTrue(text, text.contains("testStarted nodeId='t:smoke' parentNodeId='0' name='smoke'")) + assertTrue(text, text.contains("testFailed nodeId='t:smoke' message='The test timed out'")) + assertEquals("", tree.close()) + } +} diff --git a/mide-plugins/nvim/lua/mcpp/commands.lua b/mide-plugins/nvim/lua/mcpp/commands.lua index bf143cd..99eef81 100644 --- a/mide-plugins/nvim/lua/mcpp/commands.lua +++ b/mide-plugins/nvim/lua/mcpp/commands.lua @@ -160,6 +160,12 @@ function panels.graph(row, _, root) views.show(row.title, row.action.graph == 'modules' and 'graph/modules' or 'graph/packages', { root = root }, views.graph_text) end +-- What the last runs of the tests say (design 2026-11-10), as text: the store mcpp keeps, read by +-- the backend. +function panels.tests(row, _, root) + views.show(row.title, 'test/insights', { root = root }, tests.insights_text) +end + -- The pack panel's choices one after another: the format, the closure, the linkage where mcpp takes -- one; then the pack, and what it wrote. function panels.pack(row, _, root) diff --git a/mide-plugins/nvim/lua/mcpp/session.lua b/mide-plugins/nvim/lua/mcpp/session.lua index ace26d3..86d4b05 100644 --- a/mide-plugins/nvim/lua/mcpp/session.lua +++ b/mide-plugins/nvim/lua/mcpp/session.lua @@ -13,7 +13,7 @@ local uv = vim.uv or vim.loop local is_windows = vim.fn.has('win32') == 1 -- mcpp-nvim's version: mcpp-ide's, whose release carries it. -M.version = '1.2.0' +M.version = '1.3.0' -- Neovim's version, as its release names it. function M.nvim_version() @@ -210,7 +210,7 @@ local function deliver(owner, method, params) on.output(params.text or '') elseif method == 'job/started' and on.started then on.started(params) - elseif (method == 'test/result' or method == 'test/caseResult') and on.test then + elseif (method == 'test/result' or method == 'test/caseResult' or method == 'test/caseEvent') and on.test then on.test(method, params) elseif method == 'job/finished' then state.claimed[params.jobId] = nil @@ -261,12 +261,12 @@ end local JOB_EVENTS = { ['job/started'] = true, ['job/output'] = true, ['job/plan'] = true, ['job/progress'] = true, ['job/diagnostic'] = true, ['job/artifact'] = true, ['job/record'] = true, ['job/finished'] = true, - ['test/result'] = true, ['test/summary'] = true, ['test/caseResult'] = true, + ['test/result'] = true, ['test/summary'] = true, ['test/caseResult'] = true, ['test/caseEvent'] = true, } -- Inside a flow: asks for a job (`job/start`, or another method that answers a job id) and follows --- it. `events`: output(text), started(params), test(method, result) for each `test/result` and --- `test/caseResult`; the handle's wait() answers its end. Returns the +-- it. `events`: output(text), started(params), test(method, result) for each `test/result`, +-- `test/caseResult` and `test/caseEvent`; the handle's wait() answers its end. Returns the -- handle, or nil and the error. function M.job(method, params, events) local result, err = M.call(method, params) diff --git a/mide-plugins/nvim/lua/mcpp/tests.lua b/mide-plugins/nvim/lua/mcpp/tests.lua index 09a0fdf..d899745 100644 --- a/mide-plugins/nvim/lua/mcpp/tests.lua +++ b/mide-plugins/nvim/lua/mcpp/tests.lua @@ -1,9 +1,12 @@ --- What `:Mcpp test` says about each test (design 2026-11-07, D16): a results buffer (`mcpp://tests`) --- with one section per test and per case, its status and its whole output, passed or not; the --- failures, at the place their output names, in the quickfix list; and in the sign column of a --- test's source a ▶ on its `main` and each case, which a result turns into ✓ or ✗. A program's `main` --- has its ▶ too. The places and the statuses are the backend's (`source/entries`, `test/cases`, --- `test/result`, `test/caseResult`). +-- What `:Mcpp test` says about each test (design 2026-11-07, D16; design 2026-11-10): a results buffer +-- (`mcpp://tests`) with one section per test, per part of a program of several files and per case, its +-- status and its whole output, passed or not; the failures, at the place their output names, in the +-- quickfix list, a failure that states both sides with them; and in the sign column of a test's source +-- a ▶ on its `main` and each case, which a result turns into ✓ or ✗, with the case's duration and its +-- metrics against their bounds at the end of its line. Cases arrive as the program writes them when the +-- project's mcpp states them (`test/caseEvent`), else with their part's record. The places and the +-- statuses are the backend's (`source/entries`, `test/cases`, `test/result`, `test/caseResult`, +-- `test/caseEvent`). local session = require('mcpp.session') local M = {} @@ -12,12 +15,18 @@ local t = session.t M.RESULTS = 'mcpp://tests' local NS = vim.api.nvim_create_namespace('mcpp_tests') local results_buf --- The last result of each test and case, by `root\0test` and `root\0test\0case`. +-- The last result of each test and case, by `root\0test` and `root\0test\0case`, and what its line says. local last = {} +local said = {} +-- The cases already reported in this run (a case that streamed is not said again with its record). +local reported = {} +-- Lines waiting to be written: the buffer is written at most every 100 ms while a run streams. +local pending = {} +local flushing = false -local SIGN = { entry = '▶', passed = '✓', failed = '✗', timedOut = '✗', errored = '✗', skipped = '–' } -local HL = { entry = 'DiagnosticInfo', passed = 'DiagnosticOk', failed = 'DiagnosticError', timedOut = 'DiagnosticError', - errored = 'DiagnosticError', skipped = 'DiagnosticHint' } +local SIGN = { entry = '▶', running = '◌', passed = '✓', failed = '✗', timedOut = '✗', errored = '✗', skipped = '–' } +local HL = { entry = 'DiagnosticInfo', running = 'DiagnosticHint', passed = 'DiagnosticOk', failed = 'DiagnosticError', + timedOut = 'DiagnosticError', errored = 'DiagnosticError', skipped = 'DiagnosticHint' } local function results_buffer() if results_buf and vim.api.nvim_buf_is_valid(results_buf) then @@ -31,7 +40,13 @@ local function results_buffer() return results_buf end -local function write(lines) +local function flush() + flushing = false + if #pending == 0 then + return + end + local lines = pending + pending = {} local buf = results_buffer() vim.bo[buf].modifiable = true local count = vim.api.nvim_buf_line_count(buf) @@ -40,6 +55,14 @@ local function write(lines) vim.bo[buf].modifiable = false end +local function write(lines) + vim.list_extend(pending, lines) + if not flushing then + flushing = true + vim.defer_fn(flush, 100) + end +end + local function output_lines(text) local lines = vim.split((text or ''):gsub('\r\n', '\n'):gsub('\27%[[%d;]*[A-Za-z]', ''), '\n', { plain = true }) if lines[#lines] == '' then @@ -71,30 +94,115 @@ local function located(root, location) return { filename = vim.fs.normalize(path), lnum = (location.line or 0) + 1 } end +-- A number as a person writes it: `15`, `4.2`, `0.913`. +local function number(v) + if v == math.floor(v) and math.abs(v) < 1e15 then + return string.format('%d', v) + end + return string.format('%.3g', v) +end + +-- A case's metrics against their bounds: `ready 4.2 s ≤ 15 s`. +function M.metrics_text(r) + local out = {} + for _, m in ipairs(r.metrics or {}) do + local unit = (m.unit and m.unit ~= '1') and (' ' .. m.unit) or '' + local text = string.format('%s %s%s', m.name, number(m.value or 0), unit) + for _, b in ipairs(r.bounds or {}) do + if b.metric == m.name and b.max then + text = text .. ' ≤ ' .. number(b.max) .. unit + elseif b.metric == m.name and b.min then + text = text .. ' ≥ ' .. number(b.min) .. unit + end + end + out[#out + 1] = text + end + return table.concat(out, ' · ') +end + +-- What a case's line says after a run: its duration and its metrics. +local function line_text(r) + local parts = {} + if r.status ~= 'skipped' and r.durationMs then + parts[#parts + 1] = string.format('%d ms', math.floor(r.durationMs)) + end + local metrics = M.metrics_text(r) + if metrics ~= '' then + parts[#parts + 1] = metrics + end + return #parts > 0 and table.concat(parts, ' · ') or nil +end + -- A new run: the buffer and the quickfix list start over. function M.begin(root) + pending = {} local buf = results_buffer() vim.bo[buf].modifiable = true vim.api.nvim_buf_set_lines(buf, 0, -1, false, { '# ' .. t('mcpp test in {0}', root or '?') }) vim.bo[buf].modifiable = false M.failures = {} + reported = {} end --- One result (`test/result` or `test/caseResult`) said in the buffer, kept for the signs, and its --- failure put in the quickfix list. -function M.add(method, r) - local name = method == 'test/caseResult' and (r.test .. ' › ' .. r.case) or r.test - local key = (r.root or '') .. '\0' .. r.test .. (method == 'test/caseResult' and ('\0' .. r.case) or '') +-- One case as it ended: its section, its sign, its failure in the quickfix list. +local function add_case(root, program, part, r) + local key = (root or '') .. '\0' .. program .. '\0' .. r.case + if reported[key] then + return + end + reported[key] = true last[key] = r.status - local lines = { '', string.format('## %s %s — %s (%d ms)', SIGN[r.status] or '?', name, status_text(r), r.durationMs or 0) } + said[key] = line_text(r) + local where = part and part ~= program and (part .. ' › ') or (program .. ' › ') + local lines = { '', string.format('## %s %s%s — %s (%d ms)', SIGN[r.status] or '?', where, r.case, status_text(r), + math.floor(r.durationMs or 0)) } if r.reason and r.reason ~= '' then lines[#lines + 1] = r.reason end - if method == 'test/caseResult' and r.message and r.message ~= '' and r.status == 'failed' then - for _, l in ipairs(output_lines(r.message)) do + -- A failure: what failed, then both sides when it states them; else the message as written. + local failure = (r.failures or {})[1] + local what = failure and failure.message or r.message or '' + if r.status ~= 'passed' and r.status ~= 'skipped' and what ~= '' then + for _, l in ipairs(output_lines(what)) do lines[#lines + 1] = '> ' .. l end end + if failure and failure.expected ~= nil then + lines[#lines + 1] = '> ' .. t('expected') .. ': ' .. failure.expected + lines[#lines + 1] = '> ' .. t('actual') .. ': ' .. (failure.actual or '') + end + local metrics = M.metrics_text(r) + if metrics ~= '' then + lines[#lines + 1] = metrics + end + local out = output_lines(r.output) + if #out > 0 then + lines[#lines + 1] = '```' + vim.list_extend(lines, out) + lines[#lines + 1] = '```' + end + write(lines) + if r.status == 'failed' or r.status == 'errored' then + local item = located(root, r.location) + if item then + item.text = string.format('%s › %s: %s', program, r.case, what ~= '' and what or status_text(r)) + if failure and failure.expected ~= nil then + item.text = item.text .. string.format(' (%s %s, %s %s)', t('expected'), failure.expected, t('actual'), failure.actual or '') + end + M.failures[#M.failures + 1] = item + end + end +end + +-- A program's result without cases (or a part's), as before cases existed. +local function add_program(r) + local name = r.part or r.program or r.test + local key = (r.root or '') .. '\0' .. (r.program or r.test) + last[key] = r.status + local lines = { '', string.format('## %s %s — %s (%d ms)', SIGN[r.status] or '?', name, status_text(r), r.durationMs or 0) } + if r.reason and r.reason ~= '' then + lines[#lines + 1] = r.reason + end local out = output_lines(r.output) if #out > 0 then lines[#lines + 1] = '```' @@ -114,8 +222,36 @@ function M.add(method, r) end end +-- One notification of a test job, said in the buffer, kept for the signs, and its failures put in +-- the quickfix list. +function M.add(method, r) + if method == 'test/caseEvent' then + if r.event == 'begin' then + last[(r.root or '') .. '\0' .. r.program .. '\0' .. r.case] = 'running' + else + add_case(r.root, r.program, r.part, r) + end + return + end + if method == 'test/caseResult' then + add_case(r.root, r.program or r.test, r.part, r) + return + end + local cases = r.cases or {} + if #cases == 0 then + add_program(r) + return + end + local program = r.program or r.test + last[(r.root or '') .. '\0' .. program] = r.status + for _, c in ipairs(cases) do + add_case(r.root, program, r.part, c) + end +end + -- The run ended: the quickfix list holds its failures, and the signs show the results. function M.finish() + flush() vim.fn.setqflist({}, ' ', { title = 'mcpp test', items = M.failures or {} }) for _, buf in ipairs(vim.api.nvim_list_bufs()) do if vim.api.nvim_buf_is_loaded(buf) then @@ -141,7 +277,8 @@ local function place(buf, line, status, text) }) end --- The signs of a buffer: ▶ on a program's or a test's `main` and on each case, or the last result. +-- The signs of a buffer: ▶ on a program's or a test's `main` and on each case its file declares, or +-- the last result, with what the case's line says. function M.signs(buf) local name = vim.api.nvim_buf_get_name(buf) if name == '' or vim.bo[buf].buftype ~= '' or not session.state.ready then @@ -163,9 +300,12 @@ function M.signs(buf) if cerr or not cases or not vim.api.nvim_buf_is_valid(buf) then return end + local here = vim.fs.normalize(name) for _, c in ipairs(cases.cases or {}) do - if c.line then - place(buf, c.line, last[key .. '\0' .. c.name] or 'entry') + local own = not c.file or vim.fs.normalize(c.file) == here + if c.line and own then + local ckey = key .. '\0' .. c.name + place(buf, c.line, last[ckey] or 'entry', said[ckey]) end end end) @@ -178,5 +318,132 @@ function M.sign_of(root, test, case) return SIGN[status or 'entry'] end +-- What a case's line says after a run (the tests). +function M.said_of(root, test, case) + return said[(root or '') .. '\0' .. test .. '\0' .. case] +end + +-- ── the insights as text (`:Mcpp testInsights`) ────────────────────────────────────────────── + +local BLOCKS = { '▁', '▂', '▃', '▄', '▅', '▆', '▇', '█' } + +-- A series as block characters, lowest to highest. +function M.sparkline(values) + if #values == 0 then + return '' + end + local lo, hi = math.huge, -math.huge + for _, v in ipairs(values) do + lo, hi = math.min(lo, v), math.max(hi, v) + end + local out = {} + for _, v in ipairs(values) do + local i = hi > lo and math.floor((v - lo) / (hi - lo) * 7 + 0.5) + 1 or 4 + out[#out + 1] = BLOCKS[i] + end + return table.concat(out) +end + +-- A worker's processes as a bar of 40 cells over the run (█ passed, ▓ failed). +function M.lane_bar(lane, finish) + local bar = vim.split(string.rep(' ', 40), '') + for _, item in ipairs(lane.items or {}) do + local from = math.floor((item.startMs or 0) / finish * 40) + 1 + local to = math.min(40, math.max(from, math.floor(((item.startMs or 0) + (item.ms or 0)) / finish * 40))) + local cell = item.status == 'fail' and '▓' or '█' + for i = from, to do + bar[i] = cell + end + end + return table.concat(bar) +end + +-- A run's time as a person reads it: `2026-11-08 10:00` from an ISO time or a run's identifier. +function M.when(text) + local y, mo, d, h, mi = (text or ''):match('^(%d%d%d%d)-(%d%d)-(%d%d)T(%d%d):(%d%d)') + if not y then + y, mo, d, h, mi = (text or ''):match('^(%d%d%d%d)(%d%d)(%d%d)T(%d%d)(%d%d)') + end + return y and string.format('%s-%s-%s %s:%s', y, mo, d, h, mi) or (text or '') +end + +local function duration(ms) + if ms < 1000 then + return string.format('%d ms', math.floor(ms + 0.5)) + end + return string.format('%.1f s', ms / 1000) +end + +local function who(r) + local part = r.part and r.part ~= r.program and (' (' .. r.part .. ')') or '' + return (r.case ~= '' and r.case or r.program) .. ' ' .. (r.case ~= '' and r.program or '') .. part +end + +-- test/insights as lines: the run, its failures, the metrics, the slowest, the flaky, the schedule. +function M.insights_text(x) + if not x.run or x.run == vim.NIL then + return { t('No run is kept yet: mcpp test keeps each run in {0} (mcpp 2026.11.9.1 and later).', x.store or '') } + end + local c = x.run.counts or {} + local lines = { + t('{0} runs kept · the last began {1}', x.runs or 0, M.when(x.run.started ~= '' and x.run.started or x.run.id)), + string.format('✓ %d ✗ %d ○ %d %s', (c.pass or 0) + (c.xfail or 0), (c.fail or 0) + (c.xpass or 0) + (c.invalid or 0), + c.skip or 0, duration(x.run.ms or 0)), + } + local function section(title, rows) + if #rows > 0 then + lines[#lines + 1] = '' + lines[#lines + 1] = title + vim.list_extend(lines, rows) + end + end + local rows = {} + for _, f in ipairs(x.failures or {}) do + rows[#rows + 1] = string.format(' ✗ %s %s', who(f), f.new and t('new') or t('since {0}', M.when(f.since))) + if f.expected ~= nil then + rows[#rows + 1] = string.format(' %s %s · %s %s', t('expected'), f.expected, t('actual'), f.actual or '') + end + end + section(t('Failures'), rows) + rows = {} + for _, m in ipairs(x.metrics or {}) do + local values = {} + for _, v in ipairs(m.values or {}) do + values[#values + 1] = v.value + end + local bound = (m.bound and m.bound ~= vim.NIL) and m.bound or {} + rows[#rows + 1] = string.format(' %s %s %s', M.sparkline(values), + M.metrics_text({ metrics = { { name = m.name, value = m.last, unit = m.unit } }, + bounds = { { metric = m.name, max = bound.max, min = bound.min } } }), who(m)) + end + section(t('Metrics'), rows) + rows = {} + for _, s in ipairs(x.slowest or {}) do + rows[#rows + 1] = string.format(' %8s %s (%s)', duration(s.ms), who(s), t('median {0}', duration(s.medianMs))) + end + section(t('Slowest'), rows) + rows = {} + for _, f in ipairs(x.flaky or {}) do + rows[#rows + 1] = string.format(' ⟳ %s %s', who(f), t('{0} passed · {1} failed · {2} changes', f.passed, f.failed, f.flips)) + end + section(t('Flaky'), rows) + rows = {} + local schedule = (x.schedule and x.schedule ~= vim.NIL) and x.schedule or { lanes = {} } + local finish = schedule.ms or 1 + for _, lane in ipairs(schedule.lanes or {}) do + for _, item in ipairs(lane.items or {}) do + finish = math.max(finish, (item.startMs or 0) + (item.ms or 0)) + end + end + for _, lane in ipairs(schedule.lanes or {}) do + rows[#rows + 1] = string.format(' %-10s %s', t('worker {0}', lane.worker), M.lane_bar(lane, finish)) + end + if #(schedule.criticalPath or {}) > 0 then + rows[#rows + 1] = ' ' .. t('Critical path') .. ': ' .. table.concat(schedule.criticalPath, ' → ') + end + section(t('Schedule'), rows) + return lines +end + M.namespace = NS return M diff --git a/mide-plugins/nvim/tests/smoke.lua b/mide-plugins/nvim/tests/smoke.lua index 919316b..71ce702 100644 --- a/mide-plugins/nvim/tests/smoke.lua +++ b/mide-plugins/nvim/tests/smoke.lua @@ -113,7 +113,14 @@ check('another series is refused, naming both versions and what installs this on older ~= nil and older:find('0.2.1', 1, true) ~= nil and older:find('series 0.2', 1, true) ~= nil and older:find('mcpp-ide@1.2.0', 1, true) ~= nil, older) check('another protocol is refused', session.incompatibility({ protocol = 1, series = '1.2' }, '1.2.0') ~= nil) -check('the plugin is of the backend series this repository builds', session.series_of(session.version) == '1.2') +-- The backend's version is the package's (mcpp.toml), always its series' X.Y.0. +local built = vim.fn.readfile(repo .. '/mcpp.toml') +local package_version +for _, line in ipairs(built) do + package_version = package_version or line:match('^version%s*=%s*"([^"]+)"') +end +check('the plugin is of the backend series this repository builds', + package_version ~= nil and session.series_of(session.version) == session.series_of(package_version), package_version) local diagnostics = require('mcpp.diagnostics') local views = require('mcpp.views') @@ -410,6 +417,39 @@ if mock then done = finished('pack', before) check(':Mcpp pack runs mcpp pack as a job', done ~= nil and done.kind == 'pack', done and done.status) + -- A program of several files as mcpp#917 states it (design 2026-11-10): its cases arrive as the + -- program writes them, a failure says both sides, a case's line its metrics against their bounds; + -- and the runs mcpp kept, as text. + write('.mock-mcpp-tests', 'directory-917\n') + write('.mock-mcpp-features', 'test-protocol,test-parts,test-case-stream,test-metrics\n') + write('tests/unit/a.cpp', 'import mcpplibs.mtest;\nusing namespace mcpplibs::mtest;\n\nBench alpha{"alpha", at_most{"ready", 15}, [] {}};\n') + write('tests/unit/b.cpp', 'import mcpplibs.mtest;\nusing namespace mcpplibs::mtest;\n\nUTest beta{"beta", [] {}};\n') + before = session.state.last + vim.cmd('Mcpp test') + done = finished('test', before) + local streamed = table.concat(vim.api.nvim_buf_get_lines(vim.fn.bufnr('mcpp://tests'), 0, -1, false), '\n') + check('a part\'s failing case is said with both sides', streamed:find('unit/b › beta', 1, true) ~= nil + and streamed:find('expected: 4', 1, true) ~= nil, streamed:sub(1, 400)) + check('a case\'s line says its metrics against their bounds', (tests_mod.said_of(root, 'unit', 'alpha') or ''):find('ready 4.2 s ≤ 15 s', 1, true) ~= nil, + tests_mod.said_of(root, 'unit', 'alpha')) + local qf_both = vim.fn.getqflist()[1] or {} + check('its quickfix item says both sides, once', (qf_both.text or '') == 'unit › beta: sum == 4 (expected 4, actual 5)', qf_both.text) + local store = repo .. '/protocol/fixtures/mtest-store' + for name, kind in vim.fs.dir(store, { depth = 3 }) do + if kind == 'file' then + local f = assert(io.open(store .. '/' .. name, 'rb')) + write('target/mtest/' .. name, f:read('*a')) + f:close() + end + end + vim.cmd('Mcpp test.insights') + local insight = float_text() + check(':Mcpp test.insights says what the kept runs say', insight:find('Critical path', 1, true) ~= nil + and insight:find('beta', 1, true) ~= nil and insight:find('ready', 1, true) ~= nil, insight:sub(1, 300)) + check('a metric\'s trend is drawn in blocks', tests_mod.sparkline({ 1, 2, 3 }) == '▁▅█', tests_mod.sparkline({ 1, 2, 3 })) + os.remove(root .. '/.mock-mcpp-tests') + os.remove(root .. '/.mock-mcpp-features') + -- The backend ends during a build: the build ends as failed, and the next command starts another. local function count(text) local n = 0 diff --git a/mide-plugins/vscode/data/i18n/zh-cn.json b/mide-plugins/vscode/data/i18n/zh-cn.json index 413475b..31acddf 100644 --- a/mide-plugins/vscode/data/i18n/zh-cn.json +++ b/mide-plugins/vscode/data/i18n/zh-cn.json @@ -665,5 +665,23 @@ "(the output was longer; its head and tail are shown)": "（输出更长，只显示开头和结尾）", "The test timed out.": "测试超时。", "The test did not compile.": "测试没有编译通过。", - "the program could not be built": "程序未能构建" + "the program could not be built": "程序未能构建", + "The case passed, though it is expected to fail.": "该用例本应失败，却通过了。", + "Test Insights": "测试洞察", + "No run is kept yet: mcpp test keeps each run in {0} (mcpp 2026.11.9.1 and later).": "尚无保留的运行：mcpp test 会把每次运行保存在 {0}（mcpp 2026.11.9.1 及以后）。", + "{0} runs kept · the last began {1}": "保留了 {0} 次运行 · 最近一次开始于 {1}", + "Failures": "失败", + "new": "新增", + "since {0}": "自 {0} 起", + "expected": "期望", + "actual": "实际", + "Metrics": "指标", + "Slowest": "最慢", + "median {0}": "中位数 {0}", + "Flaky": "不稳定", + "{0} passed · {1} failed · {2} changes": "通过 {0} · 失败 {1} · 变化 {2} 次", + "Schedule": "调度", + "worker {0}": "工作线程 {0}", + "Critical path": "关键路径", + "Metrics appear when the project's mcpp states them (test-metrics, mcpp#917).": "项目的 mcpp 声明指标后才会显示（test-metrics，mcpp#917）。" } diff --git a/mide-plugins/vscode/package-lock.json b/mide-plugins/vscode/package-lock.json index 966eaee..1d3c683 100644 --- a/mide-plugins/vscode/package-lock.json +++ b/mide-plugins/vscode/package-lock.json @@ -1,12 +1,12 @@ { "name": "mcpp-vscode", - "version": "1.2.0", + "version": "1.3.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "mcpp-vscode", - "version": "1.2.0", + "version": "1.3.0", "license": "Apache-2.0", "devDependencies": { "@mermaid-js/tiny": "12.2.0", diff --git a/mide-plugins/vscode/package.json b/mide-plugins/vscode/package.json index 75f1eb4..c4c9524 100644 --- a/mide-plugins/vscode/package.json +++ b/mide-plugins/vscode/package.json @@ -2,7 +2,7 @@ "name": "mcpp-vscode", "displayName": "%displayName%", "description": "%description%", - "version": "1.2.0", + "version": "1.3.0", "publisher": "mcpp-community", "license": "Apache-2.0", "icon": "media/logo.png", @@ -349,6 +349,12 @@ "title": "%command.mcpp.project.showModuleDependencyGraph.title%", "category": "%category%" }, + { + "command": "mcpp.test.insights", + "title": "%command.mcpp.test.insights.title%", + "category": "%category%", + "icon": "$(graph-line)" + }, { "command": "mcpp.languageServer.refreshState", "title": "%command.mcpp.languageServer.refreshState.title%", @@ -1478,7 +1484,7 @@ "test:parity": "npm run compile && node --test \"dist/test/parity/*.test.js\"", "bundle:webui": "node tools/bundle-webui.mjs", "check:webui": "tsc -p ../../webui/tsconfig.json", - "test:webui": "node tools/bundle-webui.mjs --out .webui-test && node --test ../../webui/tests/graph.test.mjs" + "test:webui": "node tools/bundle-webui.mjs --out .webui-test && node --test ../../webui/tests/graph.test.mjs ../../webui/tests/tests.test.mjs" }, "devDependencies": { "@mermaid-js/tiny": "12.2.0", diff --git a/mide-plugins/vscode/package.nls.json b/mide-plugins/vscode/package.nls.json index 0e456e6..1465565 100644 --- a/mide-plugins/vscode/package.nls.json +++ b/mide-plugins/vscode/package.nls.json @@ -211,6 +211,7 @@ "viewsWelcome.library": "No mcpp package index was found on this machine.\n\nRun `mcpp index update` once, or set `mcpp.library.indexPath` to an index checkout.\n[Run mcpp index update](command:mcpp.library.updateIndex)", "command.mcpp.project.showDependencyGraph.title": "Show Project Dependency Graph", "command.mcpp.project.showModuleDependencyGraph.title": "Show C++ Modules Dependency Graph", + "command.mcpp.test.insights.title": "Show Test Insights", "command.mcpp.debug.title": "Debug", "command.mcpp.pack.title": "Pack", "command.mcpp.cancelJob.title": "Cancel the Running Job", diff --git a/mide-plugins/vscode/package.nls.zh-cn.json b/mide-plugins/vscode/package.nls.zh-cn.json index bfb007f..1e39c5c 100644 --- a/mide-plugins/vscode/package.nls.zh-cn.json +++ b/mide-plugins/vscode/package.nls.zh-cn.json @@ -211,6 +211,7 @@ "viewsWelcome.library": "本机没有找到 mcpp 包索引。\n\n先运行一次 `mcpp index update`，或用 `mcpp.library.indexPath` 指向一个索引检出。\n[运行 mcpp index update](command:mcpp.library.updateIndex)", "command.mcpp.project.showDependencyGraph.title": "显示项目依赖图", "command.mcpp.project.showModuleDependencyGraph.title": "显示 C++ Modules 依赖图", + "command.mcpp.test.insights.title": "显示测试洞察", "command.mcpp.debug.title": "调试", "command.mcpp.pack.title": "打包", "command.mcpp.cancelJob.title": "取消正在运行的任务", diff --git a/mide-plugins/vscode/src/commands/ids.ts b/mide-plugins/vscode/src/commands/ids.ts index 46d6b9b..8295c59 100644 --- a/mide-plugins/vscode/src/commands/ids.ts +++ b/mide-plugins/vscode/src/commands/ids.ts @@ -94,6 +94,7 @@ export const LIBRARY_COMMANDS = { /** Settings, diagnostics and the mcpp CLI entry points added in 0.5.0. */ export const TOOL_COMMANDS = { + testInsights: "mcpp.test.insights", dependencyGraph: "mcpp.project.showDependencyGraph", moduleDependencyGraph: "mcpp.project.showModuleDependencyGraph", openSettings: "mcpp.openSettings", diff --git a/mide-plugins/vscode/src/extension.ts b/mide-plugins/vscode/src/extension.ts index d4a455b..419bc85 100644 --- a/mide-plugins/vscode/src/extension.ts +++ b/mide-plugins/vscode/src/extension.ts @@ -39,6 +39,7 @@ import { Jobs } from "./ide/jobs"; import type { IdeProjects } from "./ide/projects"; import { Session } from "./ide/session"; import { Testing } from "./ide/testing"; +import { registerTestInsights } from "./ide/insights"; import { McppCodeLens } from "./ide/codeLens"; import { changedSettings, onDidChange as onConfigurationChanged, read } from "./config/access"; import { applyRenames, pendingRenames, renamePrompt } from "./config/migrate"; @@ -348,6 +349,7 @@ export async function activate(extensionContext: vscode.ExtensionContext): Promi jobs, new Debugging(session, ideProjects, ensureMcppForAction, jobs), testing, + registerTestInsights(extensionContext, session, ideProjects), new McppCodeLens(session, testing, (root) => projectMcpp.get(root)), jobs.onDidChangeRunning.event((n) => void vscode.commands.executeCommand("setContext", "mcpp.jobRunning", n > 0)), vscode.tasks.onDidStartTask(() => void updateJobRunning()), diff --git a/mide-plugins/vscode/src/ide/codeLens.ts b/mide-plugins/vscode/src/ide/codeLens.ts index 2867184..2c3d9c5 100644 --- a/mide-plugins/vscode/src/ide/codeLens.ts +++ b/mide-plugins/vscode/src/ide/codeLens.ts @@ -45,9 +45,10 @@ export class McppCodeLens implements vscode.CodeLensProvider, vscode.Disposable vscode.languages.registerCodeLensProvider([{ language: "cpp", scheme: "file" }], this), session.onDidStart.event(() => this.changed.fire()), vscode.workspace.onDidChangeConfiguration((e) => { if (e.affectsConfiguration("mcpp.codeLens")) this.changed.fire(); }), - vscode.commands.registerCommand(CODELENS_COMMANDS.runTest, async (root: string, test: string) => { - if (!this.testing.item(root, test)) await this.testing.discover(); - const item = this.testing.item(root, test); + // A test's program, or the part of it one file is (a program of several files, design 2026-11-10). + vscode.commands.registerCommand(CODELENS_COMMANDS.runTest, async (root: string, test: string, part?: string) => { + if (!this.testing.item(root, test, part)) await this.testing.discover(); + const item = this.testing.item(root, test, part); if (item) await this.testing.runItem(item); }), vscode.commands.registerCommand(CODELENS_COMMANDS.debugTest, async (root: string, test: string) => { @@ -58,7 +59,7 @@ export class McppCodeLens implements vscode.CodeLensProvider, vscode.Disposable vscode.commands.registerCommand(CODELENS_COMMANDS.runThisFile, async () => { const entry = await this.activeEntry(); if (entry?.kind === "program") await vscode.commands.executeCommand("mcpp.run"); - else if (entry?.kind === "test" && entry.root) await vscode.commands.executeCommand(CODELENS_COMMANDS.runTest, entry.root, entry.test ?? ""); + else if (entry?.kind === "test" && entry.root) await vscode.commands.executeCommand(CODELENS_COMMANDS.runTest, entry.root, entry.test ?? "", entry.part); }), vscode.commands.registerCommand(CODELENS_COMMANDS.debugThisFile, async () => { const entry = await this.activeEntry(); diff --git a/mide-plugins/vscode/src/ide/insights.ts b/mide-plugins/vscode/src/ide/insights.ts new file mode 100644 index 0000000..615b1de --- /dev/null +++ b/mide-plugins/vscode/src/ide/insights.ts @@ -0,0 +1,99 @@ +/** + * Test Insights (design 2026-11-10): what the last runs of a project's tests say -- the run, its + * failures and since when, each metric against its bound, the slowest and the flaky cases, the + * schedule with its critical path -- read by the backend from the store mcpp keeps (`test/insights`) + * and drawn by the page mcpp-clion shows in JCEF (webui/src/tests.ts), here in a webview. It refreshes + * when a test job ends; a click on a failure opens its place. + */ +import * as path from "node:path"; +import * as vscode from "vscode"; + +import { TOOL_COMMANDS } from "../commands/ids"; +import { t } from "../i18n/t"; +import type { IdeProjects } from "./projects"; +import type { JobFinished } from "./protocol"; +import type { Session } from "./session"; +import { WebviewDocument } from "../webview/document"; + +/** The page's strings, in the session's language. */ +export function insightsLabels(): Record { + return { + title: t("Test Insights"), + noRun: t("No run is kept yet: mcpp test keeps each run in {0} (mcpp 2026.11.9.1 and later)."), + window: t("{0} runs kept · the last began {1}"), + failures: t("Failures"), + fresh: t("new"), + since: t("since {0}"), + expected: t("expected"), + actual: t("actual"), + metrics: t("Metrics"), + slowest: t("Slowest"), + median: t("median {0}"), + flaky: t("Flaky"), + flakyCounts: t("{0} passed · {1} failed · {2} changes"), + schedule: t("Schedule"), + worker: t("worker {0}"), + criticalPath: t("Critical path"), + noMetrics: t("Metrics appear when the project's mcpp states them (test-metrics, mcpp#917)."), + }; +} + +export function registerTestInsights(context: vscode.ExtensionContext, session: Session, projects: IdeProjects): vscode.Disposable { + const scripts = vscode.Uri.joinPath(context.extensionUri, "dist", "webui"); + let panel: vscode.WebviewPanel | undefined; + let root: string | undefined; + // The page brings its own style (webui/src/tests.ts); the document holds the view's nonce. + const document = new WebviewDocument("tests.css"); + + const render = async (): Promise => { + if (!panel || !root) return; + let insights: unknown; + try { + insights = await session.request("test/insights", { root }); + } catch (error) { + void vscode.window.showErrorMessage(t("mcpp: {0}", String(error instanceof Error ? error.message : error))); + return; + } + await panel.webview.postMessage({ insights, labels: insightsLabels() }); + }; + + const open = async (argument?: { root?: string }): Promise => { + root = argument?.root ?? projects.current()?.root; + if (!root) { + void vscode.window.showInformationMessage(t("Open an mcpp project first.")); + return; + } + if (!panel) { + panel = vscode.window.createWebviewPanel("mcpp.testInsights", t("Test Insights"), vscode.ViewColumn.Active, + { enableScripts: true, retainContextWhenHidden: true, localResourceRoots: [scripts] }); + const script = panel.webview.asWebviewUri(vscode.Uri.joinPath(scripts, "tests.js")).toString(); + const { nonce } = document.assets(scripts, panel.webview); + document.paint(panel.webview, ` + +
+`); + panel.webview.onDidReceiveMessage(async (message: unknown) => { + if (message === "ready") return render(); + const at = typeof message === "string" ? (JSON.parse(message) as { open?: { path: string; line: number } }).open : undefined; + if (!at || !root) return; + const file = path.isAbsolute(at.path) ? at.path : path.join(root, at.path); + const editor = await vscode.window.showTextDocument(vscode.Uri.file(file), { viewColumn: vscode.ViewColumn.One }); + const position = new vscode.Position(at.line, 0); + editor.selection = new vscode.Selection(position, position); + editor.revealRange(new vscode.Range(position, position), vscode.TextEditorRevealType.InCenter); + }); + panel.onDidDispose(() => { panel = undefined; document.invalidate(); }); + } else { + panel.reveal(); + await render(); + } + }; + + return vscode.Disposable.from( + vscode.commands.registerCommand(TOOL_COMMANDS.testInsights, open), + // A run of the tests ended: the store holds another run. + session.on("job/finished", (p: JobFinished) => { if (p.kind === "test" && p.root === root) void render(); }), + { dispose: () => panel?.dispose() }, + ); +} diff --git a/mide-plugins/vscode/src/ide/jobs.ts b/mide-plugins/vscode/src/ide/jobs.ts index bd92ccc..78e095d 100644 --- a/mide-plugins/vscode/src/ide/jobs.ts +++ b/mide-plugins/vscode/src/ide/jobs.ts @@ -6,7 +6,7 @@ import * as vscode from "vscode"; import type { JobFinished } from "./protocol"; import type { Session } from "./session"; -const JOB_EVENTS = ["job/started", "job/output", "job/plan", "job/progress", "job/diagnostic", "job/artifact", "job/record", "job/finished", "test/result", "test/summary", "test/caseResult"]; +const JOB_EVENTS = ["job/started", "job/output", "job/plan", "job/progress", "job/diagnostic", "job/artifact", "job/record", "job/finished", "test/result", "test/summary", "test/caseResult", "test/caseEvent"]; export interface JobEvents { output?(text: string): void; @@ -17,6 +17,8 @@ export interface JobEvents { testSummary?(p: any): void; /** A case's outcome (test/runCases), sent before the job's end. */ caseResult?(p: any): void; + /** A case as it begins or ends while its program runs (test-case-stream, mcpp#917). */ + caseEvent?(p: any): void; finished?(p: JobFinished): void; } @@ -83,6 +85,7 @@ export class Jobs implements vscode.Disposable { case "test/result": e.testResult?.(params); break; case "test/summary": e.testSummary?.(params); break; case "test/caseResult": e.caseResult?.(params); break; + case "test/caseEvent": e.caseEvent?.(params); break; case "job/finished": e.finished?.(params); owner.handle.finish(params); diff --git a/mide-plugins/vscode/src/ide/mapping.ts b/mide-plugins/vscode/src/ide/mapping.ts index 054d0ec..a52700d 100644 --- a/mide-plugins/vscode/src/ide/mapping.ts +++ b/mide-plugins/vscode/src/ide/mapping.ts @@ -1,6 +1,6 @@ // The protocol's values in VS Code's terms, as pure functions (the unit tests run them without an // editor): positions, severities, test identities, the debuggers present, the "when" of a command. -import type { Diagnostic, Position } from "./protocol"; +import type { CaseOutcome, Diagnostic, MetricBound, Position } from "./protocol"; /** mcpp's positions are 1-based lines and columns; VS Code's are 0-based. */ export function zeroBased(p: Position | undefined): { line: number; character: number } { @@ -57,8 +57,23 @@ export function mainLineOf(symbols: readonly { name: string; kind: number; range return symbols?.find((s) => s.kind === FUNCTION_SYMBOL && (s.name === "main" || s.name.startsWith("main(")))?.range.start.line; } +/** What separates a file of a program of several files from its program's id. */ +const FILE = "\u001e"; + +/** A file of a program of several files (design 2026-11-10): its program's id, then the part it is. */ +export function fileId(root: string, member: string, test: string, part: string): string { + return `${testId(root, member, test)}${FILE}${part}`; +} + +export function parseFileId(id: string): { root: string; member: string; test: string; part: string } | undefined { + const at = id.indexOf(FILE); + if (at < 0 || id.includes(CASE)) return undefined; + const program = parseTestId(id.slice(0, at)); + return program === undefined ? undefined : { ...program, part: id.slice(at + 1) }; +} + export function parseTestId(id: string): { root: string; member: string; test: string } | undefined { - if (id.includes(CASE)) return undefined; + if (id.includes(CASE) || id.includes(FILE)) return undefined; const parts = id.split("::"); if (parts.length < 3) return undefined; const test = parts.pop()!; @@ -88,3 +103,31 @@ export function format(template: string, args: unknown[]): string { export function panelTitle(title: string): string { return `mcpp: ${title}`; } + +/** A metric's value with its unit, three significant digits: `4.2 s`, `0.913`, `1250 ms`. */ +export function measured(value: number, unit?: string): string { + const text = Math.abs(value) >= 100 ? value.toFixed(0) : Number(value.toPrecision(3)).toString(); + return unit && unit !== "1" ? `${text} ${unit}` : text; +} + +/** A metric against its bound: `ready 4.2 s ≤ 15 s`, `recall 0.93 ≥ 0.9`; without a bound, its value. */ +export function metricText(name: string, value: number, unit: string | undefined, bound: MetricBound | undefined): string { + if (bound?.max !== undefined) return `${name} ${measured(value, unit)} ≤ ${measured(bound.max, unit)}`; + if (bound?.min !== undefined) return `${name} ${measured(value, unit)} ≥ ${measured(bound.min, unit)}`; + return `${name} ${measured(value, unit)}`; +} + +/** What follows a case's line after a run: its verdict, its duration, its metrics against their bounds. */ +export function caseSummary(o: Pick): string { + const mark = o.status === "passed" ? "✓" : o.status === "skipped" ? "○" : "✗"; + const parts = [`${mark}${o.expectedFailure ? " xfail" : ""}`]; + if (o.durationMs !== undefined && o.status !== "skipped") parts.push(`${o.durationMs} ms`); + for (const m of o.metrics ?? []) + parts.push(metricText(m.name, m.value, m.unit, o.bounds?.find((b) => b.metric === m.name))); + return parts.join(" · "); +} + +/** The test tags of a case: `kind:` and each of its own (VS Code's `@tag` filter selects by them). */ +export function caseTags(kind: string | undefined, tags: readonly string[] | undefined): string[] { + return [...(kind ? [`kind:${kind}`] : []), ...(tags ?? [])]; +} diff --git a/mide-plugins/vscode/src/ide/protocol.ts b/mide-plugins/vscode/src/ide/protocol.ts index 26037e5..247a481 100644 --- a/mide-plugins/vscode/src/ide/protocol.ts +++ b/mide-plugins/vscode/src/ide/protocol.ts @@ -178,19 +178,98 @@ export interface SourceEntries { root?: string; kind: "program" | "test" | "none"; test?: string; + /** A file of a program of several files: the part of the program it is (`unit/a`); `test` is the program. */ + part?: string; framework?: string; main?: { line: number; approximate?: boolean }; } -/** `test/cases` (§7.3): a test program's cases, from its source and, once listed, from the program. */ +/** The bounds a case states on its metrics (mtest protocol P6.5). */ +export interface MetricBound { + metric: string; + max?: number; + min?: number; +} + +/** What a case measured (mtest protocol P6), when mcpp states it (test-metrics, mcpp#917). */ +export interface Metric { + name: string; + value: number; + unit?: string; + better?: "lower" | "higher"; +} + +/** A case of `test/cases`: its place, and what the test protocol states of it (design 2026-11-10). */ +export interface TestCase { + name: string; + line?: number; + source: "static" | "listed"; + /** The file that declares it, in a program of several files. */ + file?: string; + kind?: string; + tags?: string[]; + expect?: string; + bounds?: MetricBound[]; +} + +/** `test/cases` (§7.3): a test program's cases, from its sources and, once listed, from the program. */ export interface TestCases { root: string; test: string; file: string; + /** The files of a program of several files. */ + files?: string[]; framework: string | null; listed?: boolean; main?: { line: number; approximate?: boolean }; - cases: { name: string; line?: number; source: "static" | "listed" }[]; + cases: TestCase[]; +} + +/** A program of `test/list`: its files and the parts a run divides it into (design 2026-11-10). */ +export interface TestProgram { + test: string; + member: string; + main: string; + files?: string[]; + parts?: { name: string; file: string }[]; + layout?: "file" | "directory"; +} + +/** A failure of a case: its place (0-based line) and, when stated, both sides. */ +export interface CaseFailure { + path: string; + line: number; + message: string; + expected?: string; + actual?: string; +} + +/** A case as it ended, as the backend maps the test protocol's statuses. */ +export interface CaseOutcome { + case: string; + status: "passed" | "failed" | "skipped" | "errored"; + durationMs?: number; + reason?: string; + message?: string; + location?: FailureLocation | null; + failures?: CaseFailure[]; + metrics?: Metric[]; + bounds?: MetricBound[]; + kind?: string; + tags?: string[]; + expectedFailure?: boolean; + unexpectedPass?: boolean; +} + +/** `test/caseEvent`: a case began or ended while its program runs (test-case-stream, mcpp#917). */ +export interface CaseEvent extends Partial { + jobId: string; + root: string; + program: string; + part: string; + case: string; + event: "begin" | "end"; + tMs?: number; } /** `test/caseResult`: one case's outcome, read from the framework's report. */ @@ -200,12 +279,14 @@ export interface FailureLocation { line: number; } -export interface CaseResult { +export interface CaseResult extends CaseOutcome { jobId: string; root: string; test: string; + program?: string; + part?: string; case: string; - status: "passed" | "failed" | "skipped"; + status: "passed" | "failed" | "skipped" | "errored"; durationMs: number; message: string; /** The case's own output (GoogleTest's console between `[ RUN ]` and its result); empty without boundaries. */ @@ -230,4 +311,9 @@ export interface TestResult { outputTruncated?: boolean; location?: FailureLocation | null; reason: string; + /** The program the record belongs to, and the part of it the record is (design 2026-11-10). */ + program?: string; + part?: string; + /** A protocol program's cases, each as it ended. */ + cases?: CaseOutcome[]; } diff --git a/mide-plugins/vscode/src/ide/testing.ts b/mide-plugins/vscode/src/ide/testing.ts index f298cc4..c45c534 100644 --- a/mide-plugins/vscode/src/ide/testing.ts +++ b/mide-plugins/vscode/src/ide/testing.ts @@ -1,23 +1,27 @@ /** - * mcpp's tests in VS Code's Testing view (design 2026-10-30 part 2 §7). The tests of each project - * come from `test/list` (mcpp test --list: no build), each a program located at its `main`; a - * program of a test framework has its cases as children, located at their macros (`test/cases`, - * read from the source), and after a build the program lists them (only where its source calls its - * framework's macros, and only in a trusted workspace), which adds the cases macros generate. + * mcpp's tests in VS Code's Testing view (design 2026-10-30 part 2 §7; design 2026-11-10). The tests of + * each project come from `test/list` (mcpp test --list: no build), each a program located at its + * `main`; a program of several files (`[test] programs = "directory"`) has its files as children, + * each the part of the program a run gives a process. A program of a test framework has its cases as + * children of the file that declares them (`test/cases`, read from the sources), and after a build the + * program lists them (only in a trusted workspace). A case carries the test tags `kind:` and its + * own tags, so `@kind:bench` filters the view. * - * A run of whole programs is one test job, filtered to the program asked for; a run of cases is - * `test/runCases`, whose program receives the framework's filter, and each `test/caseResult` marks - * its case. + * A run of whole programs, or of one file's part, is one test job; a run of cases is `test/runCases`. + * Results arrive per case as the program writes them when the project's mcpp states them + * (`test/caseEvent`, test-case-stream), else with each part's record (`test/result` and its cases). A + * failure that states both sides is a diff; after a run each case's line ends with its verdict, its + * duration and its metrics against their bounds. */ import * as path from "node:path"; import * as vscode from "vscode"; import { t } from "../i18n/t"; import type { Jobs } from "./jobs"; -import { baseCaseName, caseId, parseCaseId, parseTestId, testId } from "./mapping"; +import { baseCaseName, caseId, caseSummary, caseTags, fileId, parseCaseId, parseFileId, parseTestId, testId } from "./mapping"; import type { IdeProjects } from "./projects"; import type { Session } from "./session"; -import type { CaseResult, FailureLocation, JobFinished, TestCases, TestResult } from "./protocol"; +import type { CaseEvent, CaseOutcome, CaseResult, FailureLocation, JobFinished, TestCases, TestProgram, TestResult } from "./protocol"; /** Text for the test terminal, whose lines end in `\r\n`. */ function terminal(text: string): string { @@ -44,16 +48,71 @@ function attribute(run: vscode.TestRun, item: vscode.TestItem, output: string | if (truncated) run.appendOutput(terminal(`${t("(the output was longer; its head and tail are shown)")}\n`), at, item); } +/** A failed case's message: a diff when the failure states both sides (VS Code compares them). */ +function failureMessage(root: string, o: CaseOutcome, item: vscode.TestItem): vscode.TestMessage { + const first = o.failures?.[0]; + const text = o.unexpectedPass ? t("The case passed, though it is expected to fail.") : o.message || o.reason || o.status; + const message = first?.expected !== undefined && first.actual !== undefined + ? vscode.TestMessage.diff(first.message || text, first.expected, first.actual) + : new vscode.TestMessage(text); + message.location = failureAt(root, o.location, item); + return message; +} + interface Program { root: string; member: string; test: string; framework: string | null; + /** The cases by name, and the files' items by part. */ + cases: Map; + parts: Map; +} + +/** After a run, each case's line ends with what the run said of it. */ +class CaseDecorations implements vscode.Disposable { + private readonly type = vscode.window.createTextEditorDecorationType({ + after: { color: new vscode.ThemeColor("descriptionForeground"), margin: "0 0 0 2em" }, + }); + private readonly byFile = new Map>(); + private readonly disposables: vscode.Disposable[] = [ + this.type, + vscode.window.onDidChangeVisibleTextEditors(() => this.apply()), + vscode.workspace.onDidChangeTextDocument((e) => { if (this.byFile.delete(e.document.uri.fsPath)) this.apply(); }), + ]; + + set(item: vscode.TestItem, text: string): void { + if (!item.uri || !item.range) return; + const lines = this.byFile.get(item.uri.fsPath) ?? new Map(); + lines.set(item.range.start.line, text); + this.byFile.set(item.uri.fsPath, lines); + this.apply(); + } + + clear(files: Iterable): void { + for (const f of files) this.byFile.delete(f); + this.apply(); + } + + private apply(): void { + for (const editor of vscode.window.visibleTextEditors) { + const lines = this.byFile.get(editor.document.uri.fsPath); + editor.setDecorations(this.type, [...(lines ?? [])].map(([line, text]) => ({ + range: new vscode.Range(line, Number.MAX_SAFE_INTEGER, line, Number.MAX_SAFE_INTEGER), + renderOptions: { after: { contentText: text } }, + }))); + } + } + + dispose(): void { + this.disposables.forEach((d) => d.dispose()); + } } export class Testing implements vscode.Disposable { private readonly controller = vscode.tests.createTestController("mcpp", "mcpp"); - private readonly disposables: vscode.Disposable[] = [this.controller]; + private readonly decorations = new CaseDecorations(); + private readonly disposables: vscode.Disposable[] = [this.controller, this.decorations]; private readonly programs = new Map(); constructor(private readonly session: Session, private readonly jobs: Jobs, private readonly projects: IdeProjects) { @@ -74,18 +133,19 @@ export class Testing implements vscode.Disposable { ); } - /** The program a test's `main` is, by its project and name (the CodeLens runs it). */ - item(root: string, test: string): vscode.TestItem | undefined { - const group = this.controller.items.get(root); - let found: vscode.TestItem | undefined; - group?.children.forEach((child) => { - const id = parseTestId(child.id); - if (id && id.test === test) found = child; - }); - return found; + /** The program a test's `main` is, or the item of one file's part of it (the CodeLens and the Run menu run it). */ + item(root: string, test: string, part?: string): vscode.TestItem | undefined { + const program = this.programs.get(this.findProgramId(root, test) ?? ""); + if (part && program) return program.parts.get(part); + return this.controller.items.get(root)?.children.get(this.findProgramId(root, test) ?? ""); } - /** Runs one program or case, as the Testing view's ▶ does. */ + private findProgramId(root: string, test: string): string | undefined { + for (const [id, p] of this.programs) if (p.root === root && p.test === test) return id; + return undefined; + } + + /** Runs one program, file or case, as the Testing view's ▶ does. */ async runItem(item: vscode.TestItem): Promise { const source = new vscode.CancellationTokenSource(); try { @@ -97,9 +157,9 @@ export class Testing implements vscode.Disposable { async discover(list = false): Promise { for (const project of this.projects.all()) { - let tests: { test: string; member: string; main: string }[] = []; + let tests: TestProgram[] = []; try { - tests = (await this.session.request<{ tests: { test: string; member: string; main: string }[] }>("test/list", { root: project.root })).tests; + tests = (await this.session.request<{ tests: TestProgram[] }>("test/list", { root: project.root })).tests; } catch { continue; } @@ -109,41 +169,78 @@ export class Testing implements vscode.Disposable { this.controller.items.add(group); } const items: vscode.TestItem[] = []; - for (const test of tests) { - const id = testId(project.root, test.member, test.test); - const item = this.controller.createTestItem(id, test.member ? `${test.member} · ${test.test}` : test.test, - test.main ? vscode.Uri.file(test.main) : undefined); - const program: Program = { root: project.root, member: test.member, test: test.test, framework: null }; - let cases: TestCases | undefined; - try { - cases = await this.session.request("test/cases", { root: project.root, test: test.test, list }); - } catch { - cases = undefined; - } - // The program at its `main` (F11); a framework's program at its first case when it has none. - const mainLine = cases?.main?.line ?? cases?.cases.find((c) => c.line !== undefined)?.line ?? 0; - if (test.main) item.range = new vscode.Range(mainLine, 0, mainLine, 0); - if (cases?.framework) { - program.framework = cases.framework; - item.description = cases.framework; - item.children.replace(cases.cases.map((c) => { - const child = this.controller.createTestItem(caseId(project.root, test.member, test.test, c.name), c.name, - test.main ? vscode.Uri.file(test.main) : undefined); - if (test.main && c.line !== undefined) child.range = new vscode.Range(c.line, 0, c.line, 0); - return child; - })); - } - this.programs.set(id, program); - items.push(item); - } + for (const test of tests) items.push(await this.programItem(project.root, test, list)); group.children.replace(items); } } + private async programItem(root: string, test: TestProgram, list: boolean): Promise { + const id = testId(root, test.member, test.test); + const home = test.main || test.files?.[0] || ""; + const item = this.controller.createTestItem(id, test.member ? `${test.member} · ${test.test}` : test.test, home ? vscode.Uri.file(home) : undefined); + const program: Program = { root, member: test.member, test: test.test, framework: null, cases: new Map(), parts: new Map() }; + let cases: TestCases | undefined; + try { + cases = await this.session.request("test/cases", { root, test: test.test, list }); + } catch { + cases = undefined; + } + // The program at its `main` (F11); a framework's program at its first case when it has none. + const mainLine = cases?.main?.line ?? cases?.cases.find((c) => c.line !== undefined)?.line ?? 0; + if (home) item.range = new vscode.Range(mainLine, 0, mainLine, 0); + if (cases?.framework) { + program.framework = cases.framework; + item.description = cases.framework; + } + const caseItem = (c: TestCases["cases"][number]): vscode.TestItem => { + const file = c.file || home; + const child = this.controller.createTestItem(caseId(root, test.member, test.test, c.name), c.name, file ? vscode.Uri.file(file) : undefined); + if (file && c.line !== undefined) child.range = new vscode.Range(c.line, 0, c.line, 0); + child.tags = caseTags(c.kind, c.tags).map((tag) => new vscode.TestTag(tag)); + if (c.kind && c.kind !== "unit") child.description = c.kind; + program.cases.set(c.name, child); + return child; + }; + const parts = test.layout === "directory" ? test.parts ?? [] : []; + if (parts.length > 0) { + // A program of several files: each file is the part a run gives a process, its cases beneath it. + item.children.replace(parts.map((part) => { + const child = this.controller.createTestItem(fileId(root, test.member, test.test, part.name), path.basename(part.file), vscode.Uri.file(part.file)); + const own = (cases?.cases ?? []).filter((c) => c.file === part.file); + child.range = new vscode.Range(own.find((c) => c.line !== undefined)?.line ?? 0, 0, own.find((c) => c.line !== undefined)?.line ?? 0, 0); + child.children.replace(cases?.framework ? own.map(caseItem) : []); + program.parts.set(part.name, child); + return child; + })); + } else if (cases?.framework) { + item.children.replace(cases.cases.map(caseItem)); + } + this.programs.set(id, program); + return item; + } + + /** A case's outcome, marked on its item; its line then ends with what the run said. */ + private markCase(run: vscode.TestRun, program: Program, o: CaseOutcome, seen: Set, output?: string, truncated?: boolean): void { + const item = program.cases.get(o.case) ?? program.cases.get(baseCaseName(program.framework, o.case)); + if (!item) return; + seen.add(item); + const at = o.status === "failed" ? failureAt(program.root, o.location, item) : undefined; + // Every case's own output reaches its item, passed or not (D16). + attribute(run, item, output, truncated, at); + const ms = o.durationMs; + if (o.status === "passed") run.passed(item, ms); + else if (o.status === "skipped") { + if (o.reason) run.appendOutput(terminal(`${o.reason}\n`), undefined, item); + run.skipped(item); + } else if (o.status === "errored") run.errored(item, failureMessage(program.root, o, item), ms); + else run.failed(item, failureMessage(program.root, o, item), ms); + this.decorations.set(item, caseSummary(o)); + } + private async run(request: vscode.TestRunRequest, token: vscode.CancellationToken): Promise { const run = this.controller.createTestRun(request); - // One job per program whose cases were asked for; one job per project for whole programs. - const wholePrograms = new Map(); + // One job per program whose cases were asked for; one job per project for whole programs and parts. + const whole = new Map(); const casesOf = new Map(); const collect = (item: vscode.TestItem) => { if (request.exclude?.includes(item)) return; @@ -153,91 +250,114 @@ export class Testing implements vscode.Disposable { casesOf.set(owner, [...(casesOf.get(owner) ?? []), item]); return; } + const f = parseFileId(item.id); + if (f) { + whole.set(f.root, [...(whole.get(f.root) ?? []), { item, filter: f.part }]); + return; + } const id = parseTestId(item.id); if (!id) { item.children.forEach(collect); return; } - wholePrograms.set(id.root, [...(wholePrograms.get(id.root) ?? []), item]); + whole.set(id.root, [...(whole.get(id.root) ?? []), { item, filter: id.test }]); }; if (request.include) request.include.forEach(collect); else this.controller.items.forEach(collect); for (const [owner, items] of casesOf) { if (token.isCancellationRequested) break; - const program = this.programs.get(owner); - if (!program?.framework) continue; - items.forEach((i) => run.enqueued(i)); - // Every case of the program asked for: the filter is left out and the program runs them all. - let all = 0; - this.controller.items.get(program.root)?.children.get(owner)?.children.forEach(() => { all++; }); - const names = items.length === all ? [] : items.map((i) => parseCaseId(i.id)!.name); - const byName = new Map(items.map((i) => [parseCaseId(i.id)!.name, i])); - const seen = new Set(); - const handle = await this.jobs.start( - { root: program.root, test: program.test, framework: program.framework, cases: names, color: true }, - { - output: (text) => run.appendOutput(terminal(text)), - caseResult: (r: CaseResult) => { - const item = byName.get(r.case) ?? byName.get(baseCaseName(program.framework, r.case)); - if (!item) return; - seen.add(item); - // Every case's own output reaches its item, passed or not (D16). - const at = r.status === "failed" ? failureAt(program.root, r.location, item) : undefined; - attribute(run, item, r.output, r.outputTruncated, at); - if (r.status === "passed") run.passed(item, r.durationMs); - else if (r.status === "skipped") run.skipped(item); - else { - const message = new vscode.TestMessage(r.message || r.output || r.status); - message.location = at; - run.failed(item, message, r.durationMs); - } - }, - }, - "test/runCases", - ); - const cancel = token.onCancellationRequested(() => void handle.cancel()); - const done = await handle.done; - cancel.dispose(); - for (const i of items) - if (!seen.has(i)) { - if (done.status === "ok" || done.status === "failed") run.skipped(i); - else run.errored(i, new vscode.TestMessage(t("The test program did not run: {0}", done.status))); - } + await this.runCases(run, token, owner, items); } - - for (const [root, items] of wholePrograms) { + for (const [root, targets] of whole) { if (token.isCancellationRequested) break; - items.forEach((i) => run.enqueued(i)); - const index = new Map(items.map((i) => [parseTestId(i.id)!.test, i])); - const filter = items.length === 1 && request.include ? parseTestId(items[0].id)!.test : undefined; - const handle = await this.jobs.start({ kind: "test", root, filter, color: true }, { + await this.runWhole(run, token, root, targets, request.include !== undefined); + } + run.end(); + } + + private async runCases(run: vscode.TestRun, token: vscode.CancellationToken, owner: string, items: vscode.TestItem[]): Promise { + const program = this.programs.get(owner); + if (!program?.framework) return; + items.forEach((i) => run.enqueued(i)); + // Every case of the program asked for: the filter is left out and the program runs them all. + const names = items.length === program.cases.size ? [] : items.map((i) => parseCaseId(i.id)!.name); + const seen = new Set(); + const handle = await this.jobs.start( + { root: program.root, test: program.test, framework: program.framework, cases: names, color: true }, + { output: (text) => run.appendOutput(terminal(text)), - testResult: (r: TestResult) => { - const item = index.get(r.test); - if (!item) return; - const ms = r.durationMs; - const failing = r.status === "failed" || r.status === "timedOut" || r.status === "errored"; - const at = failing ? failureAt(root, r.location, item) : undefined; - // The program's output reaches its item whatever the result (D16). - attribute(run, item, r.output, r.outputTruncated, at); - if (r.status === "passed") run.passed(item, ms); - else if (r.status === "skipped") { - if (r.reason) run.appendOutput(terminal(`${r.reason}\n`), undefined, item); - run.skipped(item); - } else { - const headline = r.status === "timedOut" ? t("The test timed out.") - : r.compileFailed ? t("The test did not compile.") : undefined; - const message = new vscode.TestMessage([headline, r.output || r.reason || r.status].filter(Boolean).join("\n")); - message.location = at; - if (r.status === "errored") run.errored(item, message, ms); else run.failed(item, message, ms); - } + caseEvent: (e: CaseEvent) => this.onCaseEvent(run, program, e, seen), + caseResult: (r: CaseResult) => { + const item = program.cases.get(r.case) ?? program.cases.get(baseCaseName(program.framework, r.case)); + if (item && seen.has(item)) return; + this.markCase(run, program, r, seen, r.output, r.outputTruncated); }, - }); - const cancel = token.onCancellationRequested(() => void handle.cancel()); - const done = await handle.done; - cancel.dispose(); - if (done.status !== "ok" && done.status !== "failed") - for (const i of items) run.skipped(i); + }, + "test/runCases", + ); + const cancel = token.onCancellationRequested(() => void handle.cancel()); + const done = await handle.done; + cancel.dispose(); + for (const i of items) + if (!seen.has(i)) { + if (done.status === "ok" || done.status === "failed") run.skipped(i); + else run.errored(i, new vscode.TestMessage(t("The test program did not run: {0}", done.status))); + } + } + + /** A case begins (VS Code shows it running) or ends, as the program writes it (test-case-stream). */ + private onCaseEvent(run: vscode.TestRun, program: Program, e: CaseEvent, seen: Set): void { + if (e.event === "begin") { + const item = program.cases.get(e.case); + if (item) run.started(item); + } else if (e.status) { + this.markCase(run, program, e as CaseOutcome, seen); } - run.end(); + } + + private async runWhole(run: vscode.TestRun, token: vscode.CancellationToken, root: string, + targets: { item: vscode.TestItem; filter: string }[], included: boolean): Promise { + targets.forEach((target) => run.enqueued(target.item)); + const filter = targets.length === 1 && included ? targets[0].filter : undefined; + const programs = [...this.programs.values()].filter((p) => p.root === root); + const programOf = (name: string) => programs.find((p) => p.test === name); + this.decorations.clear(programs.flatMap((p) => [...p.cases.values()].map((c) => c.uri?.fsPath ?? ""))); + const seen = new Set(); + const handle = await this.jobs.start({ kind: "test", root, filter, color: true }, { + output: (text) => run.appendOutput(terminal(text)), + caseEvent: (e: CaseEvent) => { + const program = programOf(e.program); + if (program) this.onCaseEvent(run, program, e, seen); + }, + testResult: (r: TestResult) => { + const program = programOf(r.program ?? r.test); + const item = (r.part ? program?.parts.get(r.part) : undefined) ?? this.item(root, r.program ?? r.test); + for (const o of r.cases ?? []) { + const caseItem = program?.cases.get(o.case); + if (program && !(caseItem && seen.has(caseItem))) this.markCase(run, program, o, seen, o.status === "failed" ? o.message : undefined); + } + if (!item) return; + const ms = r.durationMs; + const failing = r.status === "failed" || r.status === "timedOut" || r.status === "errored"; + const at = failing ? failureAt(root, r.location, item) : undefined; + // The program's output reaches its item whatever the result (D16). + attribute(run, item, r.output, r.outputTruncated, at); + if (r.status === "passed") run.passed(item, ms); + else if (r.status === "skipped") { + if (r.reason) run.appendOutput(terminal(`${r.reason}\n`), undefined, item); + run.skipped(item); + } else { + const headline = r.status === "timedOut" ? t("The test timed out.") + : r.compileFailed ? t("The test did not compile.") : undefined; + const message = new vscode.TestMessage([headline, r.output || r.reason || r.status].filter(Boolean).join("\n")); + message.location = at; + if (r.status === "errored") run.errored(item, message, ms); else run.failed(item, message, ms); + } + }, + }); + const cancel = token.onCancellationRequested(() => void handle.cancel()); + const done = await handle.done; + cancel.dispose(); + if (done.status !== "ok" && done.status !== "failed") + for (const target of targets) run.skipped(target.item); } dispose(): void { diff --git a/mide-plugins/vscode/src/toml/parser.ts b/mide-plugins/vscode/src/toml/parser.ts index 25a249e..7fbd073 100644 --- a/mide-plugins/vscode/src/toml/parser.ts +++ b/mide-plugins/vscode/src/toml/parser.ts @@ -2,7 +2,7 @@ // its targets): the structure an edit in progress already has, with exact 0-based ranges, never an // exception for a half-typed `[`, string, array or inline table. The rules layer -- completion, // hover, the jumps, the diagnostics -- is the mcpp-ide backend's (design 2026-10-30 part 2 §8), whose -// scanner answers the same structure (its tests/rules goldens were written by this one). +// scanner answers the same structure (its tests/cts/rules goldens were written by this one). // // The subset read: table headers (`[[...]]` arrays of tables, quoted segments), bare, quoted and // dotted keys, strings (`"""` and `'''` multi-line ones too), integers, booleans, arrays and inline diff --git a/mide-plugins/vscode/test/artifacts.test.ts b/mide-plugins/vscode/test/artifacts.test.ts index be8543f..bdc6002 100644 --- a/mide-plugins/vscode/test/artifacts.test.ts +++ b/mide-plugins/vscode/test/artifacts.test.ts @@ -646,8 +646,10 @@ const workflows = path.join(root, "..", "..", ".github", "workflows"); test("a release publishes the VSIX CI built, with its checksum", () => { const workflow = readFileSync(path.join(workflows, "release.yml"), "utf8"); - // A series release (vX.Y.0) and each component's patch tag (D15); vscode-vX.Y.Z publishes the VSIX alone. - assert.match(workflow, /push:\s*\n\s+tags: \['v\*', 'backend-v\*', 'vscode-v\*', 'clion-v\*', 'nvim-v\*'\]/); + // A series release (vX.Y.0) and each editor plugin's patch tag (D15); vscode-vX.Y.Z publishes the VSIX alone. + // The backend has no patch tag: the packages carry it, so a change of it is a series release. + assert.match(workflow, /push:\s*\n\s+tags: \['v\*', 'vscode-v\*', 'clion-v\*', 'nvim-v\*'\]/); + assert.doesNotMatch(workflow, /backend-v\*/); assert.match(workflow, /release-plan\.sh "\$GITHUB_REF_NAME"/); assert.match(workflow, /contents: write/); // The release publishes what CI on the commit built and tested, not a second build. diff --git a/mide-plugins/vscode/test/e2e/suite/extension.e2e.ts b/mide-plugins/vscode/test/e2e/suite/extension.e2e.ts index 5b63e51..326de76 100644 --- a/mide-plugins/vscode/test/e2e/suite/extension.e2e.ts +++ b/mide-plugins/vscode/test/e2e/suite/extension.e2e.ts @@ -168,6 +168,10 @@ suite(`mcpp extension smoke (mcppls stub: ${STUB})`, () => { await vscode.commands.executeCommand("mcpp.openSettings"); }); + test("Test Insights opens for the project, with or without a kept run", async () => { + await vscode.commands.executeCommand("mcpp.test.insights"); + }); + test("build.mcpp opens as cpp, and C++ Modules is left the names it answers", async () => { const workspaceFolder = vscode.workspace.workspaceFolders?.[0]; assert.ok(workspaceFolder, "the fixture workspace should be open"); diff --git a/mide-plugins/vscode/test/ide/mapping.test.ts b/mide-plugins/vscode/test/ide/mapping.test.ts index 026426d..22fcb4a 100644 --- a/mide-plugins/vscode/test/ide/mapping.test.ts +++ b/mide-plugins/vscode/test/ide/mapping.test.ts @@ -2,7 +2,7 @@ import * as assert from "node:assert/strict"; import { test } from "node:test"; -import { availableDebuggers, baseCaseName, caseId, format, mainLineOf, parseCaseId, parseTestId, severityOf, testId, zeroBased } from "../../src/ide/mapping"; +import { availableDebuggers, baseCaseName, caseId, caseSummary, caseTags, fileId, format, mainLineOf, measured, metricText, parseCaseId, parseFileId, parseTestId, severityOf, testId, zeroBased } from "../../src/ide/mapping"; test("positions are 1-based in mcpp and 0-based in VS Code", () => { assert.deepEqual(zeroBased({ line: 1, column: 21 }), { line: 0, character: 20 }); @@ -47,3 +47,29 @@ test("main is the language service's top-level function of that name", () => { assert.equal(mainLineOf([{ name: "main", kind: 12, range: at(3) }]), undefined); assert.equal(mainLineOf(undefined), undefined); }); + +test("a file of a program of several files is a part of its program (design 2026-11-10)", () => { + const id = fileId("/w/p", "", "unit", "unit/a"); + assert.deepEqual(parseFileId(id), { root: "/w/p", member: "", test: "unit", part: "unit/a" }); + assert.equal(parseTestId(id), undefined); + assert.equal(parseCaseId(id), undefined); + assert.equal(parseFileId(testId("/w/p", "", "unit")), undefined); + assert.equal(parseFileId(caseId("/w/p", "", "unit", "alpha")), undefined); +}); + +test("a case's line after a run: its verdict, duration and metrics against their bounds", () => { + assert.equal(caseSummary({ status: "passed", durationMs: 3 }), "✓ · 3 ms"); + assert.equal(caseSummary({ status: "failed", durationMs: 12 }), "✗ · 12 ms"); + assert.equal(caseSummary({ status: "skipped", durationMs: 0 }), "○"); + assert.equal(caseSummary({ status: "passed", durationMs: 0, expectedFailure: true }), "✓ xfail · 0 ms"); + assert.equal(caseSummary({ status: "passed", durationMs: 4200, metrics: [{ name: "ready", value: 4.2, unit: "s" }], + bounds: [{ metric: "ready", max: 15 }] }), "✓ · 4200 ms · ready 4.2 s ≤ 15 s"); + assert.equal(metricText("recall", 0.9312, "1", { metric: "recall", min: 0.9 }), "recall 0.931 ≥ 0.9"); + assert.equal(metricText("peak", 1250.4, "ms", undefined), "peak 1250 ms"); + assert.equal(measured(0.000123, "s"), "0.000123 s"); +}); + +test("a case's test tags: its kind and its own", () => { + assert.deepEqual(caseTags("bench", ["slow"]), ["kind:bench", "slow"]); + assert.deepEqual(caseTags(undefined, undefined), []); +}); diff --git a/mide-plugins/vscode/tools/bundle-webui.mjs b/mide-plugins/vscode/tools/bundle-webui.mjs index 4120317..5ee629d 100644 --- a/mide-plugins/vscode/tools/bundle-webui.mjs +++ b/mide-plugins/vscode/tools/bundle-webui.mjs @@ -1,8 +1,8 @@ /** - * The graph page mcpp-clion shows in JCEF (webui/, mcpp design 2026-10-30 part 2 §9): webui/src/graph.ts - * bundled into graph.js, beside graph.html and mermaid, into webui/dist (or the directory named by - * --out), with files.txt listing them (what mcpp-clion extracts from its jar). Nothing it writes is - * kept in the repository. + * The pages mcpp-clion shows in JCEF (webui/, mcpp design 2026-10-30 part 2 §9): webui/src/graph.ts and + * webui/src/tests.ts bundled into graph.js and tests.js, beside graph.html, tests.html and mermaid, into + * webui/dist (or the directory named by --out), with files.txt listing them (what mcpp-clion extracts + * from its jar). Nothing it writes is kept in the repository. */ import { build } from "esbuild"; import fs from "node:fs"; @@ -16,17 +16,19 @@ const out = at > 0 ? path.resolve(process.argv[at + 1]) : path.join(webui, "dist fs.rmSync(out, { recursive: true, force: true }); fs.mkdirSync(path.join(out, "vendor"), { recursive: true }); -await build({ - entryPoints: [path.join(webui, "src", "graph.ts")], - bundle: true, - format: "iife", - platform: "browser", - target: "chrome120", - outfile: path.join(out, "graph.js"), - logLevel: "warning", - legalComments: "eof", -}); -fs.copyFileSync(path.join(webui, "graph.html"), path.join(out, "graph.html")); +for (const page of ["graph", "tests"]) { + await build({ + entryPoints: [path.join(webui, "src", `${page}.ts`)], + bundle: true, + format: "iife", + platform: "browser", + target: "chrome120", + outfile: path.join(out, `${page}.js`), + logLevel: "warning", + legalComments: "eof", + }); + fs.copyFileSync(path.join(webui, `${page}.html`), path.join(out, `${page}.html`)); +} const mermaid = path.join(extension, "node_modules", "@mermaid-js", "tiny"); fs.copyFileSync(path.join(mermaid, "dist", "mermaid.tiny.js"), path.join(out, "vendor", "mermaid.js")); fs.copyFileSync(path.join(mermaid, "LICENSE"), path.join(out, "vendor", "MERMAID-LICENSE")); diff --git a/mide-plugins/vscode/tools/copy-webview-assets.mjs b/mide-plugins/vscode/tools/copy-webview-assets.mjs index ae5cd84..7abca0d 100644 --- a/mide-plugins/vscode/tools/copy-webview-assets.mjs +++ b/mide-plugins/vscode/tools/copy-webview-assets.mjs @@ -1,4 +1,8 @@ import fs from 'node:fs'; +import { build } from 'esbuild'; fs.mkdirSync('dist/vendor',{recursive:true}); fs.copyFileSync('node_modules/@mermaid-js/tiny/dist/mermaid.tiny.js','dist/vendor/mermaid.js'); fs.copyFileSync('node_modules/@mermaid-js/tiny/LICENSE','dist/vendor/MERMAID-LICENSE'); +// The tests page (../../webui/src/tests.ts), the one mcpp-clion draws in JCEF, for the Test Insights webview. +await build({ entryPoints: ['../../webui/src/tests.ts'], bundle: true, format: 'iife', platform: 'browser', target: 'es2022', + outfile: 'dist/webui/tests.js', logLevel: 'warning', legalComments: 'eof' }); diff --git a/protocol/fixtures.json b/protocol/fixtures.json index 6b05bed..6556bf9 100644 --- a/protocol/fixtures.json +++ b/protocol/fixtures.json @@ -9,6 +9,13 @@ { "file": "launch.json", "schema": "launchEnvelope" }, { "file": "test.ndjson", "each": "testStreamLine" }, { "file": "test-list.ndjson", "each": "testListLine" }, + { "file": "test-protocol.ndjson", "each": "testStreamLine" }, + { "file": "test-cases.ndjson", "each": "testStreamLine" }, + { "file": "test-list-directory.ndjson", "each": "testListLine" }, + { "file": "test-cases-directory.ndjson", "each": "testStreamLine" }, + { "file": "test-directory.ndjson", "each": "testStreamLine" }, + { "file": "test-list-directory-917.ndjson", "each": "testListLine" }, + { "file": "test-directory-917.ndjson", "each": "testStreamLine" }, { "file": "build-database.json", "schema": "buildDatabaseEnvelope" }, { "file": "toolchain-list.json", "schema": "toolchainListEnvelope" }, { "file": "search.json", "schema": "searchEnvelope" }, diff --git a/protocol/fixtures/mtest-store/index.ndjson b/protocol/fixtures/mtest-store/index.ndjson new file mode 100644 index 0000000..2d64ba1 --- /dev/null +++ b/protocol/fixtures/mtest-store/index.ndjson @@ -0,0 +1,3 @@ +{"schema":"mtest.run/1","id":"20261108T100000-0a01","started":"2026-11-08T10:00:00Z","finished":"2026-11-08T10:00:01Z","status":"fail","system":"linux","arch":"x86_64","machine":"","commit":"","set":"mcpp test","selection":"","group":"","monitor":"off","ms":12,"jobs":2,"counts":{"pass":2,"fail":1}} +{"schema":"mtest.run/1","id":"20261109T100000-0b02","started":"2026-11-09T10:00:00Z","finished":"2026-11-09T10:00:01Z","status":"pass","system":"linux","arch":"x86_64","machine":"","commit":"","set":"mcpp test","selection":"","group":"","monitor":"off","ms":12,"jobs":2,"counts":{"pass":3}} +{"schema":"mtest.run/1","id":"20261110T100000-0c03","started":"2026-11-10T10:00:00Z","finished":"2026-11-10T10:00:01Z","status":"fail","system":"linux","arch":"x86_64","machine":"","commit":"","set":"mcpp test","selection":"","group":"","monitor":"off","ms":12,"jobs":2,"counts":{"pass":2,"fail":1}} diff --git a/protocol/fixtures/mtest-store/runs/20261108T100000-0a01/cases.ndjson b/protocol/fixtures/mtest-store/runs/20261108T100000-0a01/cases.ndjson new file mode 100644 index 0000000..1737d4f --- /dev/null +++ b/protocol/fixtures/mtest-store/runs/20261108T100000-0a01/cases.ndjson @@ -0,0 +1,2 @@ +{"failures":[],"id":"alpha","ms":0.5,"program":"unit/a","reason":"","status":"pass"} +{"failures":[{"file":"tests/unit/b.cpp","line":4,"message":"sum == 4","expected":"4","actual":"5"}],"id":"beta","ms":1.5,"program":"unit/b","reason":"","status":"fail"} diff --git a/protocol/fixtures/mtest-store/runs/20261108T100000-0a01/programs.ndjson b/protocol/fixtures/mtest-store/runs/20261108T100000-0a01/programs.ndjson new file mode 100644 index 0000000..2ce1ead --- /dev/null +++ b/protocol/fixtures/mtest-store/runs/20261108T100000-0a01/programs.ndjson @@ -0,0 +1,3 @@ +{"name": "unit/a", "argv": ["unit/a"], "status": "pass", "exit": 0, "ms": 1, "started_ms": 1, "worker": 0, "select": ["alpha"], "resources": [], "fixtures": []} +{"name": "unit/b", "argv": ["unit/b"], "status": "fail", "exit": 1, "ms": 2, "started_ms": 1, "worker": 1, "select": ["beta"], "resources": [], "fixtures": []} +{"name": "smoke", "argv": ["smoke"], "status": "pass", "exit": 0, "ms": 1, "started_ms": 2, "worker": 0, "select": [], "resources": [], "fixtures": []} diff --git a/protocol/fixtures/mtest-store/runs/20261108T100000-0a01/run.json b/protocol/fixtures/mtest-store/runs/20261108T100000-0a01/run.json new file mode 100644 index 0000000..e8742e1 --- /dev/null +++ b/protocol/fixtures/mtest-store/runs/20261108T100000-0a01/run.json @@ -0,0 +1,21 @@ +{ + "schema": "mtest.run/1", + "id": "20261108T100000-0a01", + "started": "2026-11-08T10:00:00Z", + "finished": "2026-11-08T10:00:01Z", + "status": "fail", + "system": "linux", + "arch": "x86_64", + "machine": "", + "commit": "", + "set": "mcpp test", + "selection": "", + "group": "", + "monitor": "off", + "ms": 12, + "jobs": 2, + "counts": { + "pass": 2, + "fail": 1 + } +} diff --git a/protocol/fixtures/mtest-store/runs/20261109T100000-0b02/cases.ndjson b/protocol/fixtures/mtest-store/runs/20261109T100000-0b02/cases.ndjson new file mode 100644 index 0000000..e5d746b --- /dev/null +++ b/protocol/fixtures/mtest-store/runs/20261109T100000-0b02/cases.ndjson @@ -0,0 +1,2 @@ +{"failures":[],"id":"alpha","ms":0.6,"program":"unit/a","reason":"","status":"pass"} +{"failures":[],"id":"beta","ms":1.4,"program":"unit/b","reason":"","status":"pass"} diff --git a/protocol/fixtures/mtest-store/runs/20261109T100000-0b02/programs.ndjson b/protocol/fixtures/mtest-store/runs/20261109T100000-0b02/programs.ndjson new file mode 100644 index 0000000..e09b371 --- /dev/null +++ b/protocol/fixtures/mtest-store/runs/20261109T100000-0b02/programs.ndjson @@ -0,0 +1,3 @@ +{"name": "unit/a", "argv": ["unit/a"], "status": "pass", "exit": 0, "ms": 1, "started_ms": 1, "worker": 0, "select": ["alpha"], "resources": [], "fixtures": []} +{"name": "unit/b", "argv": ["unit/b"], "status": "pass", "exit": 0, "ms": 2, "started_ms": 1, "worker": 1, "select": ["beta"], "resources": [], "fixtures": []} +{"name": "smoke", "argv": ["smoke"], "status": "pass", "exit": 0, "ms": 1, "started_ms": 2, "worker": 0, "select": [], "resources": [], "fixtures": []} diff --git a/protocol/fixtures/mtest-store/runs/20261109T100000-0b02/run.json b/protocol/fixtures/mtest-store/runs/20261109T100000-0b02/run.json new file mode 100644 index 0000000..1d48193 --- /dev/null +++ b/protocol/fixtures/mtest-store/runs/20261109T100000-0b02/run.json @@ -0,0 +1,20 @@ +{ + "schema": "mtest.run/1", + "id": "20261109T100000-0b02", + "started": "2026-11-09T10:00:00Z", + "finished": "2026-11-09T10:00:01Z", + "status": "pass", + "system": "linux", + "arch": "x86_64", + "machine": "", + "commit": "", + "set": "mcpp test", + "selection": "", + "group": "", + "monitor": "off", + "ms": 12, + "jobs": 2, + "counts": { + "pass": 3 + } +} diff --git a/protocol/fixtures/mtest-store/runs/20261110T100000-0c03/cases.ndjson b/protocol/fixtures/mtest-store/runs/20261110T100000-0c03/cases.ndjson new file mode 100644 index 0000000..df74f65 --- /dev/null +++ b/protocol/fixtures/mtest-store/runs/20261110T100000-0c03/cases.ndjson @@ -0,0 +1,2 @@ +{"failures":[],"id":"alpha","ms":0.4,"program":"unit/a","reason":"","status":"pass","kind":"bench","started_ms":1,"metrics":[{"name":"ready","value":4.2,"unit":"s","better":"lower"}],"bounds":[{"metric":"ready","max":15}]} +{"failures":[{"file":"tests/unit/b.cpp","line":4,"message":"sum == 4","expected":"4","actual":"5"}],"id":"beta","ms":1.6,"program":"unit/b","reason":"","status":"fail","kind":"unit","started_ms":1} diff --git a/protocol/fixtures/mtest-store/runs/20261110T100000-0c03/programs.ndjson b/protocol/fixtures/mtest-store/runs/20261110T100000-0c03/programs.ndjson new file mode 100644 index 0000000..2ce1ead --- /dev/null +++ b/protocol/fixtures/mtest-store/runs/20261110T100000-0c03/programs.ndjson @@ -0,0 +1,3 @@ +{"name": "unit/a", "argv": ["unit/a"], "status": "pass", "exit": 0, "ms": 1, "started_ms": 1, "worker": 0, "select": ["alpha"], "resources": [], "fixtures": []} +{"name": "unit/b", "argv": ["unit/b"], "status": "fail", "exit": 1, "ms": 2, "started_ms": 1, "worker": 1, "select": ["beta"], "resources": [], "fixtures": []} +{"name": "smoke", "argv": ["smoke"], "status": "pass", "exit": 0, "ms": 1, "started_ms": 2, "worker": 0, "select": [], "resources": [], "fixtures": []} diff --git a/protocol/fixtures/mtest-store/runs/20261110T100000-0c03/run.json b/protocol/fixtures/mtest-store/runs/20261110T100000-0c03/run.json new file mode 100644 index 0000000..fe1cf10 --- /dev/null +++ b/protocol/fixtures/mtest-store/runs/20261110T100000-0c03/run.json @@ -0,0 +1,21 @@ +{ + "schema": "mtest.run/1", + "id": "20261110T100000-0c03", + "started": "2026-11-10T10:00:00Z", + "finished": "2026-11-10T10:00:01Z", + "status": "fail", + "system": "linux", + "arch": "x86_64", + "machine": "", + "commit": "", + "set": "mcpp test", + "selection": "", + "group": "", + "monitor": "off", + "ms": 12, + "jobs": 2, + "counts": { + "pass": 2, + "fail": 1 + } +} diff --git a/protocol/fixtures/test-cases-directory.ndjson b/protocol/fixtures/test-cases-directory.ndjson new file mode 100644 index 0000000..b34f320 --- /dev/null +++ b/protocol/fixtures/test-cases-directory.ndjson @@ -0,0 +1,2 @@ +{"member":"","test":"unit","status":"built","exit_code":0,"signal":null,"duration_ms":0,"timed_out":false,"compile_output":"","run_output":"","reason":"","build_ms":40,"protocol":1,"listed":[{"id":"alpha","name":"alpha","file":"@ROOT@/tests/unit/a.cpp","kind":"bench","line":4,"bounds":[{"metric":"ready","max":15}]},{"id":"beta","name":"beta","file":"@ROOT@/tests/unit/b.cpp","kind":"unit","line":4,"tags":["slow"]}]} +{"summary":{"member":"","passed":0,"failed":0,"not_run":0,"not_run_reason":"","built":1,"elapsed_ms":60,"build_ms":40,"run_ms":0}} diff --git a/protocol/fixtures/test-directory-917.ndjson b/protocol/fixtures/test-directory-917.ndjson new file mode 100644 index 0000000..3642699 --- /dev/null +++ b/protocol/fixtures/test-directory-917.ndjson @@ -0,0 +1,8 @@ +{"case":{"member":"","test":"unit/a","id":"alpha","event":"begin","t_ms":3}} +{"case":{"member":"","test":"unit/a","id":"alpha","event":"end","t_ms":4,"status":"pass","duration_ms":0.4,"metrics":[{"name":"ready","value":4.2,"unit":"s","better":"lower"}],"bounds":[{"metric":"ready","max":15}],"kind":"bench"}} +{"member":"","test":"unit/a","part_of":"unit","status":"pass","exit_code":0,"signal":null,"duration_ms":1,"timed_out":false,"compile_output":"","run_output":"","reason":"","build_ms":0,"protocol":1,"cases":[{"duration_ms":0.4,"id":"alpha","status":"pass","kind":"bench","metrics":[{"name":"ready","value":4.2,"unit":"s","better":"lower"}],"bounds":[{"metric":"ready","max":15}]}]} +{"case":{"member":"","test":"unit/b","id":"beta","event":"begin","t_ms":3}} +{"case":{"member":"","test":"unit/b","id":"beta","event":"end","t_ms":5,"status":"fail","duration_ms":1.6,"failures":[{"file":"tests/unit/b.cpp","line":4,"message":"sum == 4","expected":"4","actual":"5"}],"kind":"unit","tags":["slow"]}} +{"member":"","test":"unit/b","part_of":"unit","status":"run_fail","exit_code":1,"signal":null,"duration_ms":2,"timed_out":false,"compile_output":"","run_output":"","reason":"","build_ms":0,"protocol":1,"cases":[{"duration_ms":1.6,"id":"beta","status":"fail","kind":"unit","tags":["slow"],"failures":[{"file":"tests/unit/b.cpp","line":4,"message":"sum == 4","expected":"4","actual":"5"}]}]} +{"member":"","test":"smoke","status":"pass","exit_code":0,"signal":null,"duration_ms":1,"timed_out":false,"compile_output":"","run_output":"smoke: ok\n","reason":"","build_ms":0,"protocol":null} +{"summary":{"member":"","passed":2,"failed":1,"not_run":0,"not_run_reason":"","built":0,"elapsed_ms":12,"build_ms":0,"run_ms":4}} diff --git a/protocol/fixtures/test-directory.ndjson b/protocol/fixtures/test-directory.ndjson new file mode 100644 index 0000000..f61c5aa --- /dev/null +++ b/protocol/fixtures/test-directory.ndjson @@ -0,0 +1,4 @@ +{"member":"","test":"unit/a","status":"pass","exit_code":0,"signal":null,"duration_ms":1,"timed_out":false,"compile_output":"","run_output":"","reason":"","build_ms":0,"protocol":1,"cases":[{"duration_ms":0.4,"id":"alpha","status":"pass"}]} +{"member":"","test":"unit/b","status":"run_fail","exit_code":1,"signal":null,"duration_ms":2,"timed_out":false,"compile_output":"","run_output":"","reason":"","build_ms":0,"protocol":1,"cases":[{"duration_ms":1.6,"id":"beta","status":"fail","failures":[{"file":"tests/unit/b.cpp","line":4,"message":"sum == 4","expected":"4","actual":"5"}]}]} +{"member":"","test":"smoke","status":"pass","exit_code":0,"signal":null,"duration_ms":1,"timed_out":false,"compile_output":"","run_output":"smoke: ok\n","reason":"","build_ms":0,"protocol":null} +{"summary":{"member":"","passed":2,"failed":1,"not_run":0,"not_run_reason":"","built":0,"elapsed_ms":12,"build_ms":0,"run_ms":4}} diff --git a/protocol/fixtures/test-list-directory-917.ndjson b/protocol/fixtures/test-list-directory-917.ndjson new file mode 100644 index 0000000..8e3bfee --- /dev/null +++ b/protocol/fixtures/test-list-directory-917.ndjson @@ -0,0 +1,3 @@ +{"member":"","test":"smoke","main":"@ROOT@/tests/smoke.cpp","files":["@ROOT@/tests/smoke.cpp"],"parts":[]} +{"member":"","test":"unit","main":null,"files":["@ROOT@/tests/unit/a.cpp","@ROOT@/tests/unit/b.cpp"],"parts":[{"name":"unit/a","file":"@ROOT@/tests/unit/a.cpp"},{"name":"unit/b","file":"@ROOT@/tests/unit/b.cpp"}]} +{"summary":{"total":2}} diff --git a/protocol/fixtures/test-list-directory.ndjson b/protocol/fixtures/test-list-directory.ndjson new file mode 100644 index 0000000..c0a3cac --- /dev/null +++ b/protocol/fixtures/test-list-directory.ndjson @@ -0,0 +1,3 @@ +{"member":"","test":"smoke","main":"@ROOT@/tests/smoke.cpp"} +{"member":"","test":"unit","main":"@ROOT@/tests/unit/a.cpp"} +{"summary":{"total":2}} diff --git a/protocol/mcpp-contract.schema.json b/protocol/mcpp-contract.schema.json index c327e47..d2c6606 100644 --- a/protocol/mcpp-contract.schema.json +++ b/protocol/mcpp-contract.schema.json @@ -161,13 +161,46 @@ "properties": { "member": { "type": "string" }, "test": { "type": "string" }, - "status": { "enum": ["pass", "fail", "skip", "not_run", "error"] }, + "status": { "enum": ["pass", "compile_fail", "run_fail", "not_run", "built"] }, "exit_code": { "type": ["integer", "null"] }, "duration_ms": { "type": "integer" }, "timed_out": { "type": "boolean" }, "compile_output": { "type": "string" }, "run_output": { "type": "string" }, - "reason": { "type": "string" } + "reason": { "type": "string" }, + "protocol": { "type": ["integer", "null"] }, + "part_of": { "type": "string", "description": "mcpp#917 (test-parts): the program a part's record belongs to" }, + "cases": { "type": "array", "items": { "$ref": "#/$defs/testCase" } }, + "listed": { "type": "array", "items": { "type": "object", "required": ["id"], "properties": { "id": { "type": "string" }, "file": { "type": "string" }, "line": { "type": "integer" }, "kind": { "type": "string" }, "tags": { "type": "array", "items": { "type": "string" } }, "bounds": { "$ref": "#/$defs/testBounds" } } } } + } + }, + "testBounds": { "type": "array", "items": { "type": "object", "required": ["metric"], "properties": { "metric": { "type": "string" }, "max": { "type": "number" }, "min": { "type": "number" } } } }, + "testCase": { + "description": "A case of a protocol program's record (docs/50 §8); metrics, bounds, kind and tags from test-metrics (mcpp#917).", + "type": "object", + "required": ["id", "status"], + "properties": { + "id": { "type": "string" }, + "status": { "enum": ["pass", "fail", "skip", "xfail", "xpass", "invalid"] }, + "duration_ms": { "type": "number" }, + "reason": { "type": "string" }, + "failures": { "type": "array", "items": { "type": "object", "required": ["file", "line", "message"], "properties": { "file": { "type": "string" }, "line": { "type": "integer" }, "message": { "type": "string" }, "expected": { "type": "string" }, "actual": { "type": "string" } } } }, + "metrics": { "type": "array", "items": { "type": "object", "required": ["name", "value"], "properties": { "name": { "type": "string" }, "value": { "type": "number" }, "unit": { "type": "string" } } } }, + "bounds": { "$ref": "#/$defs/testBounds" }, + "kind": { "type": "string" }, + "tags": { "type": "array", "items": { "type": "string" } } + } + }, + "testCaseRecord": { + "description": "mcpp#917 (test-case-stream): a case as it begins or ends, written while its program runs.", + "type": "object", + "required": ["case"], + "properties": { + "case": { + "type": "object", + "required": ["test", "id", "event"], + "properties": { "member": { "type": "string" }, "test": { "type": "string" }, "id": { "type": "string" }, "event": { "enum": ["begin", "end"] }, "t_ms": { "type": "number" }, "status": { "enum": ["pass", "fail", "skip", "xfail", "xpass", "invalid"] } } + } } }, "testSummary": { @@ -186,10 +219,12 @@ } } }, - "testStreamLine": { "anyOf": [ { "$ref": "#/$defs/testRecord" }, { "$ref": "#/$defs/testSummary" } ] }, + "testStreamLine": { "anyOf": [ { "$ref": "#/$defs/testRecord" }, { "$ref": "#/$defs/testCaseRecord" }, { "$ref": "#/$defs/testSummary" } ] }, "testListLine": { "anyOf": [ - { "type": "object", "required": ["test"], "properties": { "test": { "type": "string" }, "member": { "type": "string" }, "main": { "type": "string" } } }, + { "type": "object", "required": ["test"], "properties": { "test": { "type": "string" }, "member": { "type": "string" }, "main": { "type": ["string", "null"] }, + "files": { "type": "array", "items": { "type": "string" } }, + "parts": { "type": "array", "items": { "type": "object", "required": ["name", "file"], "properties": { "name": { "type": "string" }, "file": { "type": "string" } } } } } }, { "$ref": "#/$defs/testSummary" } ] }, diff --git a/protocol/protocol.schema.json b/protocol/protocol.schema.json index a951cca..aa0ae6a 100644 --- a/protocol/protocol.schema.json +++ b/protocol/protocol.schema.json @@ -106,19 +106,39 @@ "result": { "type": "object", "required": ["file", "kind"], "properties": { "file": { "type": "string" }, "root": { "type": "string" }, "kind": { "enum": ["program", "test", "none"] }, "test": { "type": "string" }, + "part": { "type": "string", "description": "A file of a program of several files: the part of the program the file is (`unit/a`); `test` is the program (design 2026-11-10)" }, "framework": { "type": "string" }, "main": { "type": "object", "required": ["line"], "properties": { "line": { "type": "integer" }, "approximate": { "type": "boolean" } } } } } }, "mcpp-ide/test/cases": { - "description": "A test program's cases (part 2 §7.3): as its source names them (a framework's macros), and with `list` -- trusted, and only for a source that calls its framework's macros -- as the program lists them. Without a framework, no cases: the program is the unit.", + "description": "A test program's cases (part 2 §7.3): as its sources name them (a framework's macros; every file of a program of several files, each case with its file), and with `list` -- trusted, and only for a source that calls its framework's macros -- as the program lists them. Without a framework, no cases: the program is the unit. A case of the test protocol carries its kind, tags, expectation and the bounds on its metrics when its source or its listing states them (design 2026-11-10).", "params": { "type": "object", "properties": { "root": { "type": "string" }, "test": { "type": "string" }, "file": { "type": "string" }, "list": { "type": "boolean" }, "target": { "type": "string" } } }, "result": { "type": "object", "required": ["root", "test", "framework", "cases"], "properties": { "root": { "type": "string" }, "test": { "type": "string" }, "file": { "type": "string" }, "framework": { "type": ["string", "null"] }, "listed": { "type": "boolean" }, - "main": { "type": "object" }, - "cases": { "type": "array", "items": { "type": "object", "required": ["name", "source"], "properties": { "name": { "type": "string" }, "line": { "type": "integer" }, "source": { "enum": ["static", "listed"] } } } } } + "main": { "type": "object" }, "files": { "type": "array", "items": { "type": "string" } }, + "cases": { "type": "array", "items": { "type": "object", "required": ["name", "source"], "properties": { "name": { "type": "string" }, "line": { "type": "integer" }, "source": { "enum": ["static", "listed"] }, + "file": { "type": "string" }, "kind": { "type": "string" }, "tags": { "type": "array", "items": { "type": "string" } }, "expect": { "type": "string" }, "bounds": { "$ref": "#/$defs/bounds" } } } } } } }, + "mcpp-ide/test/history": { + "description": "A project's test runs as mcpp kept them in mtest's store (target/mtest, or [test] store; mtest spec P12; design 2026-11-10): the last `runs` (20 by default, 50 at most), and each case's results across them, of one program or one case when asked. A store mcpp has not written answers no run.", + "params": { "type": "object", "required": ["root"], "properties": { "root": { "type": "string" }, "program": { "type": "string" }, "case": { "type": "string" }, "runs": { "type": "integer", "minimum": 1 } } }, + "result": { "type": "object", "required": ["root", "store", "runs", "cases"], "properties": { "root": { "type": "string" }, "store": { "type": "string" }, + "runs": { "type": "array", "items": { "$ref": "#/$defs/storedRun" } }, + "cases": { "type": "array", "items": { "type": "object", "required": ["program", "case", "results"], "properties": { "program": { "type": "string" }, "case": { "type": "string" }, "kind": { "type": "string" }, + "results": { "type": "array", "items": { "type": "object", "required": ["run", "status", "ms"], "properties": { "run": { "type": "string" }, "status": { "type": "string" }, "ms": { "type": "number" }, "metrics": { "$ref": "#/$defs/metrics" } } } } } } } } } + }, + "mcpp-ide/test/insights": { + "description": "What the last runs of a project's tests say (design 2026-11-10): the last run and its counts by kind; its failures, each with the run since which it fails; the slowest cases with their median; the cases whose status changed (flaky); each metric's series against its bound, with the last value's margin; the last run's schedule, a lane per worker with its critical path. `available` says which inputs the store holds, `features` which mcpp states.", + "params": { "type": "object", "required": ["root"], "properties": { "root": { "type": "string" }, "runs": { "type": "integer", "minimum": 1 } } }, + "result": { "type": "object", "required": ["root", "store", "runs", "run", "failures", "slowest", "flaky", "metrics", "schedule", "available", "features"], "properties": { + "root": { "type": "string" }, "store": { "type": "string" }, "runs": { "type": "integer" }, "run": { "oneOf": [{ "type": "null" }, { "$ref": "#/$defs/storedRun" }] }, + "byKind": { "type": "object" }, "failures": { "type": "array" }, "slowest": { "type": "array" }, "flaky": { "type": "array" }, "metrics": { "type": "array" }, + "schedule": { "oneOf": [{ "type": "null" }, { "type": "object", "required": ["lanes", "criticalPath"] }] }, + "available": { "type": "object", "properties": { "kinds": { "type": "boolean" }, "metrics": { "type": "boolean" }, "cases": { "type": "boolean" } } }, + "features": { "type": "object", "properties": { "parts": { "type": "boolean" }, "live": { "type": "boolean" }, "metrics": { "type": "boolean" } } } } } + }, "mcpp-ide/test/runCases": { "description": "Runs some of a test program's cases (all of them when `cases` is empty): a test job whose program receives the framework's filter and report arguments after `--`. Each case's outcome is sent as test/caseResult when the job ends.", "params": { "type": "object", "required": ["root", "test", "framework"], "properties": { "root": { "type": "string" }, "test": { "type": "string" }, "framework": { "type": "string" }, "cases": { "type": "array", "items": { "type": "string" } }, "profile": { "type": "string" }, "target": { "type": "string" }, "color": { "type": "boolean" } } }, @@ -216,12 +236,14 @@ "result": { "type": "object", "required": ["engine", "configuration", "launch"] } }, "mcpp-ide/test/list": { - "description": "The tests of a project without building them (mcpp test --list).", + "description": "The tests of a project without building them (mcpp test --list): each program with its own files, the file that defines its main (empty when none does), its layout, and the parts a run divides it into (a protocol program of several files runs as a process per file, `/`; mcpp states them with test-parts, else they are derived; design 2026-11-10).", "params": { "$ref": "#/$defs/rooted" }, "result": { - "type": "object", - "required": ["root", "tests"], - "properties": { "tests": { "type": "array", "items": { "type": "object", "required": ["test"], "properties": { "test": { "type": "string" }, "member": { "type": "string" }, "main": { "type": "string" } } } } } + "type": "object", "required": ["root", "tests"], + "properties": { "tests": { "type": "array", "items": { "type": "object", "required": ["test"], "properties": { + "test": { "type": "string" }, "member": { "type": "string" }, "main": { "type": "string" }, + "files": { "type": "array", "items": { "type": "string" } }, "layout": { "enum": ["file", "directory"] }, + "parts": { "type": "array", "items": { "type": "object", "required": ["name", "file"], "properties": { "name": { "type": "string" }, "file": { "type": "string" } } } } } } } } } }, "mcpp-ide/library/search": { @@ -460,7 +482,13 @@ "output": { "type": "string" }, "outputTruncated": { "type": "boolean" }, "location": { "$ref": "#/$defs/failureLocation" }, - "reason": { "type": "string" } + "reason": { "type": "string" }, + "program": { "type": "string", "description": "The program the record belongs to: mcpp's part_of, else the listed program whose name is the record's or begins it with a `/`" }, + "part": { "type": "string", "description": "The part of the program the record is, when it is one (`unit/a`)" }, + "partOf": { "type": "string" }, + "protocol": { "type": "integer" }, + "cases": { "type": "array", "items": { "$ref": "#/$defs/caseOutcome" }, "description": "A protocol program's cases, each as it ended" }, + "usage": { "type": "object", "description": "What the program used (mtest protocol P10.2), when mcpp states it (test-metrics)" } } } }, @@ -469,10 +497,17 @@ "mcpp-ide/manifest/changed": { "params": { "type": "object", "required": ["root"] } }, "mcpp-ide/manifest/diagnostics": { "params": { "type": "object", "required": ["root", "diagnostics"] } }, "mcpp-ide/project/mcppChanged": { "description": "A job's envelope named another mcpp than the project's was asked as: it is asked again on its next use.", "params": { "type": "object", "required": ["root", "version"] } }, - "mcpp-ide/test/caseResult": { "description": "One case's outcome, read from the framework's report when a test/runCases job ended, with the case's own output (GoogleTest's console between [ RUN ] and its result line; empty for a framework whose console names no case) and a failure's location.", "params": { "type": "object", "required": ["jobId", "root", "test", "case", "status", "output", "outputTruncated"], "properties": { "status": { "enum": ["passed", "failed", "skipped"] }, "durationMs": { "type": "integer" }, "message": { "type": "string" }, "output": { "type": "string" }, "outputTruncated": { "type": "boolean" }, "location": { "$ref": "#/$defs/failureLocation" } } } }, + "mcpp-ide/test/caseResult": { "description": "One case's outcome when a test/runCases job ended: read from the framework's report, or from each record of a protocol program (one per part of a program of several files), with the case's own output (GoogleTest's console between [ RUN ] and its result line; a protocol case's first failure) and a failure's location.", "params": { "allOf": [{ "$ref": "#/$defs/caseOutcome" }, { "type": "object", "required": ["jobId", "root", "test", "case", "status", "output", "outputTruncated"], "properties": { "test": { "type": "string" }, "program": { "type": "string" }, "part": { "type": "string" }, "output": { "type": "string" }, "outputTruncated": { "type": "boolean" } } }] } }, + "mcpp-ide/test/caseEvent": { "description": "A case began or ended while its program runs, as mcpp writes it (test-case-stream, mcpp#917): `begin` names the case; `end` carries its outcome. `program` is the program the part belongs to.", "params": { "allOf": [{ "type": "object", "required": ["jobId", "root", "program", "part", "case", "event"], "properties": { "program": { "type": "string" }, "part": { "type": "string" }, "member": { "type": "string" }, "event": { "enum": ["begin", "end"] }, "tMs": { "type": "number" } } }, { "$ref": "#/$defs/caseOutcomePartial" }] } }, "mcpp-ide/languageService/changed": { "description": "mcppls answered differently from the last time it was asked: another program or version, other actions, or another state of the workspace. An editor that lists the commands asks for them again.", "params": { "type": "object", "required": ["root", "state", "workspaceState"] } } }, "$defs": { + "storedRun": { "type": "object", "required": ["id", "status"], "properties": { "id": { "type": "string" }, "started": { "type": "string" }, "status": { "type": "string" }, "ms": { "type": "number" }, "jobs": { "type": "integer" }, "counts": { "type": "object" } } }, + "bounds": { "description": "The bounds a case states on its metrics (mtest protocol P6.5)", "type": "array", "items": { "type": "object", "required": ["metric"], "properties": { "metric": { "type": "string" }, "max": { "type": "number" }, "min": { "type": "number" } } } }, + "metrics": { "description": "What a case measured (mtest protocol P6), when mcpp states it (test-metrics)", "type": "array", "items": { "type": "object", "required": ["name", "value"], "properties": { "name": { "type": "string" }, "value": { "type": "number" }, "unit": { "type": "string" }, "samples": { "type": "array", "items": { "type": "number" } }, "better": { "enum": ["lower", "higher"] }, "precision": { "type": "string" } } } }, + "caseFailure": { "type": "object", "required": ["path", "line", "message"], "properties": { "path": { "type": "string" }, "line": { "type": "integer", "minimum": 0 }, "message": { "type": "string" }, "expected": { "type": "string" }, "actual": { "type": "string" } } }, + "caseOutcome": { "description": "A case as it ended: pass is passed, fail failed, skip skipped, xfail passed with expectedFailure, xpass failed with unexpectedPass, invalid (a measurement that cannot be judged) errored. Lines are 0-based; message is the first failure with its place, else the reason.", "type": "object", "required": ["case", "status"], "properties": { "case": { "type": "string" }, "status": { "enum": ["passed", "failed", "skipped", "errored"] }, "durationMs": { "type": "integer" }, "reason": { "type": "string" }, "message": { "type": "string" }, "location": { "$ref": "#/$defs/failureLocation" }, "failures": { "type": "array", "items": { "$ref": "#/$defs/caseFailure" } }, "metrics": { "$ref": "#/$defs/metrics" }, "bounds": { "$ref": "#/$defs/bounds" }, "kind": { "type": "string" }, "tags": { "type": "array", "items": { "type": "string" } }, "expectedFailure": { "type": "boolean" }, "unexpectedPass": { "type": "boolean" } } }, + "caseOutcomePartial": { "type": "object", "properties": { "case": { "type": "string" }, "status": { "enum": ["passed", "failed", "skipped", "errored"] }, "durationMs": { "type": "integer" }, "failures": { "type": "array", "items": { "$ref": "#/$defs/caseFailure" } }, "metrics": { "$ref": "#/$defs/metrics" }, "bounds": { "$ref": "#/$defs/bounds" } } }, "failureLocation": { "description": "Where a failure is: the first file:line (or file(line):) its output names, line 0-based; null when it names none.", "oneOf": [{ "type": "null" }, { "type": "object", "required": ["path", "line"], "properties": { "path": { "type": "string" }, "line": { "type": "integer", "minimum": 0 } } }] }, "settingValues": { "type": "object", "additionalProperties": true }, "resolvedSettings": { "type": "object", "required": ["values", "rejected", "renames", "deprecated"] }, diff --git a/spec/commands.json b/spec/commands.json index e1d357c..cf2e793 100644 --- a/spec/commands.json +++ b/spec/commands.json @@ -98,6 +98,9 @@ "action": { "kind": "panel", "panel": "graph", "graph": "packages" }, "quickMenu": true }, { "id": "mcpp.project.showModuleDependencyGraph", "group": "project", "action": { "kind": "panel", "panel": "graph", "graph": "modules" }, "quickMenu": true }, + { "id": "mcpp.test.insights", "group": "project", "icon": { "vscode": "$(graph-line)", "jetbrains": "AllIcons.RunConfigurations.TestState.Run" }, + "when": "mcpp.inProject", + "action": { "kind": "panel", "panel": "tests" }, "quickMenu": true }, { "id": "mcpp.languageServer.refreshState", "group": "languageService", "action": { "kind": "languageService" } }, { "id": "mcpp.openMcpplsSettings", "group": "languageService", "palette": false, "editors": [ "vscode" ], diff --git a/spec/i18n/manifest.en.json b/spec/i18n/manifest.en.json index 0e456e6..1465565 100644 --- a/spec/i18n/manifest.en.json +++ b/spec/i18n/manifest.en.json @@ -211,6 +211,7 @@ "viewsWelcome.library": "No mcpp package index was found on this machine.\n\nRun `mcpp index update` once, or set `mcpp.library.indexPath` to an index checkout.\n[Run mcpp index update](command:mcpp.library.updateIndex)", "command.mcpp.project.showDependencyGraph.title": "Show Project Dependency Graph", "command.mcpp.project.showModuleDependencyGraph.title": "Show C++ Modules Dependency Graph", + "command.mcpp.test.insights.title": "Show Test Insights", "command.mcpp.debug.title": "Debug", "command.mcpp.pack.title": "Pack", "command.mcpp.cancelJob.title": "Cancel the Running Job", diff --git a/spec/i18n/manifest.zh-cn.json b/spec/i18n/manifest.zh-cn.json index bfb007f..1e39c5c 100644 --- a/spec/i18n/manifest.zh-cn.json +++ b/spec/i18n/manifest.zh-cn.json @@ -211,6 +211,7 @@ "viewsWelcome.library": "本机没有找到 mcpp 包索引。\n\n先运行一次 `mcpp index update`，或用 `mcpp.library.indexPath` 指向一个索引检出。\n[运行 mcpp index update](command:mcpp.library.updateIndex)", "command.mcpp.project.showDependencyGraph.title": "显示项目依赖图", "command.mcpp.project.showModuleDependencyGraph.title": "显示 C++ Modules 依赖图", + "command.mcpp.test.insights.title": "显示测试洞察", "command.mcpp.debug.title": "调试", "command.mcpp.pack.title": "打包", "command.mcpp.cancelJob.title": "取消正在运行的任务", diff --git a/spec/i18n/runtime.zh-cn.json b/spec/i18n/runtime.zh-cn.json index 07d478b..51f5017 100644 --- a/spec/i18n/runtime.zh-cn.json +++ b/spec/i18n/runtime.zh-cn.json @@ -771,6 +771,7 @@ "mcpp: Pack (2/3): what the machine that runs it provides": "mcpp：打包（2/3）：运行它的机器提供什么", "mcpp: Pack (3/3): the libraries inside the program or beside it": "mcpp：打包（3/3）：库在程序里还是在程序旁边", "the project's mcpp {0} does not state {1}; mcpp {2} does": "项目的 mcpp {0} 未声明 {1}；mcpp {2} 声明了它", + "the project's mcpp {0} does not state {1}, which {2} proposes": "项目的 mcpp {0} 未声明 {1}（由 {2} 提出）", "the workspace is not trusted: the project's mcpp is not asked": "工作区不受信任：未询问项目的 mcpp", "The mcpp-ide backend ended four times in three minutes; build, run, test and the project pages go on without it.": "mcpp-ide 后端在三分钟内结束了四次；构建、运行、测试与项目页面在没有它的情况下继续工作。", "Restart the backend": "重新启动后端", @@ -813,5 +814,24 @@ "did not compile": "没有编译通过", "built, not run": "已构建，未运行", "mcpp test in {0}": "{0} 中的 mcpp test", - "(the output was longer; its head and tail are shown)": "（输出更长，只显示开头和结尾）" + "(the output was longer; its head and tail are shown)": "（输出更长，只显示开头和结尾）", + "The case passed, though it is expected to fail.": "该用例本应失败，却通过了。", + "Test Insights": "测试洞察", + "No run is kept yet: mcpp test keeps each run in {0} (mcpp 2026.11.9.1 and later).": "尚无保留的运行：mcpp test 会把每次运行保存在 {0}（mcpp 2026.11.9.1 及以后）。", + "{0} runs kept · the last began {1}": "保留了 {0} 次运行 · 最近一次开始于 {1}", + "Failures": "失败", + "new": "新增", + "since {0}": "自 {0} 起", + "expected": "期望", + "actual": "实际", + "Metrics": "指标", + "Slowest": "最慢", + "median {0}": "中位数 {0}", + "Flaky": "不稳定", + "{0} passed · {1} failed · {2} changes": "通过 {0} · 失败 {1} · 变化 {2} 次", + "Schedule": "调度", + "worker {0}": "工作线程 {0}", + "Critical path": "关键路径", + "Metrics appear when the project's mcpp states them (test-metrics, mcpp#917).": "项目的 mcpp 声明指标后才会显示（test-metrics，mcpp#917）。", + "This IDE runs without JCEF, which draws the test insights.": "此 IDE 未启用 JCEF，无法绘制测试洞察。" } diff --git a/src/bin/dev.cpp b/src/bin/dev.cpp index eb42ac7..188d9c3 100644 --- a/src/bin/dev.cpp +++ b/src/bin/dev.cpp @@ -343,7 +343,7 @@ int gen_jetbrains(std::span args) { xml += " \n"; std::map> byGroup; for (const auto& c : spec.commands["commands"]) - if (offered_in(c, "jetbrains") && !c.value("deprecated", false)) byGroup[c.value("group", "")].push_back(&c); + if (offered_in(c, "jetbrains")) byGroup[c.value("group", "")].push_back(&c); for (const auto& g : spec.commands["groups"]) { const auto gid = g.value("id", ""); if (!byGroup.contains(gid)) continue; diff --git a/src/bin/mockmcpp.cpp b/src/bin/mockmcpp.cpp index 2ba39ef..ed9b201 100644 --- a/src/bin/mockmcpp.cpp +++ b/src/bin/mockmcpp.cpp @@ -12,6 +12,9 @@ // MOCK_MCPP_LOG every argv is appended here, one line each, and the value of // XLINGS_PROJECT_DIR the run received (`` when none) // +// `.mock-mcpp-tests` in the directory a command runs in selects the programs `mcpp test` states +// (protocol/fixtures/test*-.ndjson). +// // AND A VERSION MANAGER'S SHIM, per directory (design 2026-10-30 part 2 §4.2): in the directory a // command runs in, `.mock-mcpp-version` names the version that answers (every output's version is // that one), `.mock-mcpp-features` the features its envelopes state (comma-separated; without @@ -192,6 +195,16 @@ int main(int argc, char** argv) { // A program that implements the test protocol: its listing (`--cases`) and a run of a selection // (`--case`), each a record with the protocol's fields (mcpp docs/50 §8). if (first(0) == "test") { + // `.mock-mcpp-tests` names another set of programs: `directory` is a protocol program of two + // files and a file program as mcpp 2026.11.9.1 states them; `directory-917` the same with + // mcpp#917's parts, case records and metrics (its listing of cases is the first set's). + if (const auto set = here_file(".mock-mcpp-tests"); !set.empty()) { + const auto kind = has("--list") ? "test-list" : has("--cases") ? "test-cases" : "test"; + auto name = std::format("{}-{}.ndjson", kind, set); + if (!std::filesystem::exists(fixtures_dir() / name)) name = std::format("{}-{}.ndjson", kind, set.substr(0, set.find('-'))); + out(fixture(name)); + return 0; + } out(fixture(has("--list") ? "test-list.ndjson" : has("--cases") ? "test-cases.ndjson" : has("--case") ? "test-protocol.ndjson" : "test.ndjson")); return 0; diff --git a/src/buildscript.cppm b/src/buildscript.cppm index 38550dc..104693c 100644 --- a/src/buildscript.cppm +++ b/src/buildscript.cppm @@ -14,7 +14,7 @@ // one cross-line heuristic is the action block, from `mcpp::action` to `.submit(`, up to 40 lines. // Every rule errs towards silence. Diagnostic positions are 1-based (`line:column`); a column counts // UTF-16 units. A port of mcpp-vscode 0.9's src/buildscript/{analysis,api,modules,snippets,providers}.ts -// (design 2026-10-30 part 2 §8); tests/rules holds that implementation's answers. +// (design 2026-10-30 part 2 §8); tests/cts/rules holds that implementation's answers. export module mcppide.buildscript; import std; diff --git a/src/manifest.cppm b/src/manifest.cppm index d236098..f77aced 100644 --- a/src/manifest.cppm +++ b/src/manifest.cppm @@ -10,7 +10,7 @@ // Positions in: a 0-based line and a UTF-16 column, as the editors count. Positions out: the same, // except a diagnostic's, which is 1-based (`line:column`, the form a terminal prints). A port of // mcpp-vscode 0.9's src/toml/{completion,hover,navigation,diagnostics}.ts (design 2026-10-30 part 2 -// §8); tests/rules holds that implementation's answers over a corpus, and this one gives them. +// §8); tests/cts/rules holds that implementation's answers over a corpus, and this one gives them. export module mcppide.manifest; import std; diff --git a/src/scan.cppm b/src/scan.cppm index b3947b4..b9f0163 100644 --- a/src/scan.cppm +++ b/src/scan.cppm @@ -29,11 +29,15 @@ struct Call { // A case of the mtest library -- an object (`UTest parse{"parse", ...}`, any suite type) or a // `"name"_test` declared in one -- as mtest's own static manifest reads it // (mcpplibs.mtest.manifest): its identifier, the names of what declares it and its own joined by -// `/`, and where its name is (0-based). +// `/`, and where its name is (0-based); the kind, tags and bounds on its metrics it states (empty +// kind: its program's). struct Literal { std::string id; int line = 0; int column = 0; + std::string kind; + std::vector tags; + std::vector bounds; }; struct Result { @@ -228,7 +232,7 @@ Result scan(std::string_view text, const std::vector& macros, } // mtest's cases as mtest reads them: its manifest is the one reader of its declarations. for (const auto& found : mcpplibs::mtest::manifest::scan(text, types)) - r.cases.push_back(Literal{found.id, found.line - 1, found.column - 1}); + r.cases.push_back(Literal{found.id, found.line - 1, found.column - 1, found.kind, found.tags, found.bounds}); return r; } diff --git a/src/server.cppm b/src/server.cppm index ddd97c4..0e7f14b 100644 --- a/src/server.cppm +++ b/src/server.cppm @@ -28,25 +28,28 @@ import mcppide.languageservice; import mcppide.scan; import mcpplibs.mtest.manifest; import mcppide.testing; +import mcppide.teststore; import mcpplibs.os; export namespace mcppide::server { using Json = nlohmann::json; -inline constexpr std::string_view kVersion = "1.2.1"; +inline constexpr std::string_view kVersion = "1.3.0"; // THE SERIES (mcpp design 2026-11-07, D15): `X.Y` of `X.Y.Z`, shared by the backend and every editor -// plugin; components of one series work together, and `Z` is each component's own patch. A plugin -// refuses a backend of another series, naming both versions. -inline constexpr std::string_view kSeries = "1.2"; -static_assert(kVersion.starts_with(kSeries) && kVersion.size() > kSeries.size() && kVersion[kSeries.size()] == '.'); +// plugin; components of one series work together. A plugin refuses a backend of another series, +// naming both versions. The backend is always `X.Y.0`: the VS Code and CLion packages carry it, so a +// change of the backend is a change of every package, and it begins a series (1.3.0). `Z` is an +// editor plugin's own patch. +inline constexpr std::string_view kSeries = "1.3"; +static_assert(kVersion.starts_with(kSeries) && kVersion.size() == kSeries.size() + 2 && kVersion.ends_with(".0")); inline constexpr int kProtocol = 2; // The methods `install` registers, for the protocol schema's test: a method // here and not in protocol/protocol.schema.json (or the other way) fails it. The session is an LSP // server (protocol 2, design 2026-10-30 part 2 §8): LSP's own methods, and mcpp-ide's under // `mcpp-ide/`. -inline constexpr std::array kMethods = { +inline constexpr std::array kMethods = { "initialize", "shutdown", "exit", "textDocument/completion", "textDocument/hover", "textDocument/definition", "textDocument/documentSymbol", "mcpp-ide/settings/update", "mcpp-ide/settings/registry", "mcpp-ide/settings/preset", "mcpp-ide/i18n/bundle", "mcpp-ide/selfCheck", "mcpp-ide/commands/list", @@ -60,19 +63,20 @@ inline constexpr std::array kMethods = { "mcpp-ide/pack/start", "mcpp-ide/manifest/check", "mcpp-ide/mcpp/exec", "mcpp-ide/mcpp/availability", "mcpp-ide/project/files", "mcpp-ide/languageService/exec", "mcpp-ide/project/mcpp", "mcpp-ide/project/refresh", "mcpp-ide/capabilities/explain", "mcpp-ide/report/collect", "mcpp-ide/source/entries", "mcpp-ide/test/cases", - "mcpp-ide/test/runCases"}; + "mcpp-ide/test/runCases", "mcpp-ide/test/history", "mcpp-ide/test/insights"}; // The notifications the backend reads. inline constexpr std::array kReceived = { "initialized", "textDocument/didOpen", "textDocument/didChange", "textDocument/didClose", "$/cancelRequest"}; // The notifications the backend sends. -inline constexpr std::array kNotifications = { +inline constexpr std::array kNotifications = { "textDocument/publishDiagnostics", "window/showMessage", "mcpp-ide/job/started", "mcpp-ide/job/output", "mcpp-ide/job/plan", "mcpp-ide/job/progress", "mcpp-ide/job/diagnostic", "mcpp-ide/job/artifact", "mcpp-ide/job/record", "mcpp-ide/job/finished", "mcpp-ide/test/result", "mcpp-ide/test/summary", "mcpp-ide/result/changed", "mcpp-ide/manifest/changed", "mcpp-ide/manifest/diagnostics", - "mcpp-ide/project/mcppChanged", "mcpp-ide/languageService/changed", "mcpp-ide/test/caseResult"}; + "mcpp-ide/project/mcppChanged", "mcpp-ide/languageService/changed", "mcpp-ide/test/caseResult", + "mcpp-ide/test/caseEvent"}; class Session { public: @@ -110,6 +114,23 @@ private: Result source_entries(const Json& p); Result test_cases(const Json& p); Result test_run_cases(const Json& p); + Result test_history(const Json& p); + Result test_insights(const Json& p); + // The programs of `mcpp test --list` (kept per project), and lookups among them. + std::vector list_programs(const std::filesystem::path& root, const Json& p); + std::vector programs_of(const std::filesystem::path& root); + std::optional program_named(const std::filesystem::path& root, std::string_view name); + std::optional program_of_file(const std::filesystem::path& root, const std::filesystem::path& file); + std::optional> listed_cases(const std::filesystem::path& root, const std::string& test, const Json& p, + const std::vector& statics); + std::optional> framework_listing(const std::filesystem::path& root, const std::string& test, + const Json& p, const testing::Framework& f); + // A record of a test job, with the program and part it belongs to, and its cases as the editors + // receive them (design 2026-11-10). + Json test_result_of(const std::filesystem::path& root, const Json& record, Json n); + void protocol_case_results(std::string_view jobId, const std::filesystem::path& root, const std::string& test, const Json& record); + // The store's last runs (kept per project; a run is read once). + std::vector store_runs(const std::filesystem::path& root, std::size_t count); // `mcpp.test.macros` as testing reads it: {"gtest": ["MY_TEST"]}. Json test_macros() const; // The packages of a project's plan, from its build database when one was read. @@ -190,9 +211,13 @@ private: // The whole output of the last test record of each running test job, for a case run to divide // among its cases when the job ends (test/runCases). std::map> programOutput_; - // The last test record of each running test job, whole: a program that implements the test - // protocol reports its cases in it (test/runCases). - std::map> testRecord_; + // The test records of each running test job, whole: a program that implements the test protocol + // reports its cases in them, one record per part of a program of several files (test/runCases). + std::map, std::less<>> testRecords_; + // The programs of each project as `mcpp test --list` last named them. + std::map> programs_; + // The runs of each project's store already read, by id. + std::map> storeRuns_; // The configuration (and platforms) the last plan of each project named. std::map plans_; // What `mcpp pack` accepts, read from its help once per mcpp (SPEC-014: `--pkg-format` names the @@ -268,7 +293,7 @@ Session::Session(rpc::Server& server) : rpc_(server) { jobOf_.erase(it); } if (auto it = programOutput_.find(id); it != programOutput_.end()) programOutput_.erase(it); - if (auto it = testRecord_.find(id); it != testRecord_.end()) testRecord_.erase(it); + if (auto it = testRecords_.find(id); it != testRecords_.end()) testRecords_.erase(it); } rpc_.notify("mcpp-ide/job/finished", s); // Every envelope names the mcpp that wrote it: another version than the one probed (a @@ -389,6 +414,7 @@ void Session::on_changed(const std::filesystem::path& root, std::string_view wha std::lock_guard lock(mutex_); buildDatabases_.erase(root); if (what != "result") probes_.erase(root); // a manifest may state another mcpp floor or pin + if (what != "result") programs_.erase(root); // and other tests } if (what == "result") { rpc_.notify("mcpp-ide/result/changed", Json{{"root", root.generic_string()}, {"result", detail}}); @@ -428,10 +454,28 @@ void Session::on_record(std::string_view jobId, const Json& record) { } else if (type == "test") { { std::lock_guard lock(mutex_); - programOutput_[std::string(jobId)] = record.value("compile_output", "") + record.value("run_output", ""); - testRecord_[std::string(jobId)] = record; + programOutput_[std::string(jobId)] += record.value("compile_output", "") + record.value("run_output", ""); + testRecords_[std::string(jobId)].push_back(record); } - rpc_.notify("mcpp-ide/test/result", n); + rpc_.notify("mcpp-ide/test/result", test_result_of(std::filesystem::path(root), record, std::move(n))); + } else if (type == "case") { + // A case as it begins or ends while its program runs (mcpp#917, `test-case-stream`). + const auto& c = n["case"]; + Json event = c.value("event", "") == "end" ? testing::protocol_case(c) : Json{{"case", c.value("id", "")}}; + const auto part = c.value("test", ""); + std::vector programs; + { + std::lock_guard lock(mutex_); + if (auto it = programs_.find(std::filesystem::path(root)); it != programs_.end()) programs = it->second; + } + event["jobId"] = jobId; + event["root"] = root; + event["member"] = c.value("member", ""); + event["program"] = testing::owner_of(part, c.value("part_of", ""), programs); + event["part"] = part; + event["event"] = c.value("event", ""); + if (c.contains("t_ms") && c["t_ms"].is_number()) event["tMs"] = c["t_ms"]; + rpc_.notify("mcpp-ide/test/caseEvent", event); } else if (type == "summary") { rpc_.notify("mcpp-ide/test/summary", n); } else if (!(record.is_object() && record.contains("schemaVersion") && record.value("kind", "") != "mcpp.event")) { @@ -708,6 +752,8 @@ std::expected Session::job_start(const Json& p, std::function< r.mode = jobs::Mode::ndjson; break; case jobs::Kind::test: + // The programs, so that each part's record is attributed to its program (design 2026-11-10). + (void)programs_of(root); r.args = {"test", "--message-format", "json"}; if (release) r.args.push_back("--release"); append(target); @@ -1227,25 +1273,30 @@ std::expected Session::project_refresh(const Json& p) { std::expected Session::capabilities_explain(const Json& p) { const auto root = any_root(p).lexically_normal(); const auto probe = probe_of(root); - struct Gate { const char* id; const char* feature; const char* since; }; + // `since` is the first mcpp that states the feature; a feature no release states yet names the + // issue that proposes it instead (`issue`), and its message says so. + struct Gate { const char* id; const char* feature; const char* since; const char* issue; }; static constexpr Gate kGates[] = { - {"targets.sets", "target-sets", "2026.10.30.1"}, - {"pack.formatList", "pkg-format-list", "2026.10.30.1"}, - {"tests.debug", "test-launch", "2026.10.31.2"}, + {"targets.sets", "target-sets", "2026.10.30.1", nullptr}, + {"pack.formatList", "pkg-format-list", "2026.10.30.1", nullptr}, + {"tests.debug", "test-launch", "2026.10.31.2", nullptr}, + {"tests.cases", "test-protocol", "2026.11.8.1", nullptr}, + {"tests.parts", "test-parts", "", "mcpp#917"}, + {"tests.live", "test-case-stream", "", "mcpp#917"}, + {"tests.metrics", "test-metrics", "", "mcpp#917"}, }; Json features = Json::array(); for (const auto& g : kGates) { const bool on = std::ranges::find(probe.features, g.feature) != probe.features.end(); Json f{{"id", g.id}, {"available", on}}; - if (!on) - f["reason"] = Json{{"source", "mcpp"}, - {"capability", g.feature}, - {"need", g.since}, - {"have", probe.version}, - {"message", probe.state == "ok" || probe.state == "not-probed" - ? t("the project's mcpp {0} does not state {1}; mcpp {2} does", - {probe.version, g.feature, g.since}) - : probe.message}}; + if (!on) { + const bool known = probe.state == "ok" || probe.state == "not-probed"; + const auto message = !known ? probe.message + : g.issue ? t("the project's mcpp {0} does not state {1}, which {2} proposes", {probe.version, g.feature, g.issue}) + : t("the project's mcpp {0} does not state {1}; mcpp {2} does", {probe.version, g.feature, g.since}); + f["reason"] = Json{{"source", "mcpp"}, {"capability", g.feature}, {"need", g.issue ? g.issue : g.since}, + {"have", probe.version}, {"message", message}}; + } features.push_back(std::move(f)); } return Json{{"root", root.generic_string()}, {"mcpp", probe_json(root, probe)}, {"features", features}}; @@ -1381,32 +1432,19 @@ std::expected Session::quick_menu(const Json& p) { } // The tests of a project, without building them: `mcpp test --list --message-format json` (one -// record per test, with the file that holds its `main`), else the test targets of the build database. +// record per program), each with its files and parts (design 2026-11-10), else the test targets of +// the build database. The programs are kept per project: a run's record is attributed to its +// program by them. std::expected Session::test_list(const Json& p) { const auto root = root_of(p); if (root.empty()) return std::unexpected(invalid("test/list names its project's root")); - mcpp::Client client; - { - std::lock_guard lock(mutex_); - client = client_; - } Json tests = Json::array(); - if (client.ready()) { - mcpp::RunOptions o; - o.cwd = root; - o.args = {"test", "--list", "--message-format", "json"}; - for (const auto& a : target_args(p)) o.args.push_back(a); - o.deny = deny(); - o.client = client_label(); - if (auto r = client.records(o); r && r->exitCode == 0) - for (const auto& rec : r->records) - if (rec.is_object() && rec.contains("test") && rec["test"].is_string()) - tests.push_back(Json{{"test", rec["test"]}, {"member", rec.value("member", "")}, {"main", rec.value("main", "")}}); - } + for (const auto& program : list_programs(root, p)) tests.push_back(testing::program_json(program)); if (tests.empty()) { auto db = build_database(root); if (db) for (const auto& row : views::project_summary(*db, Json(nullptr))["tests"]) - tests.push_back(Json{{"test", row.value("name", "")}, {"member", ""}, {"main", ""}}); + tests.push_back(Json{{"test", row.value("name", "")}, {"member", ""}, {"main", ""}, {"files", Json::array()}, + {"parts", Json::array()}, {"layout", "file"}}); } return Json{{"root", root.generic_string()}, {"tests", tests}}; } @@ -1472,15 +1510,38 @@ std::optional test_name_of(const std::filesystem::path& root, const } // The suite types the project's tests declare (`class Fixture : public CTS` in a module beside them), -// read across the files of tests/ as mtest's static manifest reads them. +// read across the files of tests/ as mtest's static manifest reads them. They are read again only +// when the files changed: a walk that only stats them (their count, latest change and total size) +// decides, so a source/entries or test/cases call does not read every file of tests/. mcpplibs::mtest::manifest::suite_types test_suite_types(const std::filesystem::path& root) { static const std::set cxx{".cpp", ".cc", ".cxx", ".cppm", ".ixx", ".mpp"}; - std::vector sources; + struct Kept { + std::tuple signature; + mcpplibs::mtest::manifest::suite_types types; + }; + static std::mutex mutex; + static std::map kept; + std::vector files; + std::tuple signature{}; std::error_code ec; - for (std::filesystem::recursive_directory_iterator it(root / "tests", ec), end; !ec && it != end; it.increment(ec)) - if (it->is_regular_file(ec) && cxx.contains(it->path().extension().string())) - sources.push_back(read_text(it->path())); - return mcpplibs::mtest::manifest::types_of(sources); + for (std::filesystem::recursive_directory_iterator it(root / "tests", ec), end; !ec && it != end; it.increment(ec)) { + if (!it->is_regular_file(ec) || !cxx.contains(it->path().extension().string())) continue; + files.push_back(it->path()); + auto& [count, latest, size] = signature; + ++count; + latest = std::max(latest, it->last_write_time(ec)); + size += it->file_size(ec); + } + { + std::lock_guard lock(mutex); + if (auto it = kept.find(root); it != kept.end() && it->second.signature == signature) return it->second.types; + } + std::vector sources; + for (const auto& f : files) sources.push_back(read_text(f)); + auto types = mcpplibs::mtest::manifest::types_of(sources); + std::lock_guard lock(mutex); + kept[root] = Kept{signature, types}; + return types; } // The source of a test by its name: tests/.cpp (or .cc, .cxx). @@ -1495,6 +1556,53 @@ std::filesystem::path test_source(const std::filesystem::path& root, std::string } // namespace +std::vector Session::list_programs(const std::filesystem::path& root, const Json& p) { + mcpp::Client client; + { + std::lock_guard lock(mutex_); + client = client_; + } + std::vector programs; + if (!client.ready()) return programs; + mcpp::RunOptions o; + o.cwd = root; + o.args = {"test", "--list", "--message-format", "json"}; + for (const auto& a : target_args(p)) o.args.push_back(a); + o.deny = deny(); + o.client = client_label(); + const auto types = test_suite_types(root); + const auto definesMain = [&](const std::filesystem::path& f) { return scan::scan(read_text(f), {}, types).mainLine.has_value(); }; + if (auto r = client.records(o); r && r->exitCode == 0) + for (const auto& rec : r->records) + if (rec.is_object() && rec.contains("test") && rec["test"].is_string()) + programs.push_back(testing::program_of(rec, definesMain)); + std::lock_guard lock(mutex_); + programs_[root] = programs; + return programs; +} + +// The project's programs as last listed, listed now when they were not. +std::vector Session::programs_of(const std::filesystem::path& root) { + { + std::lock_guard lock(mutex_); + if (auto it = programs_.find(root); it != programs_.end()) return it->second; + } + return list_programs(root, Json::object()); +} + +std::optional Session::program_named(const std::filesystem::path& root, std::string_view name) { + for (auto& program : programs_of(root)) + if (program.name == name) return program; + return std::nullopt; +} + +// The program of several files a file belongs to (a file program is found by its name). +std::optional Session::program_of_file(const std::filesystem::path& root, const std::filesystem::path& file) { + for (auto& program : programs_of(root)) + if (program.layout == "directory" && !testing::part_of_file(program, file).empty()) return program; + return std::nullopt; +} + // What a C++ file is to its project (§6.3): a test's or a program's `main`, or neither, with the // line of `main` read from the text (approximate: an editor takes its language service's position // when it has one). Nothing is built and nothing run. @@ -1510,6 +1618,11 @@ std::expected Session::source_entries(const Json& p) { if (auto name = test_name_of(proj->root, file)) { out["kind"] = "test"; out["test"] = *name; + // A file of a program of several files: the program, and the part of it the file is. + if (auto program = program_of_file(proj->root, file)) { + out["test"] = program->name; + out["part"] = testing::part_of_file(*program, file); + } if (const auto* f = testing::framework_of(plan_packages(proj->root, false), found, extra)) out["framework"] = f->id; } else if (found.mainLine) { out["kind"] = "program"; @@ -1518,107 +1631,137 @@ std::expected Session::source_entries(const Json& p) { return out; } -// A test program's cases (§7.3): as its source names them, and -- when `list` is asked for, the -// workspace is trusted and the source calls its framework's macros -- as the program lists them. +// A test program's cases (§7.3): as its sources name them -- every file of a program of several +// (design 2026-11-10), each case with its file -- and, when `list` is asked for and the workspace is +// trusted, as the program lists them. std::expected Session::test_cases(const Json& p) { std::filesystem::path root = root_of(p); std::string test = p.value("test", ""); std::filesystem::path file(p.value("file", "")); - if (!file.empty()) { - if (root.empty()) if (auto proj = projects::project_for(file.generic_string(), "")) root = proj->root; - if (test.empty()) if (auto name = test_name_of(root, file)) test = *name; + if (!file.empty() && root.empty()) + if (auto proj = projects::project_for(file.generic_string(), "")) root = proj->root; + std::optional program; + if (!root.empty() && !file.empty() && test.empty()) { + program = program_of_file(root, file); + if (program) test = program->name; + else if (auto name = test_name_of(root, file)) test = *name; } if (root.empty() || test.empty()) return std::unexpected(invalid("test/cases names a project's test, or a test's file")); - if (file.empty()) file = test_source(root, test); - Json out{{"root", root.generic_string()}, {"test", test}, {"file", file.generic_string()}, {"framework", nullptr}, {"cases", Json::array()}}; - if (file.empty()) return out; - const auto extra = test_macros(); - const auto found = scan::scan(read_text(file), testing::macros(extra), test_suite_types(root)); - if (found.mainLine) out["main"] = Json{{"line", *found.mainLine}, {"approximate", true}}; + if (!program) program = program_named(root, test); + std::vector files; + if (program && !program->files.empty()) for (const auto& f : program->files) files.emplace_back(f); + else if (!file.empty()) files.push_back(file); + else if (auto source = test_source(root, test); !source.empty()) files.push_back(source); + const auto shown = program && !program->main.empty() ? std::filesystem::path(program->main) : files.empty() ? file : files.front(); + Json out{{"root", root.generic_string()}, {"test", test}, {"file", shown.generic_string()}, {"framework", nullptr}, {"cases", Json::array()}}; + if (files.empty()) return out; + if (program && program->layout == "directory") out["files"] = program->files; bool trusted; { std::lock_guard lock(mutex_); trusted = trusted_; } - const auto* f = testing::framework_of(plan_packages(root, trusted), found, extra); + const auto extra = test_macros(); + const auto types = test_suite_types(root); + const auto packages = plan_packages(root, trusted); + const testing::Framework* f = nullptr; std::vector cases; - if (f) { - out["framework"] = f->id; - cases = testing::static_cases(*f, found, extra); + for (const auto& one : files) { + const auto found = scan::scan(read_text(one), testing::macros(extra), types); + if (found.mainLine && one == shown) out["main"] = Json{{"line", *found.mainLine}, {"approximate", true}}; + const auto* framework = testing::framework_of(packages, found, extra); + if (!framework || (f && framework != f)) continue; + f = framework; + for (auto c : testing::static_cases(*f, found, extra)) { + c.file = one.generic_string(); + if (c.kind.empty() && f->protocol) c.kind = testing::kind_of_path(c.file); + cases.push_back(std::move(c)); + } } + if (f) out["framework"] = f->id; // A PROGRAM THAT IMPLEMENTS THE TEST PROTOCOL LISTS ITS OWN CASES (mcpp's `test-protocol`): // `mcpp test --cases`, whatever framework wrote it; its cases then run by `--case`. - if (p.value("list", false) && trusted && has_feature(root, "test-protocol")) { - mcpp::Client client; - { - std::lock_guard lock(mutex_); - client = client_; - } - if (client.ready()) { - mcpp::RunOptions o; - o.cwd = root; - o.args = {"test", "--message-format", "json"}; - for (const auto& a : target_args(p)) o.args.push_back(a); - o.args.push_back(test); - o.args.push_back("--cases"); - o.deny = deny(); - o.client = client_label(); - if (auto r = client.records(o)) - for (const auto& rec : r->records) { - if (!rec.is_object() || rec.value("test", "") != test || !rec.contains("listed")) continue; - const auto* protocol = testing::framework("mtest"); - if (!protocol || !rec["protocol"].is_number()) continue; - std::vector listed; - for (const auto& c : testing::protocol_listing(rec)) { - auto known = std::ranges::find_if(cases, [&](const testing::Case& s) { return s.name == c.id; }); - listed.push_back(testing::Case{c.id, known != cases.end() && known->line >= 0 ? known->line : c.line, "listed"}); - } - cases = std::move(listed); - out["framework"] = protocol->id; - out["listed"] = true; - f = protocol; - } + if (p.value("list", false) && trusted && has_feature(root, "test-protocol")) + if (auto listed = listed_cases(root, test, p, cases)) { + cases = std::move(*listed); + out["framework"] = "mtest"; + out["listed"] = true; + f = testing::framework("mtest"); } - } if (!f) return out; - if (out.value("listed", false)) { - for (const auto& c : cases) { - Json row{{"name", c.name}, {"source", c.source}}; - if (c.line >= 0) row["line"] = c.line; - out["cases"].push_back(row); + if (!out.value("listed", false) && p.value("list", false) && trusted && !f->protocol && files.size() == 1) + if (auto names = framework_listing(root, test, p, *f)) { + cases = testing::merge(cases, *names, *f); + out["listed"] = true; } - return out; + for (const auto& c : cases) out["cases"].push_back(testing::case_json(c)); + return out; +} + +// A protocol program's own listing (`mcpp test --cases`), each case placed where its source +// places it when the source was read, with what either states of its kind, tags and bounds. +std::optional> Session::listed_cases(const std::filesystem::path& root, const std::string& test, + const Json& p, const std::vector& statics) { + mcpp::Client client; + { + std::lock_guard lock(mutex_); + client = client_; } - if (p.value("list", false) && trusted && !f->protocol) { - mcpp::Client client; - { - std::lock_guard lock(mutex_); - client = client_; - } - if (client.ready()) { - mcpp::RunOptions o; - o.cwd = root; - o.args = {"test", "--message-format", "json"}; - for (const auto& a : target_args(p)) o.args.push_back(a); - o.args.push_back(test); - o.args.push_back("--"); - for (const auto& a : f->list) o.args.push_back(a); - o.deny = deny(); - o.client = client_label(); - if (auto r = client.records(o)) - for (const auto& rec : r->records) - if (rec.is_object() && rec.value("test", "") == test && rec.contains("run_output")) { - cases = testing::merge(cases, testing::listed(*f, rec.value("run_output", "")), *f); - out["listed"] = true; - } + if (!client.ready()) return std::nullopt; + mcpp::RunOptions o; + o.cwd = root; + o.args = {"test", "--message-format", "json"}; + for (const auto& a : target_args(p)) o.args.push_back(a); + o.args.push_back(test); + o.args.push_back("--cases"); + o.deny = deny(); + o.client = client_label(); + auto r = client.records(o); + if (!r) return std::nullopt; + for (const auto& rec : r->records) { + if (!rec.is_object() || rec.value("test", "") != test || !rec.contains("listed") || !rec["protocol"].is_number()) continue; + std::vector listed; + for (const auto& c : testing::protocol_listing(rec)) { + const auto known = std::ranges::find_if(statics, [&](const testing::Case& s) { return s.name == c.id; }); + const bool placed = known != statics.end() && known->line >= 0; + testing::Case one{c.id, placed ? known->line : c.line, "listed"}; + one.file = !c.file.empty() ? c.file : placed ? known->file : std::string{}; + one.kind = !c.kind.empty() ? c.kind : known != statics.end() ? known->kind : std::string{}; + one.tags = !c.tags.empty() || known == statics.end() ? c.tags : known->tags; + one.expect = c.expect; + one.bounds = !c.bounds.empty() || known == statics.end() ? c.bounds : known->bounds; + listed.push_back(std::move(one)); } + return listed; } - for (const auto& c : cases) { - Json row{{"name", c.name}, {"source", c.source}}; - if (c.line >= 0) row["line"] = c.line; - out["cases"].push_back(row); + return std::nullopt; +} + +// The names a framework's own list command prints (`mcpp test -- `), for a program that +// does not implement the protocol. +std::optional> Session::framework_listing(const std::filesystem::path& root, const std::string& test, + const Json& p, const testing::Framework& f) { + mcpp::Client client; + { + std::lock_guard lock(mutex_); + client = client_; } - return out; + if (!client.ready()) return std::nullopt; + mcpp::RunOptions o; + o.cwd = root; + o.args = {"test", "--message-format", "json"}; + for (const auto& a : target_args(p)) o.args.push_back(a); + o.args.push_back(test); + o.args.push_back("--"); + for (const auto& a : f.list) o.args.push_back(a); + o.deny = deny(); + o.client = client_label(); + auto r = client.records(o); + if (!r) return std::nullopt; + for (const auto& rec : r->records) + if (rec.is_object() && rec.value("test", "") == test && rec.contains("run_output")) + return testing::listed(f, rec.value("run_output", "")); + return std::nullopt; } // Runs some of a test program's cases (§7.3): `mcpp test -- `, the report @@ -1658,19 +1801,13 @@ std::expected Session::test_run_cases(const Json& p) { if (auto it = programOutput_.find(jobId); it != programOutput_.end()) programOutput = it->second; } if (fw && fw->protocol) { - Json record; + // Every record of the job: a program of several files reports one per part. + std::vector records; { std::lock_guard lock(mutex_); - if (auto it = testRecord_.find(jobId); it != testRecord_.end()) record = it->second; - } - bool truncated = false; - for (const auto& o : testing::protocol_outcomes(record)) { - auto output = views::bounded_output(o.outcome.status == "failed" ? std::string_view(o.outcome.message) : std::string_view{}, truncated); - rpc_.notify("mcpp-ide/test/caseResult", Json{{"jobId", jobId}, {"root", root.generic_string()}, {"test", test}, - {"case", o.outcome.name}, {"status", o.outcome.status}, {"durationMs", o.outcome.durationMs}, - {"message", o.outcome.message}, {"output", std::move(output)}, - {"outputTruncated", truncated}, {"location", o.location}}); + if (auto it = testRecords_.find(jobId); it != testRecords_.end()) records = it->second; } + for (const auto& record : records) protocol_case_results(jobId, root, test, record); } else if (fw) { const auto outputs = testing::case_outputs(*fw, programOutput); for (const auto& o : testing::read_report(*fw, text)) { @@ -1690,6 +1827,92 @@ std::expected Session::test_run_cases(const Json& p) { }); } +Json Session::test_result_of(const std::filesystem::path& root, const Json& record, Json n) { + std::vector programs; + { + std::lock_guard lock(mutex_); + if (auto it = programs_.find(root); it != programs_.end()) programs = it->second; + } + const auto name = record.value("test", ""); + const auto program = testing::owner_of(name, n.value("partOf", ""), programs); + n["program"] = program; + if (program != name) n["part"] = name; + if (record.contains("protocol") && record["protocol"].is_number()) n["protocol"] = record["protocol"]; + if (record.contains("cases") && record["cases"].is_array()) { + Json cases = Json::array(); + for (const auto& o : testing::protocol_outcomes(record)) cases.push_back(o.detail); + n["cases"] = std::move(cases); + } + if (record.contains("usage") && record["usage"].is_object()) n["usage"] = record["usage"]; + return n; +} + +// The cases of one record of a test/runCases job, each sent as `test/caseResult` with its program +// and part, a failure's message as its output. +void Session::protocol_case_results(std::string_view jobId, const std::filesystem::path& root, const std::string& test, + const Json& record) { + const auto part = record.value("test", ""); + for (const auto& o : testing::protocol_outcomes(record)) { + bool truncated = false; + auto output = views::bounded_output(o.outcome.status == "failed" ? std::string_view(o.outcome.message) : std::string_view{}, truncated); + Json n = o.detail; + n["jobId"] = jobId; + n["root"] = root.generic_string(); + n["test"] = test; + n["program"] = test; + if (part != test) n["part"] = part; + n["output"] = std::move(output); + n["outputTruncated"] = truncated; + rpc_.notify("mcpp-ide/test/caseResult", n); + } +} + +std::vector Session::store_runs(const std::filesystem::path& root, std::size_t count) { + const auto dir = teststore::store_dir(root); + std::map cache; + { + std::lock_guard lock(mutex_); + cache = storeRuns_[root]; + } + auto runs = teststore::last_runs(dir, count, cache); + { + std::lock_guard lock(mutex_); + storeRuns_[root] = std::move(cache); + } + const auto programs = programs_of(root); + teststore::attribute(runs, [&](std::string_view name) { return testing::owner_of(name, "", programs); }); + return runs; +} + +// The window of runs a request asks for: `runs` (20 by default, 50 at most). +static std::size_t window_of(const Json& p) { + const auto asked = p.contains("runs") && p["runs"].is_number_integer() ? p["runs"].get() : 20; + return static_cast(std::clamp(asked, 1, 50)); +} + +// A project's test runs as mcpp kept them (design 2026-11-10): each case's results across the last runs. +std::expected Session::test_history(const Json& p) { + const auto root = root_of(p); + if (root.empty()) return std::unexpected(invalid("test/history names its project's root")); + auto out = teststore::history(store_runs(root, window_of(p)), p.value("program", ""), p.value("case", "")); + out["root"] = root.generic_string(); + out["store"] = teststore::store_dir(root).generic_string(); + return out; +} + +// What the last runs say: the run, its failures, the slowest and the flaky cases, the metrics against +// their bounds, the schedule. A store mcpp has not written yet answers no run. +std::expected Session::test_insights(const Json& p) { + const auto root = root_of(p); + if (root.empty()) return std::unexpected(invalid("test/insights names its project's root")); + auto out = teststore::insights(store_runs(root, window_of(p))); + out["root"] = root.generic_string(); + out["store"] = teststore::store_dir(root).generic_string(); + out["features"] = Json{{"parts", has_feature(root, "test-parts")}, {"live", has_feature(root, "test-case-stream")}, + {"metrics", has_feature(root, "test-metrics")}}; + return out; +} + Session::PackOptions Session::pack_options(const std::filesystem::path& cwd) { mcpp::Client client; { @@ -1874,6 +2097,8 @@ void Session::install() { s.on("mcpp-ide/test/list", [this](const Json& p, const rpc::Context&) { return test_list(p); }); s.on("mcpp-ide/source/entries", [this](const Json& p, const rpc::Context&) { return source_entries(p); }); s.on("mcpp-ide/test/cases", [this](const Json& p, const rpc::Context&) { return test_cases(p); }); + s.on("mcpp-ide/test/history", [this](const Json& p, const rpc::Context&) { return test_history(p); }); + s.on("mcpp-ide/test/insights", [this](const Json& p, const rpc::Context&) { return test_insights(p); }); s.on("mcpp-ide/test/runCases", [this](const Json& p, const rpc::Context&) { return test_run_cases(p); }); s.on("mcpp-ide/project/mcpp", [this](const Json& p, const rpc::Context&) { return project_mcpp(p); }); s.on("mcpp-ide/project/refresh", [this](const Json& p, const rpc::Context&) { return project_refresh(p); }); diff --git a/src/spec.cppm b/src/spec.cppm index 3f6bd03..2654f59 100644 --- a/src/spec.cppm +++ b/src/spec.cppm @@ -122,6 +122,9 @@ const Json& test_frameworks() { } bool offered_in(const Json& c, std::string_view editor) { + // A deprecated id is VS Code's alone: its manifest keeps declaring it for the settings and + // keybindings of earlier releases, and no other editor declares it. + if (c.value("deprecated", false) && editor != "vscode") return false; if (c.contains("editors") && c["editors"].is_array()) return std::ranges::any_of(c["editors"], [&](const Json& e) { return e.is_string() && e.get() == editor; }); const auto kind = c.contains("action") && c["action"].is_object() ? c["action"].value("kind", "") : std::string{}; diff --git a/src/testing.cppm b/src/testing.cppm index b563e23..4e81729 100644 --- a/src/testing.cppm +++ b/src/testing.cppm @@ -52,8 +52,24 @@ struct Case { std::string name; // the framework's full name: Suite.Name, or the test's name int line = -1; // 0-based, in the program's source; -1 when only the program listed it std::string source; // "static" (its source) | "listed" (the program listed it) + // Of a program that implements the test protocol (mtest's manifest, or the program's listing): + // the file that declares it (a program of several files has one per case), its kind (`unit`, + // `e2e`, `cts`, `bench`; empty when neither it nor its directory states one), its tags, `fail` + // when it is expected to fail, and the bounds on its metrics ([{metric, max?, min?}]). + std::string file; + std::string kind; + std::vector tags; + std::string expect; + Json bounds = Json::array(); }; +// A case as `test/cases` answers it: {name, source, line?, file?, kind?, tags?, expect?, bounds?}. +Json case_json(const Case& c); + +// The kind of a case that states none, by mtest's convention of directories (mtest docs/layout.md): +// the first of `unit`, `e2e`, `cts`, `bench` below a `tests` directory in its path; empty otherwise. +std::string kind_of_path(std::string_view file); + // The cases the source names, in order. std::vector static_cases(const Framework& f, const scan::Result& found, const Json& extra); @@ -99,14 +115,61 @@ struct ProtocolCase { std::string id; std::string file; int line = -1; + std::string kind; + std::vector tags; + std::string expect; + Json bounds = Json::array(); }; std::vector protocol_listing(const Json& record); + +// A case's status as the editors know it (design 2026-11-10): `pass` passed, `fail` failed, `skip` +// skipped, `xfail` passed as an expected failure, `xpass` failed as an unexpected pass, `invalid` (a +// measurement that cannot be judged, protocol P6.3) errored. +struct CaseStatus { + std::string status; + bool expectedFailure = false; + bool unexpectedPass = false; +}; +CaseStatus case_status(std::string_view protocol); + +// One case of a record's `cases` (or of a `case` record that ended), as the editors receive it: +// {case, status, durationMs, reason, message, location, failures: [{path, line, message, expected?, +// actual?}], and when stated metrics, bounds, kind, tags, expectedFailure, unexpectedPass}. A line is +// 0-based; `message` is the first failure with its place, else the reason; `location` is the first +// failure's place, or null. +Json protocol_case(const Json& c); + struct ProtocolOutcome { Outcome outcome; Json location; // {path, line} or null + Json detail; // protocol_case }; std::vector protocol_outcomes(const Json& record); +// A TEST PROGRAM AS `mcpp test --list` NAMES IT (design 2026-11-10): `unit` for the directory +// tests/unit/ under `[test] programs = "directory"`, `00-a/0` for a file. Its own files, the file +// that defines `main` (none when no file does), and the parts a run divides it into, each a process +// named `/` (mcpp 2026.11.9.1 runs a protocol program of several files as one process +// per file). mcpp states them with `test-parts` (mcpp#917: `files`, `parts`, `main: null`); before +// it they are derived: a program whose `main` lies in a directory whose path ends in the program's +// name is that directory's sources. +struct Program { + std::string name; + std::string member; + std::string main; + std::vector files; + std::vector> parts; // {name, file} + std::string layout = "file"; // "file" | "directory" +}; +Program program_of(const Json& listRecord, const std::function& definesMain); +// {test, member, main, files, parts: [{name, file}], layout} +Json program_json(const Program& p); +// The program a record of a run belongs to: `partOf` when mcpp states it (mcpp#917), else the +// program of that name, else the longest whose name and a `/` begin it, else the name itself. +std::string owner_of(std::string_view name, std::string_view partOf, const std::vector& programs); +// The part of `program` a file is, or "". +std::string part_of_file(const Program& program, const std::filesystem::path& file); + } // namespace mcppide::testing namespace mcppide::testing { @@ -139,6 +202,28 @@ std::vector lines(std::string_view text) { return out; } +Json bound_json(std::string_view metric, std::optional max, std::optional min) { + Json b{{"metric", metric}}; + if (max) b["max"] = *max; + if (min) b["min"] = *min; + return b; +} + +// The bounds a case states (protocol P6.5: [{metric, max?, min?}]), the ones with a metric. +Json bounds_of(const Json& c) { + Json out = Json::array(); + for (const auto& b : c.value("bounds", Json::array())) + if (b.is_object() && b.contains("metric") && b["metric"].is_string()) out.push_back(b); + return out; +} + +// A number under either name (mcpp's records say `duration_ms`, the store `ms`), 0 when neither is one. +double number_of(const Json& c, const char* key, const char* other) { + for (const auto* k : {key, other}) + if (auto it = c.find(k); it != c.end() && it->is_number()) return it->get(); + return 0; +} + // The macros of `f`, with what `extra` adds for it. std::vector macros_of(const Framework& f, const Json& extra) { auto out = f.macros; @@ -327,7 +412,13 @@ const Framework* framework_of(const std::vector& packages, const sc std::vector static_cases(const Framework& f, const scan::Result& found, const Json& extra) { std::vector out; if (f.protocol) { - for (const auto& c : found.cases) out.push_back(Case{c.id, c.line, "static"}); + for (const auto& c : found.cases) { + Case one{c.id, c.line, "static"}; + one.kind = c.kind; + one.tags = c.tags; + for (const auto& b : c.bounds) one.bounds.push_back(bound_json(b.metric, b.max, b.min)); + out.push_back(std::move(one)); + } return out; } const auto names = macros_of(f, extra); @@ -504,8 +595,53 @@ std::vector protocol_listing(const Json& record) { for (const auto& c : record.value("listed", Json::array())) { if (!c.is_object()) continue; const int line = c.value("line", 0); - out.push_back({c.value("id", ""), c.value("file", ""), line > 0 ? line - 1 : -1}); + ProtocolCase one{c.value("id", ""), c.value("file", ""), line > 0 ? line - 1 : -1}; + one.kind = c.value("kind", ""); + one.tags = strings(c.value("tags", Json::array())); + one.expect = c.value("expect", ""); + one.bounds = bounds_of(c); + out.push_back(std::move(one)); + } + return out; +} + +CaseStatus case_status(std::string_view protocol) { + if (protocol == "pass") return {"passed"}; + if (protocol == "skip") return {"skipped"}; + if (protocol == "xfail") return {"passed", true, false}; + if (protocol == "xpass") return {"failed", false, true}; + if (protocol == "invalid") return {"errored"}; + return {"failed"}; +} + +Json protocol_case(const Json& c) { + const auto status = case_status(c.value("status", "")); + Json out{{"case", c.value("id", "")}, {"status", status.status}, + {"durationMs", static_cast(number_of(c, "duration_ms", "ms"))}, + {"reason", c.value("reason", "")}, {"location", nullptr}, {"failures", Json::array()}}; + if (status.expectedFailure) out["expectedFailure"] = true; + if (status.unexpectedPass) out["unexpectedPass"] = true; + for (const auto& f : c.value("failures", Json::array())) { + if (!f.is_object()) continue; + const int line = f.value("line", 0); + Json one{{"path", f.value("file", "")}, {"line", line > 0 ? line - 1 : 0}, {"message", f.value("message", "")}}; + for (const auto* key : {"expected", "actual"}) + if (f.contains(key) && f[key].is_string()) one[key] = f[key]; + out["failures"].push_back(std::move(one)); } + if (!out["failures"].empty()) { + const auto& f = out["failures"][0]; + out["location"] = Json{{"path", f["path"]}, {"line", f["line"]}}; + std::string message = std::format("{}:{}: {}", f.value("path", ""), f.value("line", 0) + 1, f.value("message", "")); + if (f.contains("expected")) message += std::format(" (expected {}, actual {})", f.value("expected", ""), f.value("actual", "")); + out["message"] = std::move(message); + } else { + out["message"] = out["reason"]; + } + if (auto m = c.find("metrics"); m != c.end() && m->is_array() && !m->empty()) out["metrics"] = *m; + if (auto b = bounds_of(c); !b.empty()) out["bounds"] = std::move(b); + if (auto k = c.value("kind", ""); !k.empty()) out["kind"] = k; + if (auto t = c.find("tags"); t != c.end() && t->is_array() && !t->empty()) out["tags"] = *t; return out; } @@ -514,27 +650,105 @@ std::vector protocol_outcomes(const Json& record) { for (const auto& c : record.value("cases", Json::array())) { if (!c.is_object()) continue; ProtocolOutcome o; - o.outcome.name = c.value("id", ""); - const auto status = c.value("status", ""); - o.outcome.status = status == "pass" || status == "xfail" ? "passed" : status == "skip" ? "skipped" : "failed"; - o.outcome.durationMs = static_cast(c.value("duration_ms", 0.0)); - o.location = nullptr; - const auto failures = c.value("failures", Json::array()); - if (!failures.empty() && failures[0].is_object()) { - const auto& f = failures[0]; - o.outcome.message = std::format("{}:{}: {}", f.value("file", ""), f.value("line", 0), f.value("message", "")); - if (f.contains("expected")) - o.outcome.message += std::format(" (expected {}, actual {})", f.value("expected", ""), f.value("actual", "")); - const int line = f.value("line", 0); - o.location = Json{{"path", f.value("file", "")}, {"line", line > 0 ? line - 1 : 0}}; - } else { - o.outcome.message = c.value("reason", ""); - } + o.detail = protocol_case(c); + o.outcome.name = o.detail.value("case", ""); + o.outcome.status = o.detail.value("status", "failed"); + o.outcome.durationMs = o.detail.value("durationMs", std::int64_t{0}); + o.outcome.message = o.detail.value("message", ""); + o.location = o.detail["location"]; out.push_back(std::move(o)); } return out; } +Json case_json(const Case& c) { + Json row{{"name", c.name}, {"source", c.source}}; + if (c.line >= 0) row["line"] = c.line; + if (!c.file.empty()) row["file"] = c.file; + if (!c.kind.empty()) row["kind"] = c.kind; + if (!c.tags.empty()) row["tags"] = c.tags; + if (!c.expect.empty()) row["expect"] = c.expect; + if (!c.bounds.empty()) row["bounds"] = c.bounds; + return row; +} + +std::string kind_of_path(std::string_view file) { + static constexpr std::array kinds{"unit", "e2e", "cts", "bench"}; + bool below = false; + for (const auto& part : std::filesystem::path(file)) { + const auto name = part.generic_string(); + if (below) return std::ranges::find(kinds, name) != kinds.end() ? name : std::string{}; + below = name == "tests"; + } + return {}; +} + +Program program_of(const Json& r, const std::function& definesMain) { + Program p; + p.name = r.value("test", ""); + p.member = r.value("member", ""); + p.main = r.contains("main") && r["main"].is_string() ? r["main"].get() : std::string{}; + if (r.contains("files") && r["files"].is_array()) { + // mcpp states them (mcpp#917, `test-parts`). + p.files = strings(r["files"]); + for (const auto& part : r.value("parts", Json::array())) + if (part.is_object()) p.parts.emplace_back(part.value("name", ""), part.value("file", "")); + p.layout = p.files.size() > 1 || !p.parts.empty() ? "directory" : "file"; + return p; + } + if (p.main.empty()) return p; + const std::filesystem::path main(p.main); + const auto dir = main.parent_path(); + if (!dir.generic_string().ends_with("/" + p.name)) { + p.files = {p.main}; + return p; + } + // A program of a directory: its sources, as mcpp 2026.11.9.1 forms it (docs/08). + std::vector files; + std::error_code ec; + for (std::filesystem::directory_iterator it(dir, ec), end; !ec && it != end; it.increment(ec)) { + const auto ext = it->path().extension().string(); + if ((ext == ".cpp" || ext == ".cc" || ext == ".cxx") && it->is_regular_file(ec)) files.push_back(it->path()); + } + std::ranges::sort(files); + p.layout = "directory"; + p.main.clear(); + for (const auto& f : files) { + p.files.push_back(f.generic_string()); + p.parts.emplace_back(p.name + "/" + f.stem().generic_string(), f.generic_string()); + if (p.main.empty() && definesMain && definesMain(f)) p.main = f.generic_string(); + } + return p; +} + +Json program_json(const Program& p) { + Json parts = Json::array(); + for (const auto& [name, file] : p.parts) parts.push_back(Json{{"name", name}, {"file", file}}); + return Json{{"test", p.name}, {"member", p.member}, {"main", p.main}, {"files", p.files}, {"parts", parts}, {"layout", p.layout}}; +} + +std::string owner_of(std::string_view name, std::string_view partOf, const std::vector& programs) { + if (!partOf.empty()) return std::string(partOf); + std::string best; + for (const auto& p : programs) { + if (p.name == name) return p.name; + if (name.size() > p.name.size() && name.starts_with(p.name) && name[p.name.size()] == '/' && p.name.size() > best.size()) + best = p.name; + } + return best.empty() ? std::string(name) : best; +} + +std::string part_of_file(const Program& program, const std::filesystem::path& file) { + const auto wanted = file.lexically_normal().generic_string(); + for (const auto& [name, f] : program.parts) + if (std::filesystem::path(f).lexically_normal().generic_string() == wanted) return name; + // Another spelling of the same file (Windows' short names, a link): the file system decides. + std::error_code ec; + for (const auto& [name, f] : program.parts) + if (std::filesystem::equivalent(f, file, ec)) return name; + return {}; +} + std::vector read_report(const Framework& f, std::string_view text) { if (f.reportFormat == "junit") return junit_report(text); const Json doc = Json::parse(text, nullptr, false); diff --git a/src/teststore.cppm b/src/teststore.cppm new file mode 100644 index 0000000..440ec87 --- /dev/null +++ b/src/teststore.cppm @@ -0,0 +1,442 @@ +// The runs of a project's tests, as mcpp keeps them (design 2026-11-10): mcpp 2026.11.9.1 writes +// each `mcpp test` run into the store mtest specifies (mtest spec P12), under `target/mtest` or +// `[test] store`. The backend reads that store; it runs nothing. A store that is absent is no error: +// the history is empty and says so. +// +// What the store holds depends on the mcpp that wrote it. 2026.11.9.1 writes each process of a +// program (its start, duration, worker and build) and each case (its status, duration, reason and +// failures); with `test-metrics` (mcpp#917) a case has its kind, tags, start, metrics and bounds as +// well. Every field is read when present, and a view that needs an absent one is left out. +export module mcppide.teststore; + +import std; +import nlohmann.json; + +export namespace mcppide::teststore { + +using Json = nlohmann::json; + +struct ProgramRun { + std::string name; // the program, once attributed (attribute); the process's own name before + std::string part; // the part of the program the process ran, when it is one + std::string status; // pass | fail | build-fail | not-run | built + double ms = 0; + double startedMs = 0; + int worker = 0; + double buildMs = 0; +}; + +struct CaseRun { + std::string program; + std::string part; + std::string id; + std::string kind; + std::string status; // pass | fail | skip | xfail | xpass | invalid + double ms = 0; + std::optional startedMs; + std::string reason; + Json failures = Json::array(); + Json metrics = Json::array(); + Json bounds = Json::array(); + Json tags = Json::array(); +}; + +struct Run { + std::string id; + std::string started; + std::string status; + double ms = 0; + int jobs = 0; + Json counts = Json::object(); + std::vector programs; + std::vector cases; +}; + +// The directory of a project's store: `[test] store` of its manifest (a path from the project's +// root), else target/mtest. +std::filesystem::path store_dir(const std::filesystem::path& root); + +// The last `count` runs of the store at `dir`, oldest first. Runs already read are kept in `cache` +// (by id), so a store read again reads only its new runs. +void attribute(std::vector& runs, const std::function& owner) { + for (auto& r : runs) { + for (auto& p : r.programs) { + auto program = owner(p.name); + if (program != p.name) p.part = std::exchange(p.name, std::move(program)); + } + for (auto& c : r.cases) { + auto program = owner(c.program); + if (program != c.program) c.part = std::exchange(c.program, std::move(program)); + } + } +} + +std::vector last_runs(const std::filesystem::path& dir, std::size_t count, std::map& cache); + +// One run's files read (exposed for the tests). +Run read_run(const std::filesystem::path& runDir, const Json& indexRecord); + +// The programs the store's processes and cases belong to: a process of a program of several files is +// named by its part (`unit/a`); `owner` names its program (`unit`), and the part is kept beside it. +void attribute(std::vector& runs, const std::function& owner); + +// The runs of the window, and each case's results across them (the program's, or the one case). +// {runs: [{id, started, status, ms, counts}], cases: [{program, case, kind, results: [{run, status, +// ms, metrics}]}]} +Json history(const std::vector& runs, std::string_view program, std::string_view caseId); + +// What the window says, for an editor's view of the tests (design 2026-11-10): +// {run, byKind, failures, slowest, flaky, metrics, schedule, available: {kinds, metrics, cases}}. +Json insights(const std::vector& runs); + +} // namespace mcppide::teststore + +namespace mcppide::teststore { + +namespace { + +std::string read_file(const std::filesystem::path& p) { + std::ifstream in(p, std::ios::binary); + return std::string(std::istreambuf_iterator(in), {}); +} + +std::vector ndjson(const std::filesystem::path& p) { + std::vector out; + std::ifstream in(p, std::ios::binary); + for (std::string line; in && std::getline(in, line);) { + if (!line.empty() && line.back() == '\r') line.pop_back(); + if (line.empty()) continue; + auto j = Json::parse(line, nullptr, false); + if (j.is_object()) out.push_back(std::move(j)); + } + return out; +} + +double number(const Json& j, const char* key) { + if (auto it = j.find(key); it != j.end() && it->is_number()) return it->get(); + return 0; +} + +Json array(const Json& j, const char* key) { + if (auto it = j.find(key); it != j.end() && it->is_array()) return *it; + return Json::array(); +} + +bool failing(std::string_view status) { + return status == "fail" || status == "xpass" || status == "invalid" || status == "build-fail"; +} + +std::string key_of(const CaseRun& c) { return c.program + '\x1f' + c.id; } + +double median(std::vector v) { + if (v.empty()) return 0; + std::ranges::sort(v); + const auto n = v.size(); + return n % 2 ? v[n / 2] : (v[n / 2 - 1] + v[n / 2]) / 2; +} + +// The bound of `metric` among a case's bounds, as {max?, min?}, or null. +Json bound_of(const Json& bounds, std::string_view metric) { + for (const auto& b : bounds) + if (b.is_object() && b.value("metric", "") == metric) { + Json out = Json::object(); + if (b.contains("max") && b["max"].is_number()) out["max"] = b["max"]; + if (b.contains("min") && b["min"].is_number()) out["min"] = b["min"]; + return out; + } + return nullptr; +} + +// How far the last value is from its bound, as a share of the bound (negative: past it), or null. +Json margin_of(const Json& bound, double last) { + if (!bound.is_object()) return nullptr; + if (bound.contains("max")) { + const double max = bound["max"].get(); + return max != 0 ? Json((max - last) / std::abs(max)) : Json(nullptr); + } + if (bound.contains("min")) { + const double min = bound["min"].get(); + return min != 0 ? Json((last - min) / std::abs(min)) : Json(nullptr); + } + return nullptr; +} + +// `program`, and `part` when the row is of a part of it. +void name_row(Json& row, const std::string& program, const std::string& part) { + row["program"] = program; + if (!part.empty() && part != program) row["part"] = part; +} + +Json run_brief(const Run& r) { + return Json{{"id", r.id}, {"started", r.started}, {"status", r.status}, {"ms", r.ms}, {"jobs", r.jobs}, {"counts", r.counts}}; +} + +Json failures_of(const std::vector& runs) { + Json out = Json::array(); + const auto& last = runs.back(); + for (const auto& c : last.cases) { + if (!failing(c.status)) continue; + // The run since which it has failed: walk back while it did. + std::string since = last.id; + for (auto it = runs.rbegin() + 1; it != runs.rend(); ++it) { + const auto found = std::ranges::find_if(it->cases, [&](const CaseRun& o) { return key_of(o) == key_of(c); }); + if (found == it->cases.end() || !failing(found->status)) break; + since = it->id; + } + Json f{{"case", c.id}, {"status", c.status}, {"reason", c.reason}, {"since", since}, {"new", since == last.id}}; + name_row(f, c.program, c.part); + if (!c.failures.empty() && c.failures[0].is_object()) { + const auto& first = c.failures[0]; + f["message"] = first.value("message", ""); + f["path"] = first.value("file", ""); + const int line = first.value("line", 0); + f["line"] = line > 0 ? line - 1 : 0; + for (const auto* key : {"expected", "actual"}) + if (first.contains(key) && first[key].is_string()) f[key] = first[key]; + } + out.push_back(std::move(f)); + } + // A program that failed without a case reporting it (it did not compile, or speaks no protocol). + for (const auto& p : last.programs) { + if (!failing(p.status)) continue; + const auto process = p.part.empty() ? p.name : p.part; + const bool reported = std::ranges::any_of(last.cases, [&](const CaseRun& c) { + return (c.part.empty() ? c.program : c.part) == process && failing(c.status); + }); + if (reported) continue; + Json f{{"case", ""}, {"status", p.status}, {"since", last.id}, {"new", true}}; + name_row(f, p.name, p.part); + out.push_back(std::move(f)); + } + return out; +} + +Json slowest_of(const std::vector& runs, std::size_t limit) { + std::map> durations; + for (const auto& r : runs) + for (const auto& c : r.cases) durations[key_of(c)].push_back(c.ms); + auto cases = runs.back().cases; + std::ranges::sort(cases, std::greater{}, &CaseRun::ms); + Json out = Json::array(); + for (const auto& c : cases | std::views::take(limit)) { + Json row{{"case", c.id}, {"ms", c.ms}, {"medianMs", median(durations[key_of(c)])}, {"runs", durations[key_of(c)].size()}}; + name_row(row, c.program, c.part); + out.push_back(std::move(row)); + } + return out; +} + +Json flaky_of(const std::vector& runs) { + struct Seen { std::string program, part, id, last; int passed = 0, failed = 0, flips = 0; }; + std::map seen; + for (const auto& r : runs) + for (const auto& c : r.cases) { + if (c.status != "pass" && !failing(c.status)) continue; + auto& s = seen[key_of(c)]; + s.program = c.program; + s.part = c.part; + s.id = c.id; + const auto now = failing(c.status) ? "fail" : "pass"; + if (!s.last.empty() && s.last != now) ++s.flips; + s.last = now; + ++(failing(c.status) ? s.failed : s.passed); + } + Json out = Json::array(); + for (const auto& [key, s] : seen) + if (s.passed > 0 && s.failed > 0) { + Json row{{"case", s.id}, {"passed", s.passed}, {"failed", s.failed}, {"flips", s.flips}}; + name_row(row, s.program, s.part); + out.push_back(std::move(row)); + } + std::stable_sort(out.begin(), out.end(), [](const Json& a, const Json& b) { return a.value("flips", 0) > b.value("flips", 0); }); + return out; +} + +struct Series { + std::string program, part, id, name, unit, better; + Json bounds = Json::array(); + Json values = Json::array(); +}; + +// A case's metrics of one run added to their series (kept in the order they were first seen). +void add_metrics(std::map& series, std::vector& order, const Run& r, const CaseRun& c) { + for (const auto& m : c.metrics) { + if (!m.is_object() || !m.contains("value") || !m["value"].is_number()) continue; + const auto name = m.value("name", ""); + const auto key = key_of(c) + '\x1f' + name; + auto [it, fresh] = series.try_emplace(key); + if (fresh) order.push_back(key); + auto& s = it->second; + s.program = c.program; + s.part = c.part; + s.id = c.id; + s.name = name; + s.unit = m.value("unit", s.unit); + s.better = m.value("better", s.better); + if (!c.bounds.empty()) s.bounds = c.bounds; + s.values.push_back(Json{{"run", r.id}, {"value", m["value"]}}); + } +} + +Json metrics_of(const std::vector& runs) { + std::map series; + std::vector order; + for (const auto& r : runs) + for (const auto& c : r.cases) add_metrics(series, order, r, c); + Json out = Json::array(); + for (const auto& key : order) { + const auto& s = series[key]; + const double last = s.values.back()["value"].get(); + const auto bound = bound_of(s.bounds, s.name); + Json row{{"case", s.id}, {"name", s.name}, {"unit", s.unit}, {"values", s.values}, + {"last", last}, {"bound", bound}, {"margin", margin_of(bound, last)}}; + name_row(row, s.program, s.part); + if (!s.better.empty()) row["better"] = s.better; + out.push_back(std::move(row)); + } + return out; +} + +// The schedule of the last run: a lane per worker, each process with its start and duration, and the +// critical path -- the processes of the lane that ends last (a worker's processes run one after +// another, P12.3), the build the run waited for before them. +Json schedule_of(const Run& r) { + std::map lanes; + std::map ends; + double build = 0; + for (const auto& p : r.programs) { + Json item{{"startMs", p.startedMs}, {"ms", p.ms}, {"status", p.status}, {"buildMs", p.buildMs}}; + name_row(item, p.name, p.part); + lanes[p.worker].push_back(std::move(item)); + ends[p.worker] = std::max(ends[p.worker], p.startedMs + p.ms); + build = std::max(build, p.buildMs); + } + Json out{{"lanes", Json::array()}, {"criticalPath", Json::array()}, {"ms", r.ms}, {"buildMs", build}}; + int longest = -1; + for (const auto& [worker, end] : ends) + if (longest < 0 || end > ends[longest]) longest = worker; + for (auto& [worker, items] : lanes) { + std::stable_sort(items.begin(), items.end(), [](const Json& a, const Json& b) { return a.value("startMs", 0.0) < b.value("startMs", 0.0); }); + out["lanes"].push_back(Json{{"worker", worker}, {"items", items}}); + if (worker == longest) + for (const auto& i : items) out["criticalPath"].push_back(i["program"]); + } + return out; +} + +} // namespace + +std::filesystem::path store_dir(const std::filesystem::path& root) { + // `[test] store = ""`: read from the manifest's text (its section and key, as mcpp writes them). + std::istringstream text(read_file(root / "mcpp.toml")); + bool inTest = false; + for (std::string line; std::getline(text, line);) { + const auto first = line.find_first_not_of(" \t"); + if (first == std::string::npos || line[first] == '#') continue; + if (line[first] == '[') { + inTest = line.compare(first, 6, "[test]") == 0; + continue; + } + if (!inTest || line.compare(first, 5, "store") != 0) continue; + const auto open = line.find('"'), close = line.rfind('"'); + if (open != std::string::npos && close > open) return root / line.substr(open + 1, close - open - 1); + } + return root / "target" / "mtest"; +} + +Run read_run(const std::filesystem::path& runDir, const Json& indexRecord) { + Run r; + const auto runJson = Json::parse(read_file(runDir / "run.json"), nullptr, false); + const auto& rec = runJson.is_object() ? runJson : indexRecord; + r.id = rec.value("id", indexRecord.value("id", "")); + r.started = rec.value("started", ""); + r.status = rec.value("status", ""); + r.ms = number(rec, "ms"); + r.jobs = static_cast(number(rec, "jobs")); + if (rec.contains("counts") && rec["counts"].is_object()) r.counts = rec["counts"]; + for (const auto& p : ndjson(runDir / "programs.ndjson")) + r.programs.push_back(ProgramRun{.name = p.value("name", ""), .status = p.value("status", ""), .ms = number(p, "ms"), + .startedMs = number(p, "started_ms"), .worker = static_cast(number(p, "worker")), + .buildMs = number(p, "build_ms")}); + for (const auto& c : ndjson(runDir / "cases.ndjson")) { + CaseRun one{.program = c.value("program", ""), .id = c.value("id", ""), .kind = c.value("kind", ""), + .status = c.value("status", ""), .ms = number(c, "ms")}; + if (auto it = c.find("started_ms"); it != c.end() && it->is_number()) one.startedMs = it->get(); + one.reason = c.value("reason", ""); + one.failures = array(c, "failures"); + one.metrics = array(c, "metrics"); + one.bounds = array(c, "bounds"); + one.tags = array(c, "tags"); + r.cases.push_back(std::move(one)); + } + return r; +} + +std::vector last_runs(const std::filesystem::path& dir, std::size_t count, std::map& cache) { + std::vector index = ndjson(dir / "index.ndjson"); + // An identifier sorts by time (P12.2). + std::ranges::sort(index, {}, [](const Json& j) { return j.value("id", ""); }); + const auto from = index.size() > count ? index.size() - count : 0; + std::vector out; + for (std::size_t i = from; i < index.size(); ++i) { + const auto id = index[i].value("id", ""); + if (id.empty()) continue; + auto it = cache.find(id); + if (it == cache.end()) { + std::error_code ec; + if (!std::filesystem::is_directory(dir / "runs" / id, ec)) continue; + it = cache.emplace(id, read_run(dir / "runs" / id, index[i])).first; + } + out.push_back(it->second); + } + return out; +} + +Json history(const std::vector& runs, std::string_view program, std::string_view caseId) { + Json out{{"runs", Json::array()}, {"cases", Json::array()}}; + std::map byCase; + std::vector order; + for (const auto& r : runs) { + out["runs"].push_back(run_brief(r)); + for (const auto& c : r.cases) { + if (!program.empty() && c.program != program) continue; + if (!caseId.empty() && c.id != caseId) continue; + Json row{{"case", c.id}, {"kind", c.kind}, {"results", Json::array()}}; + name_row(row, c.program, ""); + auto [it, fresh] = byCase.try_emplace(key_of(c), std::move(row)); + if (fresh) order.push_back(key_of(c)); + Json result{{"run", r.id}, {"status", c.status}, {"ms", c.ms}, {"metrics", c.metrics}}; + if (!c.part.empty()) result["part"] = c.part; + it->second["results"].push_back(std::move(result)); + } + } + for (const auto& k : order) out["cases"].push_back(std::move(byCase[k])); + return out; +} + +Json insights(const std::vector& runs) { + Json out{{"runs", runs.size()}, {"run", nullptr}, {"byKind", Json::object()}, {"failures", Json::array()}, + {"slowest", Json::array()}, {"flaky", Json::array()}, {"metrics", Json::array()}, {"schedule", nullptr}, + {"available", Json{{"kinds", false}, {"metrics", false}, {"cases", false}}}}; + if (runs.empty()) return out; + const auto& last = runs.back(); + out["run"] = run_brief(last); + for (const auto& c : last.cases) { + auto& k = out["byKind"][c.kind]; + if (!k.is_object()) k = Json::object(); + k[c.status] = k.value(c.status, 0) + 1; + } + out["available"]["cases"] = !last.cases.empty(); + out["available"]["kinds"] = std::ranges::any_of(last.cases, [](const CaseRun& c) { return !c.kind.empty(); }); + out["available"]["metrics"] = std::ranges::any_of(runs, [](const Run& r) { + return std::ranges::any_of(r.cases, [](const CaseRun& c) { return !c.metrics.empty(); }); + }); + out["failures"] = failures_of(runs); + out["slowest"] = slowest_of(runs, 10); + out["flaky"] = flaky_of(runs); + out["metrics"] = metrics_of(runs); + out["schedule"] = schedule_of(last); + return out; +} + +} // namespace mcppide::teststore diff --git a/src/toml.cppm b/src/toml.cppm index 812b73d..407da02 100644 --- a/src/toml.cppm +++ b/src/toml.cppm @@ -10,7 +10,7 @@ // the line (mcppide.text turns it into the editor's UTF-16 column). // // A port of mcpp-vscode 0.9's src/toml/parser.ts (design 2026-10-30 part 2 §8); the goldens in -// tests/rules are that implementation's answers. +// tests/cts/rules are that implementation's answers. export module mcppide.toml; import std; diff --git a/src/views.cppm b/src/views.cppm index 02f7acd..8b9e5de 100644 --- a/src/views.cppm +++ b/src/views.cppm @@ -57,8 +57,8 @@ Json targets(const Json& toolchainListEnvelope, std::string_view selected, bool std::string platform_text(const Json& platform); // One record of a `--message-format json` stream (a build's events, a test's records), as the -// adapters consume it: {"type": "plan" | "progress" | "diagnostic" | "artifact" | "test" | "summary" | -// "other", ...}. +// adapters consume it: {"type": "plan" | "progress" | "diagnostic" | "artifact" | "test" | "case" | +// "summary" | "other", ...}. // // A TEST'S STATUS IS mcpp's, VALUE FOR VALUE (mcpp docs/50, the test record; design 2026-11-07, D16): // `pass` is passed; `run_fail` failed, or timedOut when `timed_out`; `compile_fail` failed, with @@ -539,12 +539,16 @@ Json stream_record(const Json& r) { {"failed", s.value("failed", 0)}, {"notRun", s.value("not_run", 0)}, {"notRunReason", s.value("not_run_reason", "")}, {"elapsedMs", s.value("elapsed_ms", 0)}}; } + // A case as it begins or ends, written while its program runs (mcpp#917, `test-case-stream`). + if (r.contains("case") && r["case"].is_object()) return Json{{"type", "case"}, {"case", r["case"]}}; if (r.contains("test") && r["test"].is_string()) { const auto raw = r.value("status", ""); const bool timedOut = r.value("timed_out", false); std::string status; std::string reason = r.value("reason", ""); Json out{{"type", "test"}, {"member", r.value("member", "")}, {"test", r["test"]}}; + // The program a part's record belongs to, when mcpp states it (mcpp#917, `test-parts`). + if (r.contains("part_of") && r["part_of"].is_string()) out["partOf"] = r["part_of"]; if (raw == "pass") status = "passed"; else if (raw == "run_fail") status = timedOut ? "timedOut" : "failed"; else if (raw == "compile_fail") { status = "failed"; out["compileFailed"] = true; } diff --git a/tests/bench/backend.cpp b/tests/bench/backend.cpp new file mode 100644 index 0000000..1669871 --- /dev/null +++ b/tests/bench/backend.cpp @@ -0,0 +1,96 @@ +// What a person waits for in an editor, measured where the backend spends it: the rules layer's +// answer to a keystroke in a large manifest, the reading of a large test file for its entries and +// cases (`source/entries`, `test/cases`), and the insights over a full window of stored runs +// (`test/insights`). Each bound is a median per call, set well above what a CI runner measures, so a +// bound that fails names a regression rather than a slow machine. +import std; +import mcppide.testsupport; +import mcpplibs.mtest; +import mcpplibs.mtest.bench; +import nlohmann.json; +import mcppide.documents; +import mcppide.scan; +import mcppide.teststore; + +using namespace mcpplibs::mtest; + +using namespace mcppide; +using Json = nlohmann::json; + +namespace { + +// A manifest of `n` dependencies, as a large workspace member's. +documents::Document manifest(int n) { + documents::Document d; + d.uri = "file:///bench/mcpp.toml"; + d.path = "/bench/mcpp.toml"; + d.kind = documents::Kind::manifest; + d.lines = {"[package]", "name = \"bench\"", "version = \"0.1.0\"", "", "[dependencies]"}; + for (int i = 0; i < n; ++i) d.lines.push_back(std::format("dep{} = \"1.{}.0\"", i, i % 10)); + d.lines.push_back(""); + d.lines.push_back("[build]"); + d.lines.push_back("sources = [\"src/*.cppm\"]"); + return d; +} + +// A test file of `n` cases in suites of ten, as a large unit test file's. +std::string test_file(int n) { + std::string text = "import std;\nimport mcpplibs.mtest;\nusing namespace mcpplibs::mtest;\n\n"; + for (int s = 0; s < n / 10; ++s) { + text += std::format("UTest suite{0}{{\"suite {0}\", [] {{\n", s); + for (int c = 0; c < 10; ++c) + text += std::format(" \"case {0}\"_test = [] {{ expect(eq({0} + 1, {1})); }}; // \"not a case\"\n", c, c + 1); + text += "}};\n\n"; + } + return text; +} + +// A window of `runs` stored runs of `cases` cases each, every tenth failing in turn and every case +// reporting one metric against its bound. +std::vector window(int runs, int cases) { + std::vector out; + for (int r = 0; r < runs; ++r) { + teststore::Run run; + run.id = std::format("20261110T{:06}-0000", r); + run.status = "fail"; + for (int p = 0; p < 8; ++p) + run.programs.push_back(teststore::ProgramRun{.name = std::format("unit/f{}", p), .status = "pass", .ms = 100.0 + p, + .startedMs = 10.0 * p, .worker = p % 4}); + for (int c = 0; c < cases; ++c) { + teststore::CaseRun one{.program = std::format("unit/f{}", c % 8), .id = std::format("case {}", c), .kind = "unit", + .status = (c + r) % 10 == 0 ? "fail" : "pass", .ms = 1.0 + c % 7}; + one.metrics = Json::array({Json{{"name", "ready"}, {"value", 1.0 + (c + r) % 5}, {"unit", "s"}}}); + one.bounds = Json::array({Json{{"metric", "ready"}, {"max", 15}}}); + run.cases.push_back(std::move(one)); + } + out.push_back(std::move(run)); + } + return out; +} + +} // namespace + +Bench keystroke{"a keystroke in a manifest of 400 dependencies", at_most{"manifest.keystroke", 0.05}, [] { + const auto d = manifest(400); + const documents::Options o; + const int last = static_cast(d.lines.size()) - 1; + bench::measure("manifest.keystroke", [&] { + bool needCatalogue = false; + bench::keep(documents::diagnostics(d, o)); + bench::keep(documents::completion(d, last, 10, o, nullptr, needCatalogue)); + }); +}}; + +Bench testFile{"the entries and cases of a test file of 2000 cases", at_most{"tests.scan", 0.1}, [] { + const auto text = test_file(2000); + bench::measure("tests.scan", [&] { + const auto found = scan::scan(text, {}); + testsupport::check(found.cases.size() == 2200); + bench::keep(found); + }, {.samples = 10}); +}}; + +Bench insights{"the insights over 50 runs of 400 cases", at_most{"tests.insights", 0.5}, [] { + const auto runs = window(50, 400); + bench::measure("tests.insights", [&] { bench::keep(teststore::insights(runs)); }, {.samples = 10}); +}}; diff --git a/tests/cts/contracts.cpp b/tests/cts/contracts.cpp new file mode 100644 index 0000000..89c83a6 --- /dev/null +++ b/tests/cts/contracts.cpp @@ -0,0 +1,82 @@ +// The two contracts, held to their schemas (design 2026-10-30 part 2 §2): every golden fixture +// (mcpp's real output, and the shapes mcpp#917 proposes) against the definition of +// protocol/mcpp-contract.schema.json that protocol/fixtures.json names for it; the methods the backend +// registers and the notifications it sends against protocol/protocol.schema.json, both ways. +import std; +import mcppide.testsupport; +import mcpplibs.mtest; +import nlohmann.json; +import mcppide.server; +import mcppide.schema; +import mcppide.spec; + +using testsupport::check; +using namespace mcpplibs::mtest; + +using namespace mcppide; +using Json = nlohmann::json; + +namespace { + +Json load(const std::filesystem::path& p) { return Json::parse(testsupport::read(p)); } + +bool conforms(const Json& value, const Json& schema, const Json& root, std::string_view what) { + const auto problems = schema::validate(value, schema, root); + for (const auto& p : problems) std::println(std::cerr, "{}: {}", what, p); + return problems.empty(); +} + +} // namespace + +CTS contracts{"contracts", [] { + "mcpp's output conforms to the contract"_test = [] { + const auto contract = load(testsupport::repo() / "protocol" / "mcpp-contract.schema.json"); + const auto index = load(testsupport::repo() / "protocol" / "fixtures.json"); + for (const auto& f : index["fixtures"]) { + const auto path = testsupport::fixtures() / f["file"].get(); + if (!std::filesystem::exists(path)) { + check(f.value("optional", false)); + continue; + } + const auto text = testsupport::read(path); + if (f.contains("each")) { + const Json def{{"$ref", "#/$defs/" + f["each"].get()}}; + std::istringstream lines(text); + int n = 0; + for (std::string line; std::getline(lines, line);) { + if (line.empty()) continue; + check(conforms(Json::parse(line), def, contract, std::format("{}:{}", f["file"].get(), ++n))); + } + check(n > 0); + } else { + const Json def{{"$ref", "#/$defs/" + f["schema"].get()}}; + check(conforms(Json::parse(text), def, contract, f["file"].get())); + } + } + }; + "the backend registers what its protocol states"_test = [] { + const auto protocol = load(testsupport::repo() / "protocol" / "protocol.schema.json"); + std::set methods, notifications; + for (auto it = protocol["methods"].begin(); it != protocol["methods"].end(); ++it) methods.insert(it.key()); + for (auto it = protocol["notifications"].begin(); it != protocol["notifications"].end(); ++it) notifications.insert(it.key()); + std::set registered(server::kMethods.begin(), server::kMethods.end()); + std::set sent(server::kNotifications.begin(), server::kNotifications.end()); + for (const auto& m : methods) if (!registered.contains(m)) std::println(std::cerr, "schema method not registered: {}", m); + for (const auto& m : registered) if (!methods.contains(m)) std::println(std::cerr, "registered method not in the schema: {}", m); + check(methods == registered); + check(notifications == sent); + }; + // An editor is offered the commands it declares: a deprecated id stays VS Code's (its manifest keeps + // declaring it), and CLion, Neovim and an agent are never offered one, whatever mcppls forwards. + "a deprecated command is offered to VS Code alone"_test = [] { + const auto& table = spec::commands(); + int deprecated = 0; + for (const auto& c : table["commands"]) { + if (!c.value("deprecated", false)) continue; + ++deprecated; + check(spec::offered_in(c, "vscode")); + for (const auto* editor : {"jetbrains", "nvim", "cli"}) check(!spec::offered_in(c, editor)); + } + check(deprecated > 0); + }; +}}; diff --git a/tests/test_rules.cpp b/tests/cts/rules.cpp similarity index 97% rename from tests/test_rules.cpp rename to tests/cts/rules.cpp index 9e8fd28..63111a1 100644 --- a/tests/test_rules.cpp +++ b/tests/cts/rules.cpp @@ -1,10 +1,11 @@ // The rules layer answers what mcpp-vscode 0.9's TypeScript answered (design 2026-10-30 part 2 §8): -// over each file of tests/rules/corpus, at every position of every line (and past them), the +// over each file of tests/cts/rules/corpus, at every position of every line (and past them), the // scanner's structure and cursor contexts, the completions, hovers, jumps, version contexts and -// diagnostics equal tests/rules/golden, which that implementation wrote (with the documents of its +// diagnostics equal tests/cts/rules/golden, which that implementation wrote (with the documents of its // parser tests, golden/snippets.json). A difference is printed with its file and position. import std; import mcppide.testsupport; +import mcpplibs.mtest; import nlohmann.json; import mcppide.text; import mcppide.manifest; @@ -12,6 +13,7 @@ import mcppide.buildscript; import mcppide.toml; using testsupport::check; +using namespace mcpplibs::mtest; using namespace mcppide; using Json = nlohmann::json; @@ -276,8 +278,8 @@ void check_catalogue(const Json& golden) { } // namespace -int main() { - const auto root = testsupport::repo() / "tests" / "rules"; +CTS rules{"rules", [] { + const auto root = testsupport::repo() / "tests" / "cts" / "rules"; std::vector files; for (const auto& e : std::filesystem::directory_iterator(root / "corpus")) files.push_back(e.path()); std::ranges::sort(files); @@ -312,6 +314,4 @@ int main() { if (failures) std::println(std::cerr, "{} mismatch(es)", failures); check(failures == 0); - std::println("rules: the goldens hold"); - return 0; -} +}}; diff --git a/tests/rules/corpus/brackets.toml b/tests/cts/rules/corpus/brackets.toml similarity index 100% rename from tests/rules/corpus/brackets.toml rename to tests/cts/rules/corpus/brackets.toml diff --git a/tests/rules/corpus/broken.toml b/tests/cts/rules/corpus/broken.toml similarity index 100% rename from tests/rules/corpus/broken.toml rename to tests/cts/rules/corpus/broken.toml diff --git a/tests/rules/corpus/comments.toml b/tests/cts/rules/corpus/comments.toml similarity index 100% rename from tests/rules/corpus/comments.toml rename to tests/cts/rules/corpus/comments.toml diff --git a/tests/rules/corpus/complete.toml b/tests/cts/rules/corpus/complete.toml similarity index 100% rename from tests/rules/corpus/complete.toml rename to tests/cts/rules/corpus/complete.toml diff --git a/tests/rules/corpus/crlf.toml b/tests/cts/rules/corpus/crlf.toml similarity index 100% rename from tests/rules/corpus/crlf.toml rename to tests/cts/rules/corpus/crlf.toml diff --git a/tests/rules/corpus/editing.mcpp b/tests/cts/rules/corpus/editing.mcpp similarity index 100% rename from tests/rules/corpus/editing.mcpp rename to tests/cts/rules/corpus/editing.mcpp diff --git a/tests/rules/corpus/editing.toml b/tests/cts/rules/corpus/editing.toml similarity index 100% rename from tests/rules/corpus/editing.toml rename to tests/cts/rules/corpus/editing.toml diff --git a/tests/rules/corpus/empty.toml b/tests/cts/rules/corpus/empty.toml similarity index 100% rename from tests/rules/corpus/empty.toml rename to tests/cts/rules/corpus/empty.toml diff --git a/tests/rules/corpus/plain.mcpp b/tests/cts/rules/corpus/plain.mcpp similarity index 100% rename from tests/rules/corpus/plain.mcpp rename to tests/cts/rules/corpus/plain.mcpp diff --git a/tests/rules/corpus/rules.mcpp b/tests/cts/rules/corpus/rules.mcpp similarity index 100% rename from tests/rules/corpus/rules.mcpp rename to tests/cts/rules/corpus/rules.mcpp diff --git a/tests/rules/corpus/unclosed.toml b/tests/cts/rules/corpus/unclosed.toml similarity index 100% rename from tests/rules/corpus/unclosed.toml rename to tests/cts/rules/corpus/unclosed.toml diff --git a/tests/rules/golden/brackets.toml.json b/tests/cts/rules/golden/brackets.toml.json similarity index 100% rename from tests/rules/golden/brackets.toml.json rename to tests/cts/rules/golden/brackets.toml.json diff --git a/tests/rules/golden/broken.toml.json b/tests/cts/rules/golden/broken.toml.json similarity index 100% rename from tests/rules/golden/broken.toml.json rename to tests/cts/rules/golden/broken.toml.json diff --git a/tests/rules/golden/catalogue.json b/tests/cts/rules/golden/catalogue.json similarity index 100% rename from tests/rules/golden/catalogue.json rename to tests/cts/rules/golden/catalogue.json diff --git a/tests/rules/golden/comments.toml.json b/tests/cts/rules/golden/comments.toml.json similarity index 100% rename from tests/rules/golden/comments.toml.json rename to tests/cts/rules/golden/comments.toml.json diff --git a/tests/rules/golden/complete.toml.json b/tests/cts/rules/golden/complete.toml.json similarity index 100% rename from tests/rules/golden/complete.toml.json rename to tests/cts/rules/golden/complete.toml.json diff --git a/tests/rules/golden/crlf.toml.json b/tests/cts/rules/golden/crlf.toml.json similarity index 100% rename from tests/rules/golden/crlf.toml.json rename to tests/cts/rules/golden/crlf.toml.json diff --git a/tests/rules/golden/editing.mcpp.json b/tests/cts/rules/golden/editing.mcpp.json similarity index 100% rename from tests/rules/golden/editing.mcpp.json rename to tests/cts/rules/golden/editing.mcpp.json diff --git a/tests/rules/golden/editing.toml.json b/tests/cts/rules/golden/editing.toml.json similarity index 100% rename from tests/rules/golden/editing.toml.json rename to tests/cts/rules/golden/editing.toml.json diff --git a/tests/rules/golden/empty.toml.json b/tests/cts/rules/golden/empty.toml.json similarity index 100% rename from tests/rules/golden/empty.toml.json rename to tests/cts/rules/golden/empty.toml.json diff --git a/tests/rules/golden/plain.mcpp.json b/tests/cts/rules/golden/plain.mcpp.json similarity index 100% rename from tests/rules/golden/plain.mcpp.json rename to tests/cts/rules/golden/plain.mcpp.json diff --git a/tests/rules/golden/rules.mcpp.json b/tests/cts/rules/golden/rules.mcpp.json similarity index 100% rename from tests/rules/golden/rules.mcpp.json rename to tests/cts/rules/golden/rules.mcpp.json diff --git a/tests/rules/golden/snippets.json b/tests/cts/rules/golden/snippets.json similarity index 100% rename from tests/rules/golden/snippets.json rename to tests/cts/rules/golden/snippets.json diff --git a/tests/rules/golden/unclosed.toml.json b/tests/cts/rules/golden/unclosed.toml.json similarity index 100% rename from tests/rules/golden/unclosed.toml.json rename to tests/cts/rules/golden/unclosed.toml.json diff --git a/tests/test_jobs.cpp b/tests/e2e/jobs.cpp similarity index 97% rename from tests/test_jobs.cpp rename to tests/e2e/jobs.cpp index eec4e03..dfe04b6 100644 --- a/tests/test_jobs.cpp +++ b/tests/e2e/jobs.cpp @@ -2,12 +2,14 @@ // `failed` with its diagnostics, and a cancelled one ends `cancelled` with mcpp's record saying so. import std; import mcppide.testsupport; +import mcpplibs.mtest; import nlohmann.json; import mcpplibs.os; import mcppide.mcpp; import mcppide.jobs; using testsupport::check; +using namespace mcpplibs::mtest; namespace jobs = mcppide::jobs; using Json = nlohmann::json; @@ -21,7 +23,7 @@ struct Seen { } // namespace -int main() { +E2E jobQueue{"jobs", [] { const auto mock = testsupport::built("mcpp-ide-mock-mcpp"); check(!mock.empty()); // The mock reads its fixtures and its behaviour from the environment of each run. @@ -82,5 +84,4 @@ int main() { std::error_code ec; std::filesystem::remove_all(root, ec); - std::println("test_jobs: ok"); -} +}}; diff --git a/tests/test_languageservice.cpp b/tests/e2e/languageservice.cpp similarity index 92% rename from tests/test_languageservice.cpp rename to tests/e2e/languageservice.cpp index e83426a..c7e51ef 100644 --- a/tests/test_languageservice.cpp +++ b/tests/e2e/languageservice.cpp @@ -4,14 +4,16 @@ // (mcppls 0.0.18 and later), or a program. import std; import mcppide.testsupport; +import mcpplibs.mtest; import mcppide.languageservice; using testsupport::check; +using namespace mcpplibs::mtest; using namespace mcppide; namespace ls = mcppide::languageservice; -int main() { +E2E languageservice{"languageservice", [] { check(ls::version_in("0.0.16\n") == "0.0.16"); check(ls::version_in("mcppls 0.0.18\n") == "0.0.18"); check(ls::version_in("mcppls 0.0.19,\n") == "0.0.19"); @@ -19,6 +21,9 @@ int main() { check(ls::classify("/x/mcppls", "PATH", "0.0.16\n").state == ls::State::outdated); check(ls::classify("/x/mcppls", "PATH", "mcppls 0.0.18\n").state == ls::State::ok); check(ls::classify("/x/mcppls", "PATH", "0.1.0").state == ls::State::ok); + // mcppls 0.21 numbers a series and a patch (0.0.21 is released as 0.21.0): a version like any other. + check(ls::version_in("mcppls 0.21.0\n") == "0.21.0"); + check(ls::classify("/x/mcppls", "PATH", "mcppls 0.21.0\n").state == ls::State::ok); check(ls::classify("/x/mcppls", "PATH", "usage: mcppls").state == ls::State::unreadable); // A candidate that does not exist is passed over; with nothing found the state is `missing` or // whatever the machine's own mcppls answers -- never an error. @@ -76,11 +81,11 @@ int main() { check(ls::payload_command(payload) == imaged); write(payload / "payload.json", R"({"payload-version": 6, "server": {"path": "bin/mcppls.okb"}})"); check(ls::payload_server(payload).problem.contains("payload-version 6")); - // The manifests of the published payloads (tests/payloads: mcppls 0.0.19's for linux-x64, version 4, + // The manifests of the published payloads (tests/e2e/payloads: mcppls 0.0.19's for linux-x64, version 4, // and 0.0.20's, version 5, as mcppls's S6 examples hold them, their file lists cut short). - { + "the published payloads"_test = [&] { const auto published = [&](std::string_view name) { - std::ifstream in(testsupport::repo() / "tests" / "payloads" / std::string(name), std::ios::binary); + std::ifstream in(testsupport::repo() / "tests" / "e2e" / "payloads" / std::string(name), std::ios::binary); return std::string(std::istreambuf_iterator(in), {}); }; const auto served = testsupport::temp() / std::format("mcppide-published-{}", std::random_device{}()); @@ -104,7 +109,7 @@ int main() { check(ls::payload_server(served, "linux-x64").command == std::vector({served / "bin" / "okrun", served / "bin" / "mcppls.okb"})); check(ls::payload_server(served, "darwin-arm64").problem.contains("serves linux-x64")); std::filesystem::remove_all(served); - } + }; // A refused payload named by the adapter is said, when nothing else is found. write(payload / "payload.json", R"({"payload-version": 5, "server": {"path": "bin/mcppls.okb"}, "platforms": {"plan9-x64": {"runner": "bin/okrun"}}})"); const auto refused = ls::candidates({payload.string()}); @@ -114,5 +119,4 @@ int main() { const auto found = ls::candidates({payload.string()}); check(!found.empty() && found.front().source == "editor" && found.front().command.back() == payload / "bin" / "mcppls"); std::filesystem::remove_all(payload); - std::println("test_languageservice: ok"); -} +}}; diff --git a/tests/test_mcppls_status.cpp b/tests/e2e/mcppls_status.cpp similarity index 98% rename from tests/test_mcppls_status.cpp rename to tests/e2e/mcppls_status.cpp index ec6cb35..5aff639 100644 --- a/tests/test_mcppls_status.cpp +++ b/tests/e2e/mcppls_status.cpp @@ -4,6 +4,7 @@ // mcppls sits in a payload made here, as an editor's own payload would. import std; import mcppide.testsupport; +import mcpplibs.mtest; import nlohmann.json; import mcppide.rpc; import mcppide.server; @@ -11,6 +12,7 @@ import mcppide.languageservice; import mcpplibs.os; using testsupport::check; +using namespace mcpplibs::mtest; using namespace mcppide; using Json = nlohmann::json; @@ -55,9 +57,9 @@ Json command(const Json& list, std::string_view id) { } // namespace -int main() { +E2E mcpplsStatus{"mcppls status", [] { // The document, read: what `status` printed, and the frozen table when it printed nothing usable. - { + "the document read"_test = [&] { const auto r = ls::parse_status(R"({"schema": 1, "capabilities": ["status", 7], "workspace": {"state": "ready"}, "actions": [{"id": "restart", "vscode": "mcppls.restartServer"}, {"title": "no id"}]})"); check(r.source == "status"); @@ -66,7 +68,7 @@ int main() { const auto broken = ls::parse_status("mcppls: unknown command status"); check(broken.source == "frozen" && !broken.detail.empty()); check(broken.actions == ls::frozen_actions()); - } + }; const auto mock = testsupport::built("mcpp-ide-mock-mcppls"); check(!mock.empty()); @@ -136,6 +138,4 @@ int main() { std::error_code ignored; std::filesystem::remove_all(base, ignored); - std::println("test_mcppls_status: ok"); - return 0; -} +}}; diff --git a/tests/payloads/mcppls-0.0.19-linux-x64.json b/tests/e2e/payloads/mcppls-0.0.19-linux-x64.json similarity index 100% rename from tests/payloads/mcppls-0.0.19-linux-x64.json rename to tests/e2e/payloads/mcppls-0.0.19-linux-x64.json diff --git a/tests/payloads/mcppls-0.0.20.json b/tests/e2e/payloads/mcppls-0.0.20.json similarity index 100% rename from tests/payloads/mcppls-0.0.20.json rename to tests/e2e/payloads/mcppls-0.0.20.json diff --git a/tests/test_probe.cpp b/tests/e2e/probe.cpp similarity index 98% rename from tests/test_probe.cpp rename to tests/e2e/probe.cpp index 6e862a1..637b761 100644 --- a/tests/test_probe.cpp +++ b/tests/e2e/probe.cpp @@ -5,12 +5,14 @@ // version, after which the project is asked again. The mock mcpp plays the shim per directory. import std; import mcppide.testsupport; +import mcpplibs.mtest; import nlohmann.json; import mcppide.rpc; import mcppide.server; import mcppide.mcpp; using testsupport::check; +using namespace mcpplibs::mtest; using namespace mcppide; using Json = nlohmann::json; @@ -59,7 +61,7 @@ std::filesystem::path project(const std::filesystem::path& base, std::string_vie } // namespace -int main() { +E2E probe{"probe", [] { const auto mock = testsupport::built("mcpp-ide-mock-mcpp"); check(!mock.empty()); const auto base = testsupport::temp() / std::format("mcppide-probe-{}", std::chrono::steady_clock::now().time_since_epoch().count()); @@ -115,11 +117,11 @@ int main() { // editor started from another project's terminal inherits (mcpp docs/50). The removal is the // spawn's own instruction; a variable set in this process now would reach no child, since a // program's environment is the one it was started with. - { + "no run carries another project's pin"_test = [&] { mcpp::Client client(mock, mcpp::Protocol{}); const auto o = client.spawn_options(mcpp::RunOptions{}); check(std::ranges::any_of(o.env, [](const auto& kv) { return kv.first == "XLINGS_PROJECT_DIR" && !kv.second; })); - } + }; // A job's envelope names another version: the project is asked again, and the editor told. std::ofstream(old / ".mock-mcpp-version") << "2026.10.31.1\n"; @@ -147,6 +149,4 @@ int main() { std::error_code ignored; std::filesystem::remove_all(base, ignored); - std::println("test_probe: ok"); - return 0; -} +}}; diff --git a/tests/test_protocol.cpp b/tests/e2e/protocol.cpp similarity index 73% rename from tests/test_protocol.cpp rename to tests/e2e/protocol.cpp index 1d54c72..d2b3009 100644 --- a/tests/test_protocol.cpp +++ b/tests/e2e/protocol.cpp @@ -1,18 +1,16 @@ -// The two contracts, held to their schemas: -// - every golden fixture (mcpp's real output) against the definition of -// protocol/mcpp-contract.schema.json that protocol/fixtures.json names for it; -// - the methods the backend registers and the notifications it sends against -// protocol/protocol.schema.json, both ways; -// - a session over the mock mcpp: each answer against its method's `result`, and -// every notification a build and a test send against its `params`. +// A session over the mock mcpp, held to the backend's protocol (protocol/protocol.schema.json): each +// answer against its method's `result`, and every notification a build and a test send against its +// `params`. (The schemas themselves are held to mcpp's output and to the backend in cts/contracts.cpp.) import std; import mcppide.testsupport; +import mcpplibs.mtest; import nlohmann.json; import mcppide.rpc; import mcppide.server; import mcppide.schema; using testsupport::check; +using namespace mcpplibs::mtest; using namespace mcppide; using Json = nlohmann::json; @@ -62,41 +60,8 @@ struct Wire { } // namespace -int main() { - const auto contract = load(testsupport::repo() / "protocol" / "mcpp-contract.schema.json"); - const auto index = load(testsupport::repo() / "protocol" / "fixtures.json"); - for (const auto& f : index["fixtures"]) { - const auto path = testsupport::fixtures() / f["file"].get(); - if (!std::filesystem::exists(path)) { - check(f.value("optional", false)); - continue; - } - const auto text = testsupport::read(path); - if (f.contains("each")) { - const Json def{{"$ref", "#/$defs/" + f["each"].get()}}; - std::istringstream lines(text); - int n = 0; - for (std::string line; std::getline(lines, line);) { - if (line.empty()) continue; - check(conforms(Json::parse(line), def, contract, std::format("{}:{}", f["file"].get(), ++n))); - } - check(n > 0); - } else { - const Json def{{"$ref", "#/$defs/" + f["schema"].get()}}; - check(conforms(Json::parse(text), def, contract, f["file"].get())); - } - } - +E2E protocol{"protocol", [] { const auto protocol = load(testsupport::repo() / "protocol" / "protocol.schema.json"); - std::set methods, notifications; - for (auto it = protocol["methods"].begin(); it != protocol["methods"].end(); ++it) methods.insert(it.key()); - for (auto it = protocol["notifications"].begin(); it != protocol["notifications"].end(); ++it) notifications.insert(it.key()); - std::set registered(server::kMethods.begin(), server::kMethods.end()); - std::set sent(server::kNotifications.begin(), server::kNotifications.end()); - for (const auto& m : methods) if (!registered.contains(m)) std::println(std::cerr, "schema method not registered: {}", m); - for (const auto& m : registered) if (!methods.contains(m)) std::println(std::cerr, "registered method not in the schema: {}", m); - check(methods == registered); - check(notifications == sent); // A session over the mock mcpp. const auto mock = testsupport::built("mcpp-ide-mock-mcpp"); @@ -393,8 +358,95 @@ int main() { for (const auto& n : caseResults) check(conforms(n, protocol["notifications"]["mcpp-ide/test/caseResult"]["params"], protocol, "mcpp-ide/test/caseResult")); } + // A program of several files (design 2026-11-10): as mcpp 2026.11.9.1 states it (its parts + // derived), and as mcpp#917 states it (parts, case records, metrics). A run's part records are + // attributed to their program, each part's cases reported, and the store's runs read back. + for (const std::string set : {"directory", "directory-917"}) { + const bool rich = set == "directory-917"; + const auto dir = root.parent_path() / (root.filename().string() + "-" + set); + std::filesystem::create_directories(dir / "tests" / "unit"); + std::ofstream(dir / "mcpp.toml") << "[package]\nname = \"app\"\nversion = \"0.1.0\"\n\n[test]\nprograms = \"directory\"\n"; + std::ofstream(dir / ".mock-mcpp-tests") << set; + std::ofstream(dir / ".mock-mcpp-features") << (rich ? "test-protocol,test-parts,test-case-stream,test-metrics" : "test-protocol"); + std::ofstream(dir / "tests" / "smoke.cpp") << "int main() { return 0; }\n"; + std::ofstream(dir / "tests" / "unit" / "a.cpp") + << "import mcpplibs.mtest;\nusing namespace mcpplibs::mtest;\n\nBench alpha{\"alpha\", at_most{\"ready\", 15}, [] {}};\n"; + std::ofstream(dir / "tests" / "unit" / "b.cpp") << "import mcpplibs.mtest;\nusing namespace mcpplibs::mtest;\n\nUTest beta{\"beta\", [] {}};\n"; + const auto a = (dir / "tests" / "unit" / "a.cpp").generic_string(), b = (dir / "tests" / "unit" / "b.cpp").generic_string(); + // mcpp names a file as the file system spells the project (Windows' long names), the test as + // its temporary directory does (it may be a short name): one file, either spelling. + const auto same = [](const Json& named, const std::string& file) { + std::error_code ec; + return named.is_string() && std::filesystem::equivalent(named.get(), file, ec); + }; + const auto listed = result_of("mcpp-ide/test/list", Json{{"root", dir.generic_string()}}); + const auto unit = std::ranges::find_if(listed["tests"], [](const Json& t) { return t["test"] == "unit"; }); + check(unit != listed["tests"].end()); + if (unit != listed["tests"].end()) { + check((*unit)["layout"] == "directory" && (*unit)["main"] == "" && (*unit)["files"].size() == 2 + && same((*unit)["files"][0], a) && same((*unit)["files"][1], b)); + check((*unit)["parts"].size() == 2 && (*unit)["parts"][1]["name"] == "unit/b" && same((*unit)["parts"][1]["file"], b)); + } + const auto entries = result_of("mcpp-ide/source/entries", Json{{"file", b}}); + check(entries["kind"] == "test" && entries["test"] == "unit" && entries["part"] == "unit/b"); + // Before a listing, the program's files (its cases are read from them where its plan holds + // mtest; the mock's plan does not); the listing places each case in its file. + const auto statics = result_of("mcpp-ide/test/cases", Json{{"root", dir.generic_string()}, {"test", "unit"}}); + check(statics["test"] == "unit" && statics["files"].size() == 2 && same(statics["files"][0], a) && same(statics["files"][1], b)); + const auto listedCases = result_of("mcpp-ide/test/cases", Json{{"root", dir.generic_string()}, {"test", "unit"}, {"list", true}}); + check(listedCases.value("listed", false) && listedCases["framework"] == "mtest" && listedCases["cases"].size() == 2); + if (listedCases["cases"].size() == 2) { + const auto& alpha = listedCases["cases"][0]; + check(alpha["name"] == "alpha" && same(alpha["file"], a) && alpha["line"] == 3 && alpha["kind"] == "bench" && alpha["bounds"][0]["max"] == 15); + check(listedCases["cases"][1]["tags"] == Json::array({"slow"}) && same(listedCases["cases"][1]["file"], b)); + } + const auto resultsBefore = wire.notifications("mcpp-ide/test/result").size(); + const auto eventsBefore = wire.notifications("mcpp-ide/test/caseEvent").size(); + const auto finishedBefore = wire.notifications("mcpp-ide/job/finished").size(); + result_of("mcpp-ide/job/start", Json{{"kind", "test"}, {"root", dir.generic_string()}}); + for (int i = 0; i < 2000 && wire.notifications("mcpp-ide/job/finished").size() <= finishedBefore; ++i) + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + const auto results = wire.notifications("mcpp-ide/test/result"); + check(results.size() == resultsBefore + 3); + if (results.size() == resultsBefore + 3) { + const auto& partB = results[resultsBefore + 1]; + check(results[resultsBefore]["program"] == "unit" && results[resultsBefore]["part"] == "unit/a"); + check(partB["program"] == "unit" && partB["part"] == "unit/b" && partB["cases"].size() == 1); + check(partB["cases"][0]["status"] == "failed" && partB["cases"][0]["failures"][0]["expected"] == "4" + && partB["cases"][0]["failures"][0]["actual"] == "5" && partB["cases"][0]["location"]["line"] == 3); + check(results[resultsBefore + 2]["program"] == "smoke" && !results[resultsBefore + 2].contains("part")); + if (rich) check(results[resultsBefore]["cases"][0]["metrics"][0]["value"] == 4.2); + } + const auto events = wire.notifications("mcpp-ide/test/caseEvent"); + check(events.size() == eventsBefore + (rich ? 4 : 0)); + if (rich && events.size() == eventsBefore + 4) { + check(events[eventsBefore]["event"] == "begin" && events[eventsBefore]["program"] == "unit" && events[eventsBefore]["case"] == "alpha"); + check(events[eventsBefore + 1]["event"] == "end" && events[eventsBefore + 1]["status"] == "passed" + && events[eventsBefore + 1]["metrics"][0]["name"] == "ready"); + check(events[eventsBefore + 3]["status"] == "failed" && events[eventsBefore + 3]["part"] == "unit/b"); + } + for (const auto* name : {"mcpp-ide/test/result", "mcpp-ide/test/caseEvent"}) + for (const auto& n : wire.notifications(name)) check(conforms(n, protocol["notifications"][name]["params"], protocol, name)); + // The store mcpp keeps (target/mtest): three runs, the last with mcpp#917's fields. + const auto noRuns = result_of("mcpp-ide/test/insights", Json{{"root", dir.generic_string()}}); + check(noRuns["run"].is_null() && noRuns["runs"] == 0 && noRuns["features"]["live"] == rich); + std::filesystem::create_directories(dir / "target" / "mtest"); + std::filesystem::copy(testsupport::repo() / "protocol" / "fixtures" / "mtest-store", dir / "target" / "mtest", + std::filesystem::copy_options::recursive); + const auto insights = result_of("mcpp-ide/test/insights", Json{{"root", dir.generic_string()}}); + check(insights["runs"] == 3 && insights["run"]["id"] == "20261110T100000-0c03"); + check(insights["failures"].size() == 1 && insights["failures"][0]["case"] == "beta" && insights["failures"][0]["program"] == "unit" + && insights["failures"][0]["part"] == "unit/b" && insights["failures"][0]["new"] == true && insights["failures"][0]["expected"] == "4"); + check(insights["flaky"].size() == 1 && insights["flaky"][0]["flips"] == 2); + check(insights["metrics"].size() == 1 && insights["metrics"][0]["last"] == 4.2 && insights["metrics"][0]["bound"]["max"] == 15); + check(insights["schedule"]["lanes"].size() == 2 && insights["schedule"]["criticalPath"] == Json::array({"unit", "smoke"})); + check(insights["available"]["metrics"] == true && insights["available"]["kinds"] == true); + const auto history = result_of("mcpp-ide/test/history", Json{{"root", dir.generic_string()}, {"program", "unit"}, {"case", "beta"}}); + check(history["runs"].size() == 3 && history["cases"].size() == 1 && history["cases"][0]["results"].size() == 3); + if (history["cases"].size() == 1) check(history["cases"][0]["results"][1]["status"] == "pass"); + std::filesystem::remove_all(dir); + } std::filesystem::remove_all(protocolRoot); std::filesystem::remove_all(streamed); std::filesystem::remove_all(root); - std::println("test_protocol: ok"); -} +}}; diff --git a/tests/test_policy.cpp b/tests/unit/policy.cpp similarity index 93% rename from tests/test_policy.cpp rename to tests/unit/policy.cpp index 78aace4..dd16c3f 100644 --- a/tests/test_policy.cpp +++ b/tests/unit/policy.cpp @@ -2,14 +2,16 @@ // needing a preview or an acknowledgement is not confirmed without it. import std; import mcppide.testsupport; +import mcpplibs.mtest; import nlohmann.json; import mcppide.policy; using testsupport::check; +using namespace mcpplibs::mtest; using namespace mcppide; -int main() { +UTest policy{"policy", [] { policy::CleanOptions o; for (auto a : policy::kActions) check(policy::plan(a, o).has_value()); check(!policy::plan("everything", o)); @@ -25,5 +27,4 @@ int main() { check(!policy::confirmed(all, nlohmann::json{{"confirmed", true}, {"previewShown", true}})); check(policy::confirmed(all, nlohmann::json{{"confirmed", true}, {"previewShown", true}, {"acknowledged", true}})); check(policy::confirmed(*policy::plan("cacheVerify", o), nullptr)); - std::println("test_policy: ok"); -} +}}; diff --git a/tests/test_rpc.cpp b/tests/unit/rpc.cpp similarity index 93% rename from tests/test_rpc.cpp rename to tests/unit/rpc.cpp index b24b58b..d670b82 100644 --- a/tests/test_rpc.cpp +++ b/tests/unit/rpc.cpp @@ -3,16 +3,18 @@ // METHOD_NOT_FOUND, a cancelled request answers REQUEST_CANCELLED. import std; import mcppide.testsupport; +import mcpplibs.mtest; import nlohmann.json; import mcppide.rpc; using testsupport::check; +using namespace mcpplibs::mtest; using namespace mcppide; using Json = nlohmann::json; -int main() { - { +UTest rpc{"rpc", [] { + "frames reassemble"_test = [&] { rpc::FrameReader r; const auto a = rpc::encode(Json{{"jsonrpc", "2.0"}, {"id", 1}, {"method", "x"}}); const auto b = rpc::encode(Json{{"jsonrpc", "2.0"}, {"id", 2}, {"method", "y"}}); @@ -23,8 +25,8 @@ int main() { auto second = r.next(); check(second && Json::parse(*second)["id"] == 2); check(!r.next()); - } - { + }; + "the server answers and refuses"_test = [&] { std::mutex m; std::vector out; rpc::Server s([&](std::string_view frame) { @@ -58,6 +60,5 @@ int main() { check(!cancelled); // the handler answered (it saw the flag) with a result, which is kept s.dispatch("not json"); check(out.back()["error"]["code"] == rpc::PARSE_ERROR); - } - std::println("test_rpc: ok"); -} + }; +}}; diff --git a/tests/test_settings.cpp b/tests/unit/settings.cpp similarity index 93% rename from tests/test_settings.cpp rename to tests/unit/settings.cpp index 0b15db3..013ee2f 100644 --- a/tests/test_settings.cpp +++ b/tests/unit/settings.cpp @@ -1,16 +1,18 @@ // The registry: defaults, rejections, renames, presets, and the runtime strings. import std; import mcppide.testsupport; +import mcpplibs.mtest; import nlohmann.json; import mcppide.settings; import mcppide.i18n; using testsupport::check; +using namespace mcpplibs::mtest; using namespace mcppide; using Json = nlohmann::json; -int main() { +UTest settings{"settings", [] { auto r = settings::resolve(Json{{"mcpp.build.profile", "fast"}, {"mcpp.cache.topN", 3}, {"mcpp.task.buildArgs", Json::array({1})}}); check(r["values"]["mcpp.build.profile"] == "dev"); check(r["values"]["mcpp.cache.topN"] == 3 || r["values"].contains("mcpp.views.cache.topN")); @@ -26,5 +28,4 @@ int main() { check(i18n::normalize_locale("zh_CN") == "zh-cn" && i18n::normalize_locale("de") == "en"); check(i18n::translate("{0} and {1}", "en", {"a", "b"}) == "a and b"); check(i18n::translate("Actions", "zh-cn") == "操作"); - std::println("test_settings: ok"); -} +}}; diff --git a/tests/test_testing.cpp b/tests/unit/testing.cpp similarity index 96% rename from tests/test_testing.cpp rename to tests/unit/testing.cpp index 9ff6898..11f153e 100644 --- a/tests/test_testing.cpp +++ b/tests/unit/testing.cpp @@ -5,6 +5,7 @@ // backend's source/entries and test/cases over a project on disk. import std; import mcppide.testsupport; +import mcpplibs.mtest; import nlohmann.json; import mcppide.scan; import mcpplibs.mtest.manifest; @@ -13,6 +14,7 @@ import mcppide.rpc; import mcppide.server; using testsupport::check; +using namespace mcpplibs::mtest; using namespace mcppide; using Json = nlohmann::json; @@ -27,9 +29,9 @@ const testing::Framework& fw(std::string_view id) { } // namespace -int main() { +UTest sources{"sources", [] { // main: the first `int main(` or `auto main(` at file scope, outside comments and literals. - { + "main is found outside comments and literals"_test = [&] { const auto r = scan::scan(R"(// int main() in a comment /* int main() */ const char* s = "int main("; @@ -43,10 +45,10 @@ main(int argc, char** argv) { return 0; } check(r.mainLine && *r.mainLine == 8); check(!scan::scan("void f() { obj.main(); std::main(); }", {}).mainLine); check(scan::scan("export auto main() -> int { return 0; }", {}).mainLine == 0); - } + }; // Macro calls at the start of a declaration, with their arguments. const auto macros = testing::macros(Json::object()); - { + "macro calls and their arguments"_test = [&] { const auto r = scan::scan(R"(#include TEST(Math, Adds) { EXPECT_EQ(1 + 1, 2); } // TEST(Not, This) @@ -60,10 +62,10 @@ TEST(Math, check(r.calls[1].name == "TEST_P" && r.calls[1].line == 3); check(r.calls[2].line == 5 && r.calls[2].arguments[1] == "Spans"); check(scan::literal_value(R"("a \"b\"" " c")") == "a \"b\" c"); - } + }; // A program's framework: its package in the plan and its macros in its source. - { + "a framework needs its package and its macros"_test = [&] { const auto gtestSource = scan::scan("TEST(A, B) {}\nTEST_F(F, C) {}\n", macros); check(testing::framework_of({"app", "compat.gtest"}, gtestSource, Json::object()) == &fw("gtest")); check(testing::framework_of({"app"}, gtestSource, Json::object()) == nullptr); @@ -83,10 +85,10 @@ TEST(Math, check(catchCases.size() == 2 && catchCases[0].name == "adds" && catchCases[1].name == "Scenario: grows"); const auto doctestCases = testing::static_cases(fw("doctest"), both, Json::object()); check(doctestCases[1].name == " Scenario: grows"); - } + }; // Listings, base names and the merge. - { + "listings, base names and the merge"_test = [&] { const auto names = testing::listed(fw("gtest"), "Math.\n Adds\n Spans\nInst/Param.\n Works/0 # GetParam() = 1\n Works/1 # GetParam() = 2\nTyped/0. # TypeParam = int\n Holds\n"); check(names == std::vector({"Math.Adds", "Math.Spans", "Inst/Param.Works/0", "Inst/Param.Works/1", "Typed/0.Holds"})); check(testing::base_name(fw("gtest"), "Inst/Param.Works/1") == "Param.Works"); @@ -101,20 +103,20 @@ TEST(Math, check(listedCatch == std::vector({"adds", "a, b"})); const auto listedDoctest = testing::listed(fw("doctest"), "[doctest] doctest version is \"2.4.12\"\n===\nadds\n Scenario: grows\n===\n[doctest] unskipped test cases passing the current filters: 2\n"); check(listedDoctest == std::vector({"adds", " Scenario: grows"})); - } + }; // The arguments after `--`: the filter escaped by the framework's rules, and the report. - { + "the filter and the report arguments"_test = [&] { const auto g = testing::run_args(fw("gtest"), {"Math.Adds", "Inst/Param.Works/0"}, "/r/out"); check(g == std::vector({"--gtest_filter=Math.Adds:*/Math.Adds/*:Math/*.Adds:Inst/Param.Works/0", "--gtest_output=json:/r/out"})); const auto c = testing::run_args(fw("catch2"), {"a, b", "x[1]"}, "/r/out"); check(c == std::vector({"a\\, b,x\\[1\\]", "--reporter", "json::out=/r/out"})); const auto d = testing::run_args(fw("doctest"), {}, "/r/out"); check(d == std::vector({"--reporters=junit", "--out=/r/out"})); - } + }; // Reports. - { + "reports"_test = [&] { const auto g = testing::read_report(fw("gtest"), R"({"testsuites": [{"name": "Math", "testsuite": [ {"name": "Adds", "status": "RUN", "result": "COMPLETED", "time": "0.002s"}, {"name": "Spans", "status": "RUN", "result": "COMPLETED", "time": "0s", "failures": [{"failure": "a.cpp:3\nExpected 2"}]}, @@ -148,13 +150,13 @@ TEST(Math, check(outputs.at("Math.Spans").contains("a.cpp:3: Failure") && outputs.at("Math.Spans").ends_with("[ FAILED ] Math.Spans (0 ms)\n")); check(outputs.at("Math.Crashes") == "[ RUN ] Math.Crashes\nhalf way\n"); check(testing::case_outputs(fw("catch2"), "[ RUN ] x\n").empty()); - } + }; // mtest's cases as mtest's static manifest reads them: each object (`UTest`, `Suite`, ...) is a // case, and so is each `"name"_test` declared in one, under its name; never a literal in a // comment or one not followed by `_test`. The framework is claimed by its package and its // cases, and a record of `mcpp test` lists and reports them (the test protocol). - { + "mtest cases by its manifest"_test = [&] { const auto r = scan::scan(R"(import mcpplibs.mtest; using namespace mcpplibs::mtest; // "commented"_test @@ -190,11 +192,11 @@ int main() { "top"_test = [] {}; return report(); } && outcomes[1].outcome.message.contains("expected 2, actual 1")); check(outcomes.size() == 4 && outcomes[2].outcome.status == "skipped" && outcomes[2].outcome.message == "later"); check(outcomes.size() == 4 && outcomes[3].outcome.status == "passed"); - } + }; // A suite type the project's tests declare in a module beside them is a suite type in every // file that uses it: the server reads the types across tests/ before it scans a file. - { + "suite types across sources"_test = [&] { const std::vector sources{ "export module cts;\nimport mcpplibs.mtest;\nexport class Fixture : public mcpplibs::mtest::CTS {\n" "public:\n using CTS::CTS;\n};\n"}; @@ -203,11 +205,11 @@ int main() { "top"_test = [] {}; return report(); } const auto r = scan::scan(cases, {}, types); check(r.cases.size() == 2 && r.cases[0].id == "one" && r.cases[1].id == "two@warm" && r.cases[1].line == 2); check(scan::scan(cases, {}).cases.empty()); - } + }; // source/entries and test/cases over a project on disk (no mcpp: the plan's packages are not // known, so no framework is claimed, and the static entries still answer). - { + "source entries and cases on disk"_test = [&] { const auto base = testsupport::temp() / std::format("mcppide-testing-{}", std::random_device{}()); std::filesystem::create_directories(base / "src"); std::filesystem::create_directories(base / "tests" / "unit"); @@ -246,8 +248,5 @@ int main() { "top"_test = [] {}; return report(); } check(cases["framework"].is_null() && cases["cases"].empty()); std::error_code ignored; std::filesystem::remove_all(base, ignored); - } - - std::println("test_testing: ok"); - return 0; -} + }; +}}; diff --git a/tests/unit/teststore.cpp b/tests/unit/teststore.cpp new file mode 100644 index 0000000..e06974c --- /dev/null +++ b/tests/unit/teststore.cpp @@ -0,0 +1,136 @@ +// The runs of a project's tests as mcpp keeps them (mtest spec P12; design 2026-11-10): a store read +// back, its rows attributed to their programs, and what the window says -- the failures and since +// when, the slowest, the flaky, the metrics against their bounds, the schedule and its critical path. +// And the model of a program: the parts of a directory program derived as mcpp 2026.11.9.1 forms +// them, or taken as mcpp#917 states them; a record attributed to its program; a case's status. +import std; +import mcppide.testsupport; +import mcpplibs.mtest; +import nlohmann.json; +import mcppide.scan; +import mcppide.testing; +import mcppide.teststore; + +using testsupport::check; +using namespace mcpplibs::mtest; + +using namespace mcppide; +using Json = nlohmann::json; + +UTest teststore{"teststore", [] { + const auto fixture = testsupport::repo() / "protocol" / "fixtures" / "mtest-store"; + + // The store's last runs, oldest first; a run read once is kept, and a window takes the newest. + "the last runs of a store"_test = [&] { + std::map cache; + auto runs = teststore::last_runs(fixture, 20, cache); + check(runs.size() == 3 && cache.size() == 3); + check(runs.front().id == "20261108T100000-0a01" && runs.back().id == "20261110T100000-0c03"); + check(runs.back().programs.size() == 3 && runs.back().cases.size() == 2); + check(runs.back().cases[0].kind == "bench" && runs.back().cases[0].metrics.size() == 1); + check(runs.front().cases[0].kind.empty() && !runs.front().cases[0].startedMs); + const auto two = teststore::last_runs(fixture, 2, cache); + check(two.size() == 2 && two.front().id == "20261109T100000-0b02"); + std::map none; + check(teststore::last_runs(fixture / "absent", 20, none).empty()); + + // The parts are a program's: `unit/a` and `unit/b` are `unit`'s. + teststore::attribute(runs, [](std::string_view name) { return name.starts_with("unit/") ? std::string("unit") : std::string(name); }); + check(runs.back().cases[1].program == "unit" && runs.back().cases[1].part == "unit/b"); + check(runs.back().programs[0].name == "unit" && runs.back().programs[0].part == "unit/a"); + check(runs.back().programs[2].name == "smoke" && runs.back().programs[2].part.empty()); + + const auto in = teststore::insights(runs); + check(in["runs"] == 3 && in["run"]["id"] == "20261110T100000-0c03" && in["run"]["status"] == "fail"); + check(in["byKind"]["bench"]["pass"] == 1 && in["byKind"]["unit"]["fail"] == 1); + // beta failed, passed, failed: it fails since the last run, and it is flaky. + check(in["failures"].size() == 1 && in["failures"][0]["case"] == "beta" && in["failures"][0]["since"] == "20261110T100000-0c03"); + check(in["failures"][0]["new"] == true && in["failures"][0]["part"] == "unit/b" && in["failures"][0]["line"] == 3); + check(in["failures"][0]["expected"] == "4" && in["failures"][0]["actual"] == "5"); + check(in["flaky"].size() == 1 && in["flaky"][0]["passed"] == 1 && in["flaky"][0]["failed"] == 2 && in["flaky"][0]["flips"] == 2); + check(in["slowest"][0]["case"] == "beta" && in["slowest"][0]["ms"] == 1.6 && in["slowest"][0]["medianMs"] == 1.5); + check(in["metrics"].size() == 1 && in["metrics"][0]["name"] == "ready" && in["metrics"][0]["values"].size() == 1); + check(std::abs(in["metrics"][0]["margin"].get() - 0.72) < 1e-9 && in["metrics"][0]["better"] == "lower"); + // Two workers; the critical path is the lane that ends last (worker 0: unit/a, then smoke). + check(in["schedule"]["lanes"].size() == 2 && in["schedule"]["criticalPath"] == Json::array({"unit", "smoke"})); + check(in["available"]["metrics"] == true && in["available"]["kinds"] == true && in["available"]["cases"] == true); + + const auto h = teststore::history(runs, "unit", "beta"); + check(h["runs"].size() == 3 && h["cases"].size() == 1 && h["cases"][0]["results"][1]["status"] == "pass"); + check(h["cases"][0]["results"][2]["part"] == "unit/b"); + check(teststore::insights({})["run"].is_null()); + }; + + // `[test] store` names the store; target/mtest otherwise. + "the store of a project"_test = [&] { + const auto root = testsupport::temp() / std::format("mcppide-store-{}", std::chrono::steady_clock::now().time_since_epoch().count()); + std::filesystem::create_directories(root); + std::ofstream(root / "mcpp.toml") << "[package]\nname = \"a\"\n\n[test]\nprograms = \"directory\"\n"; + check(teststore::store_dir(root) == root / "target" / "mtest"); + std::ofstream(root / "mcpp.toml") << "[package]\nname = \"a\"\n\n[test]\nstore = \".runs\"\n\n[build]\nstore = \"x\"\n"; + check(teststore::store_dir(root) == root / ".runs"); + std::filesystem::remove_all(root); + }; + + // A program of a directory, as mcpp 2026.11.9.1 forms it: its sources, a part per file, and `main` + // only where a file defines it; a program of one file is that file. mcpp#917 states them. + "the programs of a directory"_test = [&] { + const auto root = testsupport::temp() / std::format("mcppide-program-{}", std::chrono::steady_clock::now().time_since_epoch().count()); + std::filesystem::create_directories(root / "tests" / "unit"); + for (const auto* f : {"b.cpp", "a.cpp", "notes.md"}) std::ofstream(root / "tests" / "unit" / f) << "\n"; + const auto a = (root / "tests" / "unit" / "a.cpp").generic_string(), b = (root / "tests" / "unit" / "b.cpp").generic_string(); + const auto none = [](const std::filesystem::path&) { return false; }; + auto unit = testing::program_of(Json{{"test", "unit"}, {"main", a}}, none); + check(unit.layout == "directory" && unit.main.empty() && unit.files == std::vector{a, b}); + check(unit.parts.size() == 2 && unit.parts[0].first == "unit/a" && unit.parts[1].second == b); + const auto withMain = testing::program_of(Json{{"test", "unit"}, {"main", a}}, + [&](const std::filesystem::path& f) { return f.generic_string() == b; }); + check(withMain.main == b); + const auto one = testing::program_of(Json{{"test", "unit/a"}, {"main", a}}, none); + check(one.layout == "file" && one.files == std::vector{a} && one.parts.empty() && one.main == a); + const auto stated = testing::program_of(Json{{"test", "unit"}, {"main", nullptr}, {"files", Json::array({a, b})}, + {"parts", Json::array({Json{{"name", "unit/a"}, {"file", a}}})}}, none); + check(stated.layout == "directory" && stated.main.empty() && stated.parts.size() == 1); + const auto json = testing::program_json(unit); + check(json["layout"] == "directory" && json["parts"][1]["name"] == "unit/b" && json["main"] == ""); + + const std::vector programs{unit, one, testing::program_of(Json{{"test", "smoke"}, {"main", "x/smoke.cpp"}}, none)}; + check(testing::owner_of("unit/b", "", programs) == "unit"); + check(testing::owner_of("unit/a", "", programs) == "unit/a"); // a program of that name + check(testing::owner_of("smoke", "", programs) == "smoke"); + check(testing::owner_of("other/x", "", programs) == "other/x"); + check(testing::owner_of("unit/b", "stated", programs) == "stated"); + check(testing::part_of_file(unit, b) == "unit/b" && testing::part_of_file(unit, "x.cpp").empty()); + check(testing::kind_of_path(a) == "unit" && testing::kind_of_path("/p/tests/bench/x.cpp") == "bench"); + check(testing::kind_of_path("/p/tests/x.cpp").empty() && testing::kind_of_path("/p/src/unit/x.cpp").empty()); + std::filesystem::remove_all(root); + }; + + // A case's status as the editors know it, and its failures, metrics and bounds. + "a case as the editors receive it"_test = [&] { + check(testing::case_status("pass").status == "passed" && testing::case_status("skip").status == "skipped"); + check(testing::case_status("xfail").status == "passed" && testing::case_status("xfail").expectedFailure); + check(testing::case_status("xpass").status == "failed" && testing::case_status("xpass").unexpectedPass); + check(testing::case_status("invalid").status == "errored" && testing::case_status("fail").status == "failed"); + const auto c = testing::protocol_case(Json{{"id", "r"}, {"status", "fail"}, {"duration_ms", 2.6}, + {"failures", Json::array({Json{{"file", "t.cpp"}, {"line", 7}, {"message", "a == b"}, {"expected", "1"}, {"actual", "2"}}})}, + {"metrics", Json::array({Json{{"name", "ready"}, {"value", 3}}})}, {"bounds", Json::array({Json{{"metric", "ready"}, {"max", 5}}, Json{{"max", 1}}})}, + {"kind", "bench"}, {"tags", Json::array({"slow"})}}); + check(c["case"] == "r" && c["status"] == "failed" && c["durationMs"] == 2 && c["location"]["line"] == 6); + check(c["failures"][0]["expected"] == "1" && c["message"] == "t.cpp:7: a == b (expected 1, actual 2)"); + check(c["metrics"].size() == 1 && c["bounds"].size() == 1 && c["kind"] == "bench" && c["tags"][0] == "slow"); + const auto plain = testing::protocol_case(Json{{"id", "s"}, {"status", "skip"}, {"reason", "no network"}}); + check(plain["message"] == "no network" && plain["location"].is_null() && !plain.contains("metrics") && !plain.contains("expectedFailure")); + }; + + // A case's kind and the bounds on its metrics, as its source states them (mtest's manifest). + "a case as its source states it"_test = [&] { + const auto found = scan::scan("import mcpplibs.mtest;\nusing namespace mcpplibs::mtest;\n" + "Bench alpha{\"alpha\", at_most{\"ready\", 15}, [] {}};\nUTest beta{\"beta\", [] {}};\n", {}); + const auto cases = testing::static_cases(*testing::framework("mtest"), found, Json::object()); + check(cases.size() == 2 && cases[0].kind == "bench" && cases[0].bounds.size() == 1 && cases[0].bounds[0]["max"] == 15); + check(cases.size() == 2 && cases[1].kind == "unit" && cases[1].bounds.empty()); + const auto row = testing::case_json(cases[0]); + check(row["kind"] == "bench" && row["bounds"][0]["metric"] == "ready" && !row.contains("tags") && !row.contains("file")); + }; +}}; diff --git a/tests/test_views.cpp b/tests/unit/views.cpp similarity index 99% rename from tests/test_views.cpp rename to tests/unit/views.cpp index 73e2399..360cab6 100644 --- a/tests/test_views.cpp +++ b/tests/unit/views.cpp @@ -1,18 +1,20 @@ // The view models, from the golden fixtures: what each panel and tree shows. import std; import mcppide.testsupport; +import mcpplibs.mtest; import nlohmann.json; import mcppide.views; import mcppide.policy; using testsupport::check; +using namespace mcpplibs::mtest; using namespace mcppide; using Json = nlohmann::json; static Json fixture(const char* name) { return Json::parse(testsupport::read(testsupport::fixtures() / name)); } -int main() { +UTest views{"views", [] { auto cache = fixture("cache-list.json"); views::CacheOptions o; o.topN = 3; @@ -157,5 +159,4 @@ int main() { check(views::failure_location("C:/p/b.cpp:7: Failure")["path"] == "C:/p/b.cpp"); check(views::failure_location("started at 12:30: ok").is_null()); check(views::failure_location("no location here").is_null()); - std::println("test_views: ok"); -} +}}; diff --git a/testsupport/mcpp.toml b/testsupport/mcpp.toml index 12c6593..0b8dac6 100644 --- a/testsupport/mcpp.toml +++ b/testsupport/mcpp.toml @@ -1,7 +1,7 @@ [package] name = "mcpp-ide-testsupport" version = "0.1.0" -description = "What every backend test of mcpp-ide shares: a check that says where it failed, and the paths of the fixtures and the built programs" +description = "What every backend test of mcpp-ide shares: a check that ends its case where it failed (mtest), and the paths of the fixtures and the built programs" license = "Apache-2.0" [language] @@ -11,3 +11,6 @@ import_std = true [targets.mcpp-ide-testsupport] kind = "lib" + +[dependencies] +mcpplibs.mtest = "0.5.2" diff --git a/testsupport/src/testsupport.cppm b/testsupport/src/testsupport.cppm index 1d5decd..ded5d7f 100644 --- a/testsupport/src/testsupport.cppm +++ b/testsupport/src/testsupport.cppm @@ -1,17 +1,18 @@ -// mcppide.testsupport: what every backend test shares -- a check that says where it failed, and the -// paths of the fixtures and the built programs. A module, as the backend's own units are: no test -// includes a header (mcpp-ide-dev check-sources). +// mcppide.testsupport: what every backend test shares -- a check that ends its case where it failed, +// and the paths of the fixtures and the built programs. A module, as the backend's own units are: no +// test includes a header (mcpp-ide-dev check-sources). The tests are mtest's (tests/unit, tests/e2e, +// tests/cts, tests/bench; mtest docs/layout.md). export module mcppide.testsupport; import std; +import mcpplibs.mtest; export namespace testsupport { -// A failed check names its place and ends the test. +// A check is mtest's fatal expectation: a failure is reported at the check's own place, and the +// case ends there (the checks after it read what it checked). inline void check(bool cond, std::source_location at = std::source_location::current()) { - if (cond) return; - std::println(std::cerr, "{}:{}: check failed in {}", at.file_name(), at.line(), at.function_name()); - std::exit(1); + mcpplibs::mtest::expect(mcpplibs::mtest::fatal(cond), at); } // A name in the generic spelling: `/` between components. The tests run as ok images too, whose @@ -21,9 +22,14 @@ inline std::filesystem::path generic(std::string text) { for (auto& c : text) if (c == '\\') c = '/'; return std::filesystem::path(text); } -// The repository, from this file's own path (testsupport/src/testsupport.cppm). +// The repository, from this file's own path (testsupport/src/testsupport.cppm). A build that names a +// source from its package's root (the ok images') gives a relative name; the tests run in the +// repository, so it is taken from there. Either way the answer is absolute: a test starts programs in +// directories of its own. inline std::filesystem::path repo() { - return generic(std::source_location::current().file_name()).parent_path().parent_path().parent_path(); + const auto named = generic(std::source_location::current().file_name()); + if (named.is_absolute()) return named.parent_path().parent_path().parent_path(); + return generic(std::filesystem::current_path().string()); } // The temporary directory, in the generic spelling. inline std::filesystem::path temp() { return generic(std::filesystem::temp_directory_path().string()); } diff --git a/webui/src/tests.ts b/webui/src/tests.ts new file mode 100644 index 0000000..6a79ec8 --- /dev/null +++ b/webui/src/tests.ts @@ -0,0 +1,263 @@ +// The tests page: what the last runs of a project's tests say (the backend's `mcpp-ide/test/insights`, +// read from the store mcpp keeps; design 2026-11-10). mcpp-clion shows it in JCEF beside the graph +// page, mcpp-vscode in a webview. The page decides nothing: the host hands it the backend's answer and +// its strings (`mcppRender`), and a click on a failure asks the host to open its place +// (`mcppHostSend`). A section whose data the store does not hold is left out; the note says why. + +export interface Labels { + title: string; + /** Said when the store holds no run; `{0}` is the store's directory. */ + noRun: string; + /** `{0}` runs kept, `{1}` the run's start. */ + window: string; + failures: string; + /** A failure that began in the last run. */ + fresh: string; + /** `{0}`: the run since which a case fails. */ + since: string; + expected: string; + actual: string; + metrics: string; + slowest: string; + /** `{0}`: a duration. */ + median: string; + flaky: string; + /** `{0}` passes, `{1}` failures, `{2}` changes. */ + flakyCounts: string; + schedule: string; + /** `{0}`: a worker's number. */ + worker: string; + criticalPath: string; + /** Said when mcpp states no metrics (test-metrics, mcpp#917). */ + noMetrics: string; +} + +export interface Bound { max?: number; min?: number } + +export interface Insights { + store: string; + runs: number; + run: { id: string; started?: string; status: string; ms?: number; jobs?: number; counts?: Record } | null; + byKind?: Record>; + failures: { program: string; part?: string; case: string; status: string; message?: string; path?: string; line?: number; + expected?: string; actual?: string; since: string; new: boolean }[]; + slowest: { program: string; part?: string; case: string; ms: number; medianMs: number }[]; + flaky: { program: string; part?: string; case: string; passed: number; failed: number; flips: number }[]; + metrics: { program: string; part?: string; case: string; name: string; unit?: string; values: { run: string; value: number }[]; + last: number; bound: Bound | null; margin: number | null; better?: string }[]; + schedule: { lanes: { worker: number; items: { program: string; part?: string; startMs: number; ms: number; status: string }[] }[]; + criticalPath: string[]; ms?: number } | null; + available?: { kinds?: boolean; metrics?: boolean; cases?: boolean }; + features?: { parts?: boolean; live?: boolean; metrics?: boolean }; +} + +export interface TestsPage { + insights: Insights; + labels: Labels; + dark?: boolean; +} + +declare global { + interface Window { + mcppRender(page: TestsPage): void; + mcppHostSend?: (text: string) => void; + } +} + +const STYLE = ` +:root { --fg: #1f1f1f; --bg: #ffffff; --muted: #6b6f76; --line: #d9dbe0; --ok: #2e7d32; --bad: #c62828; --warn: #b26a00; --accent: #3367d6; } +body.dark { --fg: #dfe1e5; --bg: #2b2d30; --muted: #9a9ea6; --line: #43454a; --ok: #6abf69; --bad: #ef6c6c; --warn: #e0a54a; --accent: #7aa2f7; } +body.vscode { --fg: var(--vscode-foreground); --bg: var(--vscode-editor-background); --muted: var(--vscode-descriptionForeground); + --line: var(--vscode-panel-border, var(--vscode-widget-border, #8884)); --ok: var(--vscode-testing-iconPassed); + --bad: var(--vscode-testing-iconFailed); --warn: var(--vscode-editorWarning-foreground); --accent: var(--vscode-textLink-foreground); } +body { margin: 0; padding: 12px 16px; font: 13px system-ui, sans-serif; color: var(--fg); background: var(--bg); } +h1 { font-size: 15px; margin: 0 0 2px; } +h2 { font-size: 12px; text-transform: uppercase; letter-spacing: .04em; color: var(--muted); margin: 18px 0 6px; font-weight: 600; } +.sub { color: var(--muted); margin-bottom: 8px; } +.counts span { margin-right: 12px; font-variant-numeric: tabular-nums; } +.ok { color: var(--ok); } .bad { color: var(--bad); } .warn { color: var(--warn); } .muted { color: var(--muted); } +.chip { display: inline-block; border: 1px solid var(--line); border-radius: 10px; padding: 0 8px; margin: 4px 6px 0 0; font-size: 12px; } +.row { display: grid; grid-template-columns: minmax(0, 1fr) auto; gap: 8px; padding: 4px 0; border-top: 1px solid var(--line); align-items: center; } +.row > :nth-child(2) { white-space: nowrap; } +.row:first-of-type { border-top: 0; } +.name { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } +.name small { color: var(--muted); } +a[data-open] { color: inherit; text-decoration: none; cursor: pointer; } +a[data-open]:hover .case { text-decoration: underline; } +.detail { grid-column: 1 / -1; color: var(--muted); font: 12px ui-monospace, monospace; white-space: pre-wrap; } +.badge { font-size: 11px; border-radius: 3px; padding: 0 5px; border: 1px solid currentColor; } +.bar { height: 4px; background: var(--accent); border-radius: 2px; opacity: .7; } +svg text { fill: var(--muted); font-size: 10px; } +.spark { vertical-align: middle; margin-right: 6px; } +.spark polyline { fill: none; stroke: var(--accent); stroke-width: 1.5; } +.spark line { stroke: var(--bad); stroke-dasharray: 3 2; stroke-width: 1; } +.lane rect { fill: var(--accent); opacity: .55; } +.lane rect.fail { fill: var(--bad); opacity: .7; } +.lane rect.critical { opacity: 1; stroke: var(--fg); stroke-width: 1; } +.note { color: var(--muted); margin-top: 16px; font-size: 12px; } +`; + +export function escapeHtml(s: string): string { + return s.replace(/&/g, "&").replace(//g, ">").replace(/"/g, """); +} + +function fill(text: string, ...args: (string | number)[]): string { + return text.replace(/\{(\d)\}/g, (_, i: string) => String(args[Number(i)] ?? "")); +} + +/** A duration as a person reads it: `0.4 ms`, `12 ms`, `3.4 s`, `2 min 5 s`. */ +export function duration(ms: number): string { + if (ms < 1) return `${ms.toFixed(1)} ms`; + if (ms < 1000) return `${Math.round(ms)} ms`; + if (ms < 60_000) return `${(ms / 1000).toFixed(1)} s`; + return `${Math.floor(ms / 60_000)} min ${Math.round((ms % 60_000) / 1000)} s`; +} + +/** A run's time as a person reads it: `2026-11-08 10:00` from an ISO time or a run's identifier (P12.2). */ +export function when(text: string): string { + const iso = /^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2})/.exec(text); + const id = /^(\d{4})(\d{2})(\d{2})T(\d{2})(\d{2})/.exec(text); + const m = iso ?? id; + return m ? `${m[1]}-${m[2]}-${m[3]} ${m[4]}:${m[5]}` : text; +} + +/** A metric's value with its unit: three significant digits. */ +export function measured(value: number, unit?: string): string { + const text = Math.abs(value) >= 100 ? value.toFixed(0) : Number(value.toPrecision(3)).toString(); + return unit && unit !== "1" ? `${text} ${unit}` : text; +} + +/** `≤ 15 s`, `≥ 0.9`, `0.9 – 15 s`. */ +export function boundText(b: Bound | null, unit?: string): string { + if (!b) return ""; + if (b.max !== undefined && b.min !== undefined) return `${measured(b.min)} – ${measured(b.max, unit)}`; + if (b.max !== undefined) return `≤ ${measured(b.max, unit)}`; + if (b.min !== undefined) return `≥ ${measured(b.min, unit)}`; + return ""; +} + +function where(row: { program: string; part?: string; case: string }): string { + const part = row.part ? ` ${escapeHtml(row.part.slice(row.program.length + 1) || row.part)}` : ""; + return `${escapeHtml(row.case || row.program)} ${escapeHtml(row.case ? row.program : "")}${part}`; +} + +/** A series as a line 120×24, the bound dashed when it lies within reach of the values. */ +export function sparkline(values: number[], bound: Bound | null): string { + if (values.length === 0) return ""; + const limit = bound?.max ?? bound?.min; + const all = limit !== undefined ? [...values, limit] : values; + const lo = Math.min(...all), hi = Math.max(...all); + const span = hi - lo || 1; + const y = (v: number) => (22 - ((v - lo) / span) * 20).toFixed(1); + const step = values.length > 1 ? 118 / (values.length - 1) : 0; + const points = values.map((v, i) => `${(1 + i * step).toFixed(1)},${y(v)}`).join(" "); + const line = limit !== undefined ? `` : ""; + return `${line}`; +} + +function summary(page: TestsPage): string { + const { insights: x, labels: l } = page; + const run = x.run!; + const c = run.counts ?? {}; + const n = (k: string) => c[k] ?? 0; + const bad = n("fail") + n("xpass") + n("invalid") + n("build-fail"); + const counts = [`✓ ${n("pass") + n("xfail")}`]; + if (bad) counts.push(`✗ ${bad}`); + if (n("skip") + n("not-run")) counts.push(`○ ${n("skip") + n("not-run")}`); + if (run.ms) counts.push(`${duration(run.ms)}`); + const kinds = Object.entries(x.byKind ?? {}).filter(([k]) => k).map(([k, v]) => { + const total = Object.values(v).reduce((a, b) => a + b, 0); + const failing = (v.fail ?? 0) + (v.xpass ?? 0) + (v.invalid ?? 0); + return `${escapeHtml(k)} ${total}${failing ? ` ✗ ${failing}` : ""}`; + }).join(""); + return `

${escapeHtml(l.title)}

${escapeHtml(fill(l.window, x.runs, when(run.started || run.id)))}
+
${counts.join("")}
${kinds ? `
${kinds}
` : ""}`; +} + +function failures(page: TestsPage): string { + const { insights: x, labels: l } = page; + if (x.failures.length === 0) return ""; + const rows = x.failures.map((f) => { + const open = f.path ? ` data-open="${escapeHtml(JSON.stringify({ path: f.path, line: f.line ?? 0 }))}"` : ""; + const badge = f.new ? `${escapeHtml(l.fresh)}` : `${escapeHtml(fill(l.since, when(f.since)))}`; + const sides = f.expected !== undefined ? `\n${l.expected}: ${f.expected}\n${l.actual}: ${f.actual ?? ""}` : ""; + const detail = (f.message ?? "") + sides; + return `
✗ ${where(f)}${badge}${detail ? `
${escapeHtml(detail)}
` : ""}
`; + }); + return `

${escapeHtml(l.failures)}

${rows.join("")}`; +} + +function metrics(page: TestsPage): string { + const { insights: x, labels: l } = page; + if (x.metrics.length === 0) return ""; + const rows = x.metrics.map((m) => { + const tone = m.margin === null ? "" : m.margin < 0 ? "bad" : m.margin < 0.2 ? "warn" : "ok"; + const bound = boundText(m.bound, m.unit); + const value = `${escapeHtml(measured(m.last, m.unit))}${bound ? ` ${escapeHtml(bound)}` : ""}`; + return `
${escapeHtml(m.name)} · ${where(m)}${sparkline(m.values.map((v) => v.value), m.bound)} ${value}
`; + }); + return `

${escapeHtml(l.metrics)}

${rows.join("")}`; +} + +function slowest(page: TestsPage): string { + const { insights: x, labels: l } = page; + if (x.slowest.length === 0) return ""; + const top = Math.max(...x.slowest.map((s) => s.ms), 1); + const rows = x.slowest.map((s) => `
${where(s)}
 +${escapeHtml(duration(s.ms))} ${escapeHtml(fill(l.median, duration(s.medianMs)))}
`); + return `

${escapeHtml(l.slowest)}

${rows.join("")}`; +} + +function flaky(page: TestsPage): string { + const { insights: x, labels: l } = page; + if (x.flaky.length === 0) return ""; + const rows = x.flaky.map((f) => `
⟳ ${where(f)} +${escapeHtml(fill(l.flakyCounts, f.passed, f.failed, f.flips))}
`); + return `

${escapeHtml(l.flaky)}

${rows.join("")}`; +} + +/** The last run's schedule: a lane per worker, each process a bar, the critical path outlined. */ +export function schedule(page: TestsPage): string { + const { insights: x, labels: l } = page; + const s = x.schedule; + if (!s || s.lanes.length === 0) return ""; + const end = Math.max(s.ms ?? 0, ...s.lanes.flatMap((lane) => lane.items.map((i) => i.startMs + i.ms)), 1); + const width = 600, row = 18; + const critical = new Set(s.criticalPath); + const lanes = s.lanes.map((lane, i) => { + const bars = lane.items.map((item) => { + const x0 = (item.startMs / end) * width, w = Math.max(1.5, (item.ms / end) * width); + const cls = [item.status === "fail" || item.status === "build-fail" ? "fail" : "", critical.has(item.program) ? "critical" : ""].join(" ").trim(); + return `${escapeHtml(`${item.part ?? item.program} · ${duration(item.ms)}`)}`; + }).join(""); + return `${escapeHtml(fill(l.worker, lane.worker))}${bars}`; + }).join(""); + const height = s.lanes.length * row + 14; + const axis = `0${escapeHtml(duration(end))}`; + const path = s.criticalPath.length ? `
${escapeHtml(l.criticalPath)}: ${s.criticalPath.map(escapeHtml).join(" → ")}
` : ""; + return `

${escapeHtml(l.schedule)}

${lanes}${axis}${path}`; +} + +/** The page's whole document body, from the backend's answer and the host's strings. */ +export function renderTests(page: TestsPage): string { + const { insights: x, labels: l } = page; + if (!x.run) return `

${escapeHtml(l.title)}

${escapeHtml(fill(l.noRun, x.store))}

`; + const note = x.features?.metrics || x.available?.metrics ? "" : `

${escapeHtml(l.noMetrics)}

`; + return summary(page) + failures(page) + metrics(page) + slowest(page) + flaky(page) + schedule(page) + note; +} + +if (typeof window !== "undefined") { + window.mcppRender = (page: TestsPage): void => { + if (!document.getElementById("mcpp-tests-style")) { + const style = document.createElement("style"); + style.id = "mcpp-tests-style"; + style.textContent = STYLE; + document.head.appendChild(style); + } + document.body.classList.toggle("dark", !!page.dark); + const target = document.getElementById("tests") ?? document.body; + target.innerHTML = renderTests(page); + for (const a of Array.from(target.querySelectorAll("[data-open]"))) + a.addEventListener("click", () => window.mcppHostSend?.(JSON.stringify({ open: JSON.parse(a.dataset.open!) }))); + }; +} diff --git a/webui/tests.html b/webui/tests.html new file mode 100644 index 0000000..4bb661e --- /dev/null +++ b/webui/tests.html @@ -0,0 +1,11 @@ + + + + +mcpp tests + + + +
+ + diff --git a/webui/tests/tests.test.mjs b/webui/tests/tests.test.mjs new file mode 100644 index 0000000..59cbf02 --- /dev/null +++ b/webui/tests/tests.test.mjs @@ -0,0 +1,114 @@ +// The tests page as mcpp-clion shows it in JCEF and mcpp-vscode in its Test Insights webview, in +// Chromium: a real `mcpp-ide serve` answers `mcpp-ide/test/insights` from the store mcpp keeps (the +// fixture's three runs, the last with mcpp#917's kinds and metrics), and the page draws what it is +// handed -- the run, its failure with both sides, the metric against its bound, the flaky case, the +// schedule with its critical path -- asks the IDE to open a failure's place when it is clicked, and +// says so when no run is kept. +// +// Run from mide-plugins/vscode: npm run test:webui (it bundles the page first). MCPP_IDE_BACKEND and +// MCPP_IDE_MOCK_MCPP name the programs; else the newest of the repository's target/ are taken. +import assert from "node:assert/strict"; +import { spawn } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import test from "node:test"; +import { createRequire } from "node:module"; +import { fileURLToPath, pathToFileURL } from "node:url"; + +const here = path.dirname(fileURLToPath(import.meta.url)); +const repo = path.resolve(here, "..", ".."); +const extension = path.join(repo, "mide-plugins", "vscode"); +const require = createRequire(path.join(extension, "package.json")); +const { chromium } = require("playwright"); +const rpc = require("vscode-jsonrpc/node"); + +function built(name) { + const env = name === "mcpp-ide" ? process.env.MCPP_IDE_BACKEND : process.env.MCPP_IDE_MOCK_MCPP; + if (env) return env; + const exe = process.platform === "win32" ? `${name}.exe` : name; + let best, bestTime = 0; + const target = path.join(repo, "target"); + if (!fs.existsSync(target)) return undefined; + for (const triple of fs.readdirSync(target, { withFileTypes: true }).filter((e) => e.isDirectory()).map((e) => e.name)) + for (const config of fs.readdirSync(path.join(target, triple), { withFileTypes: true })) { + if (!config.isDirectory()) continue; + const p = path.join(target, triple, config.name, "bin", exe); + if (fs.existsSync(p) && fs.statSync(p).mtimeMs > bestTime) { best = p; bestTime = fs.statSync(p).mtimeMs; } + } + return best; +} + +const page = path.join(extension, ".webui-test", "tests.html"); + +const labels = { + title: "Test Insights", noRun: "no run in {0}", window: "{0} runs · {1}", failures: "Failures", fresh: "new", since: "since {0}", + expected: "expected", actual: "actual", metrics: "Metrics", slowest: "Slowest", median: "median {0}", flaky: "Flaky", + flakyCounts: "{0}/{1}/{2}", schedule: "Schedule", worker: "w{0}", criticalPath: "Critical path", noMetrics: "no metrics", +}; + +test("the tests page draws what the store's runs say", async (t) => { + const backend = built("mcpp-ide"), mock = built("mcpp-ide-mock-mcpp"); + if (!backend || !mock) { t.skip("build the backend first (mcpp build)"); return; } + assert.ok(fs.existsSync(page), "bundle the page first (npm run test:webui)"); + const root = fs.mkdtempSync(path.join(os.tmpdir(), "mcpp-tests-")); + fs.mkdirSync(path.join(root, "tests", "unit"), { recursive: true }); + fs.writeFileSync(path.join(root, "mcpp.toml"), '[package]\nname = "app"\nversion = "0.1.0"\n\n[test]\nprograms = "directory"\n'); + fs.writeFileSync(path.join(root, ".mock-mcpp-tests"), "directory"); + for (const f of ["a.cpp", "b.cpp"]) fs.writeFileSync(path.join(root, "tests", "unit", f), "\n"); + fs.writeFileSync(path.join(root, "tests", "smoke.cpp"), "int main() {}\n"); + + const child = spawn(backend, ["serve"], { stdio: ["pipe", "pipe", "inherit"] }); + const connection = rpc.createMessageConnection(new rpc.StreamMessageReader(child.stdout), new rpc.StreamMessageWriter(child.stdin)); + connection.listen(); + const browser = await chromium.launch(); + try { + await connection.sendRequest("initialize", { + processId: process.pid, + workspaceFolders: [{ uri: pathToFileURL(root).href, name: "app" }], + capabilities: {}, + initializationOptions: { editor: "jetbrains", settings: { "mcpp.path": mock } }, + }); + const empty = await connection.sendRequest("mcpp-ide/test/insights", { root }); + fs.cpSync(path.join(repo, "protocol", "fixtures", "mtest-store"), path.join(root, "target", "mtest"), { recursive: true }); + await connection.sendRequest("mcpp-ide/test/list", { root }); + const insights = await connection.sendRequest("mcpp-ide/test/insights", { root }); + assert.equal(insights.runs, 3, JSON.stringify(insights).slice(0, 400)); + + const tab = await browser.newPage(); + const opened = []; + await tab.exposeFunction("mcppHostSend", (text) => opened.push(JSON.parse(text))); + await tab.goto(pathToFileURL(page).href); + const draw = (x) => tab.evaluate(([i, l]) => window.mcppRender({ insights: i, labels: l, dark: false }), [x, labels]); + + await draw(insights); + const text = await tab.locator("#tests").textContent(); + assert.match(text, /Test Insights/); + assert.match(text, /3 runs/); + assert.match(text, /beta/); + assert.match(text, /expected: 4/); + assert.match(text, /ready/); + assert.match(text, /≤ 15 s/); + assert.match(text, /Flaky/); + assert.match(text, /Critical path: unit → smoke/); + assert.equal(await tab.locator("#tests svg.spark").count(), 1); + assert.equal(await tab.locator("#tests g.lane").count(), 2); + assert.ok(await tab.locator("#tests rect.critical").count() >= 2); + await tab.locator("#tests a[data-open]").first().click(); + await tab.waitForTimeout(200); + assert.deepEqual(opened, [{ open: { path: "tests/unit/b.cpp", line: 3 } }]); + + // No run kept: the page says where mcpp keeps them. + await draw(empty); + assert.match(await tab.locator("#tests").textContent(), /no run in .*target\/mtest/); + assert.equal(await tab.locator("#tests svg").count(), 0); + } finally { + await browser.close(); + const ended = new Promise((resolve) => child.on("exit", resolve)); + await connection.sendRequest("shutdown").catch(() => undefined); + await connection.sendNotification("exit").catch(() => undefined); + await ended; + connection.dispose(); + fs.rmSync(root, { recursive: true, force: true }); + } +});