mcpp test: case results as they happen, metrics and bounds, a program's parts, and the store as P12 states it #917

Summary

An editor shows mcpp test through mcpp test --message-format json and the run kept under target/mtest. mcpp-ide's test views (mcpp-ide#8: VS Code, CLion and Neovim) need four additions to these outputs:

• a case's result as it happens;
• a case's metrics and their bounds;
• the parts of a program of several files;
• the fields of the mtest store (spec P12) that a reader of a run uses.

mcpp already reads every one of these from a test program. It does not carry them into the records or the store.

Each addition is a field or record kind added to an existing output (the guarantees docs/50 applies to the test stream: fields are added, never removed). Each is announced by a feature that a client gates on.

Measured (mcpp 2026.11.9.1)

A project with [test] programs = "directory" and two files, tests/unit/a.cpp and tests/unit/b.cpp, each with one mtest case:

The gaps:

• The parts are not stated. The list names unit; the run reports unit/a and unit/b. A client that lists the program cannot tell which records belong to it, or which file a part is, without reproducing the split rule. main names the first file of the directory, which a program of "directory" does not begin at when no file defines main.
• Results arrive per process. src/build/test_protocol.cppm reads MTEST_REPORT after the process ends. The protocol writes the report as NDJSON appended event by event (P5.1), so begin and end are on disk while the program runs. A program of many cases (a conformance or e2e program runs for minutes) shows nothing until it ends.
• Metrics are dropped. The parser keeps program, case, begin, failure, end, missing and done. It ignores metric (P5.2, P6) and usage (P10.2), and the case event's bounds (P6.5). A Bench case's measurements and the limits it states reach neither the record nor the store.
• The store lacks fields P12 lists.
  • cases.ndjson has no kind, started_ms, metrics, bounds or tags.
  • failures has no expected or actual, though the record carries them.
  • programs.ndjson has no build_started_ms or usage.
  • mtest report|analyze|graph --store target/mtest therefore have no kinds, no metrics, no trend and no case-level schedule for a run of mcpp test.

Proposal

1. test-parts: the parts of a program

Under --list and --cases, the record of a program that runs as several processes gains parts:

{"test":"unit","main":null,"files":["tests/unit/a.cpp","tests/unit/b.cpp"],
 "parts":[{"name":"unit/a","file":".../tests/unit/a.cpp"},{"name":"unit/b","file":".../tests/unit/b.cpp"}]}

• files lists the program's own sources: for a program of "directory", every file of it.
• main is the file that defines main, or null when none does.
• parts is how [test] split divides a run ("file": one per file; "none": absent; "case": absent, since a case's process is named by the case).
• A run record of a part gains part_of: "unit".

2. test-case-stream: case results as they happen

mcpp test --message-format json reads MTEST_REPORT while the program runs. For each begin and end it writes a record before the program's record:

{"case":{"member":"","test":"unit/a","id":"alpha","event":"begin","t_ms":12}}
{"case":{"member":"","test":"unit/a","id":"alpha","event":"end","status":"fail","ms":3.1,"reason":"",
         "failures":[{"file":"tests/unit/a.cpp","line":3,"message":"…","expected":"1","actual":"2"}]}}

• t_ms counts from the start of the run, as started_ms does in the store.
• The program's record still follows with all its cases, so a client that reads only program records is unchanged.
• A program killed mid-run leaves its streamed records valid: a begin without an end fails, as P5.4 states.

3. test-metrics: metrics and bounds

• A case in the record's cases gains:
  • metrics: [{"name","value","unit","samples"?,"better"?,"precision"?}];
  • bounds: [{"metric","max"?,"min"?}];
  • kind and tags, from the case's case event or its begin (P3.4).
• The program's record gains usage (P10.2).
• listed entries gain bounds.
• The end record of test-case-stream carries the same fields.

4. The store as P12 states it

• cases.ndjson gains kind, tags, started_ms, metrics and bounds.
• failures entries gain expected and actual.
• programs.ndjson gains build_started_ms and usage.
• mtest analyze trend --store target/mtest then shows a Bench case's metrics beside its bounds over mcpp test's runs.
• mtest graph --run last --store target/mtest --format gantt then draws the schedule at case level.

Features

Consumer

mcpp-ide#8 implements the client side against these names and gates each view on its feature. With an older mcpp, the views are:

• the tree, from the part names;
• results per process;
• no metrics, with the reason in capabilities/explain.

The fixtures of mcpp-ide's mock mcpp record the shapes above. A change of a name or field here is a change there.

Compatibility

Every change adds a field or a record kind:

• a case record has one top-level key, case, beside the stream's test, summary, group_build and workspace_summary records, so a client that dispatches on the keys it knows skips it;
• the store's added fields are P12's, which mtest already reads.

No existing field changes meaning.