Repository navigation
test-perf-hooks-histogram-external-memory failures on node-test-comit-plinux #67374
Description
Test
parallel/test-perf-hooks-histogram-eternal-memory
Platform
Linux PPC64LE
Console output
---
duration_ms: 123.749
exitcode: 1
severity: fail
stack: |-
node:internal/process/promises:324
triggerUncaughtException(err, true /* fromPromise */);
^
AssertionError [ERR_ASSERTION]: Expected a major garbage collection caused by histogram memory
at /home/iojs/build/workspace/node-test-commit-plinux/nodes/rhel8-power9le/test/parallel/test-perf-hooks-histogram-external-memory.js:36:10 {
generatedMessage: false,
code: 'ERR_ASSERTION',
actual: false,
expected: true,
operator: '==',
diff: 'simple'
}
Node.js v28.0.0-pre
...Build links
Looking at recent runs of the PR tester I have noticed that there is a high proportion of failures due to the plinux job (Links not public visible).
Looking at the most recent failures - it most mostly on a single machine, but not exclusively so:
| Job | machine | PR | Failure |
|---|---|---|---|
| 67470 | osu-rhel8-5 | 67046 | parallel/test-perf-hooks-histogram-external-memory |
| 67473 | osu-rhel8-5 | 67241 | parallel/test-perf-hooks-histogram-external-memory.js |
| 67475 | osu-rhel8-4 | 67359 | parallel/test-perf-hooks-histogram-external-memory.js |
| 67481 | osu-rhel8-4 | 66344 | parallel/test_perf_hooks_histogram_external_memory/ |
| 67482 | osu-rhel8-4 | 67245 | parallel/test_perf_hooks_histogram_external_memory |
| 67487 | osu-rhel8-4 | 67359 | parallel/test_perf_hooks_histogram_external_memory |
| 67489 | osu-rhel10-1 | 66732 | parallel/test_perf_hooks_histogram_external_memory |
| 67491 | osu-rhel8-5 | ? | parallel/test_perf_hooks_histogram_external_memory |
| 67493 | osu-rhel8-4 | 66726 | parallel/test_perf_hooks_histogram_external_memory |
| 67494 | osu-rhel8-4 | 66616 | parallel/test_perf_hooks_histogram_external_memory |
| 69495 | osu-rhel8-5 | 65794 | parallel/test_perf_hooks_histogram_external_memory |
It looks like today's reliability report is the first in a while to mention it suggesting it was potentially a recent change that has caused it.
Additional information
No response
Activity
sxa commentedon Nov 10, 2026
The first failure was in https://ci.nodejs.org/job/node-test-commit-plinux/67470/ at 0722 UTC yesterday on the
-5machine. The last good run on the-4machine (where the problem seems most prolific) was at https://ci.nodejs.org/job/node-test-commit-plinux/nodes=rhel8-power9le/67464/testReport/(root)/parallel/ at 2035 the previous day.If it was down to a code change that would suggest it could be one of these:
I'm going to try some tests on the most problematic machine but I'll also tag @mcollina in case there's any possibility that one of those made a change that could cause a problem in that test.
[-]`test-perf-hooks-histogram-eternal-memory` failures on node-test-comit-plinux[/-][+]`test-perf-hooks-histogram-external-memory` failures on node-test-comit-plinux[/+]sxa commentedon Nov 10, 2026
I have temporarily disabled the
rhel8-power9lelabel on test-osuosl-rhel8-ppc64_le-4 to reduce the error rate on these jobs.sxa commentedon Nov 10, 2026
Noting also that on the main test results page of node-test-commit-plinux there is a difference in whether the tests section shows the test case which fails e.g.:
https://ci.nodejs.org/job/node-test-commit-plinux/nodes=rhel8-power9le/67470/
https://ci.nodejs.org/job/node-test-commit-plinux/nodes=rhel8-power9le/67473/
In both cases it was the same failure on the same machine, but there was a difference in how it was reported.
sxa commentedon Nov 10, 2026
test-osuosl-rh1l0-ppc64_le-1 failed the test 19 iterations out of 1000.
mcollina commentedon Nov 10, 2026
not of my knowledge
sxa commentedon Nov 10, 2026
Testing on
test-osuosl-rhel10-ppc64_le-1with a "manual bisect" since it was quite a small timeframe:eb3ae472b6- failed 9/10082940d6fa5(just prior to the ones listed above) - Passed 100/1009aa379e22f(just after the ones listed above0 - Failed 7/1007ae68bd0f4passed 100/100f344e7f552failed 9/100The last two lines definitely point the finger at the prelead streams into startup snapshot change.
I've just re-tested the one before it
7ae68bd0f4and it passes 1000/1000 iterationssxa commentedon Nov 10, 2026
Fired some AI at it and it suggests increasing the histogram count in the test case to increase memory pressure to ensure that the GC event is triggered. I have confirmed that increasing the invocations of
histogram.snapshot()to 800 from 400 in the test allows it to pass reliably so given the analysis below - specifically the "race condition" section - it seems plausible that this is a valid fix. I will prepare a PR, but I want to verify that it works with a high number of iterations on a RHEL8 system (ideally-4based on the initial analysis of failures)The last PR test run executed on osu-rhel8-4 and passed ok at https://ci.nodejs.org/job/node-test-commit-plinux/nodes=rhel10-ppc64le/67464/ - the earlier runs have already removed themselves from the CI.
Root Cause Analysis
The intermittent failure on Linux/ppc64le is caused by PR #66311 which preloads streams into the startup snapshot. This change affects V8's external memory accounting and GC triggering specifically on the PowerPC architecture.
Why It Fails
1. Memory Accounting Timing:
When streams are preloaded into the startup snapshot, the initial memory layout and V8's external memory accounting are altered. On PPC64LE, this appears to affect when V8 recognizes memory pressure from external allocations.
2. Architecture-Specific Behavior:
PPC64LE has different:
These differences can affect V8's GC heuristics and threshold calculations.
3. Race Condition:
The test creates 400 histogram snapshots (~140MB native memory) and expects this external memory pressure to trigger a major GC. With the snapshot changes, the timing of when V8 accounts for this external memory may have shifted, creating a race condition where the test completes before the GC event is observed.
Probable Root Cause
The snapshot preloading changes how V8 initializes its heap and external memory tracking. On PPC64LE, this initialization difference causes V8 to either:
sxa commentedon Nov 10, 2026
Note: This is not only visible on plinux - here is a failure of that test on linuxone (Linux/s390x) https://ci.nodejs.org/job/node-test-commit-linuxone/59810/nodes=rhel8-s390x/