diff --git a/.github/workflows/core-candidate.yml b/.github/workflows/core-candidate.yml index 0b8d962c..5f1ba36b 100644 --- a/.github/workflows/core-candidate.yml +++ b/.github/workflows/core-candidate.yml @@ -56,6 +56,8 @@ jobs: python-version: '3.12' - name: Install fixed semantic references run: python -m pip install --require-hashes -r experiments/differential/requirements.txt + - name: Install fixed CPU observer + run: python -m pip install -r benchmarks/requirements.txt - name: Select fixed Rust toolchain run: rustup default 1.92.0 - name: Build locked candidate diff --git a/CMakeLists.txt b/CMakeLists.txt index 5825ca6b..61d22796 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -292,7 +292,7 @@ if(BUILD_TESTING) endif() termforge_add_python_test(core -m unittest experiments.core.test_core experiments.core.test_unicode -v) termforge_add_python_test(recovery -m unittest experiments.core.test_recovery -v) - termforge_add_python_test(measure -m unittest experiments.measure.test_measure experiments.measure.test_r54_contract.CostPlanTests experiments.measure.test_r54_batch experiments.measure.test_r54_hosted -v) + termforge_add_python_test(measure -m unittest experiments.measure.test_measure experiments.measure.test_r54_contract.CostPlanTests experiments.measure.test_r54_batch experiments.measure.test_r54_hosted experiments.measure.test_r54_cpu_diagnostic -v) termforge_add_python_test(binary_paths -m unittest experiments.test_build_paths -v) termforge_add_python_test(conpty_margin_contract -m unittest experiments.conpty.test_margin_contract -v) termforge_add_python_test(benchmark_tools -m unittest discover -s benchmarks/tests -v) diff --git a/docs/EVIDENCE.md b/docs/EVIDENCE.md index e7402838..566598b7 100644 --- a/docs/EVIDENCE.md +++ b/docs/EVIDENCE.md @@ -164,5 +164,6 @@ python -m unittest experiments.shell.test_workflow_sessions -v | 候选 `.3`→`.4` 字素改动 | [固定字素验证](../experiments/shell/results/2026-10-26-streaming-graphemes/validation.json)、[性能限制与原始样本身份](research/streaming-graphemes.md)、[#54](https://github.com/JavaZeroo/termforge/issues/54) | 同宿主交错诊断中 project/interactive 吞吐中位数约下降 16%/9%；共享宿主、allocator 包装、无真实呈现/输入时延，使其不能作为正式预算或性能通过。#54 受控定位仍阻塞 #34 的性能裁决。 | | 固定字素成本的批次协议与执行 | [R54 v1 合同和构建/环境记录](../experiments/measure/R54-COST.md)、[执行器](../experiments/measure/r54_batch.py)、[合成进程回归](../experiments/measure/test_r54_batch.py)、[实际预检及限定验证](../experiments/measure/results/2026-10-27-r54-runner/validation.json) | 固定两个种子、各 112 个位置，字节原样保留、首错停止、完整批次后逐组配对统计。共享工作站十区间活动预检未通过，未启动成本样本；测量相关 26 项测试通过，首次 Linux 候选 CI 的采样依赖导入错误及修正分别留证。5% 是环境诊断护栏，并非产品性能容差；实际批次、扰动归因及用户预算确认由 #54 接续。 | | 首次 hosted 字素成本批次 | [来源、停止位置与归档](../experiments/measure/results/2026-10-27-r54-hosted/README.md)、[机器可读记录](../experiments/measure/results/2026-10-27-r54-hosted/validation.json) | 两份固定构建与4次sanity通过，空闲预检通过；第24位置的未归属CPU活动估计5.410%触发原5%护栏，保留23个有效记录和1个失败记录，第二批未启动。全部24次输出正确，但批次不完整，不生成配对成本摘要；临时VM逐进程归因、完整受控批次和预算仍未完成。 | +| CPU计量边界诊断（#54） | [固定六位置及归档](../experiments/measure/results/2026-10-27-r54-cpu-boundary/README.md)、[独立复核](../experiments/measure/results/2026-10-27-r54-cpu-boundary/verify.py) | 同一共享Windows宿主、原已审计构建的六次输出和来源核验通过；分别保留完整子进程、内层区间、runner和全机计数。区间外CPU为0.078125–0.109375s，仍有明显未归属活动，不算空闲成本批次，不反推旧hosted原因或修改原5%护栏。 | 目前没有足够证据判断 TermForge 距离 WindTerm 快或慢多少。正式结论必须同时提供负载/配置/环境身份、正确性、完成端点、观测开销、重复样本与限制。未知字段保持未知；历史校准未完成不转成要求用户反复手动运行的前置。执行规则见 [ADR-0002](adr/0002-automation-first-investigation.md)，核心选型重审见 [ADR-0003](adr/0003-core-reference-and-reuse.md)。 diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 6619b2a3..9f3c415e 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -71,7 +71,7 @@ | 自动正确性与构建 | **限定验收**：Windows shell/ConPTY/呈现及 Windows/Linux 核心 CI、差分/恢复/队列/生命周期场景 | 产品打包、干净机器使用、依赖升级、长期回归和更广平台矩阵；#40 的 CI 纯夹具超时、窄窗口选择 #73、四 shell 原事件 #80、hosted 准备 #81 和[复制夹具 OpenClipboard 5 #85](https://github.com/JavaZeroo/termforge/issues/85) 等独立失败仍未闭合；#72 保存判据与 #80 同步条件的限定修正不代替其他配置或未知根因验收 | R2 / R4 | | Windows 产品交付 | **实现中，未验收**：首个 [v0.1.0-alpha.1 预发布](releases/v0.1.0-alpha.1.md)提供 Windows x64、静态 MSVC 运行库、附资源许可和构建身份的免安装 ZIP；未签名，已验环境限 Windows 11 本机与 Server 2025 CI，布局仍写入 LOCALAPPDATA | 这是 R1 体验包，尚非 R2 日常开发版；干净机器与更广平台矩阵、签名/安装、配置升级及持续运行仍待完成。Windows 10 未验收，Server 2022 已知资源门槛未过 | R2 / R3 | | Linux/macOS 与其他架构 | **未实现**客户端；Linux 核心 CI 已有。[官方平台范围台账](../research/windterm-2.7.0-platform-scope.md)已固定 2.7.0 的 Windows x86/x64、macOS x64、Linux x64 资产及声明下限；这不是 TermForge 支持或实测结论 | 最终需真实 Linux/macOS 客户端；平台调查可提前进行，整合及完整验收排在 Windows 11 x64 完整版本之后。其他系统版本、最低版本和架构由用户后续明确；旧系统及 ARM 等保持调查记录，不自动承诺或排除，也不能由核心 CI 推断产品支持 | 持续调查；R5 全平台出口 | -| 吞吐、响应、内存与稳定性 | **实现中，未验收**：已有机制实验、连续 CPU 区间校准与合成回归；候选字素改动的受限诊断中 project/interactive 吞吐中位数约下降 16%/9%，由 [#54](https://github.com/JavaZeroo/termforge/issues/54) 跟踪。[冻结批次执行器](../experiments/measure/R54-COST.md#冻结批次执行器windows)核对构建来源、原始输出、停止规则与逐轮配对统计；[独立 hosted 构建与实测](../experiments/measure/results/2026-10-27-r54-hosted/README.md)已验证来源和失败保留，但首批因环境护栏停止，尚无完整成本批次 | 旧诊断的共享宿主、allocator 包装与缺少真实呈现/输入时延端点使其不构成正式预算；新的不完整批次也不作吞吐结论。仍需同负载/同正确性、无插桩对照、观测开销和长期/多会话公平性；可靠原版基线后提出逐项预算及允许波动范围，**由用户明确确认**。与 WindTerm 的实际差距**待取证** | 当前优先；贯穿 R1–R5 | +| 吞吐、响应、内存与稳定性 | **实现中，未验收**：已有机制实验、连续 CPU 区间校准与合成回归；候选字素改动的受限诊断中 project/interactive 吞吐中位数约下降 16%/9%，由 [#54](https://github.com/JavaZeroo/termforge/issues/54) 跟踪。[冻结批次执行器](../experiments/measure/R54-COST.md#冻结批次执行器windows)核对构建来源、原始输出、停止规则与逐轮配对统计；[独立 hosted 构建与实测](../experiments/measure/results/2026-10-27-r54-hosted/README.md)已验证来源和失败保留，但首批因环境护栏停止，尚无完整成本批次；[独立CPU边界诊断](../experiments/measure/results/2026-10-27-r54-cpu-boundary/README.md)另记完整子进程/执行器分量，不改原护栏或判成本通过 | 旧诊断的共享宿主、allocator 包装与缺少真实呈现/输入时延端点使其不构成正式预算；新的不完整批次也不作吞吐结论。仍需同负载/同正确性、无插桩对照、观测开销和长期/多会话公平性；可靠原版基线后提出逐项预算及允许波动范围，**由用户明确确认**。与 WindTerm 的实际差距**待取证** | 当前优先；贯穿 R1–R5 | | 大历史与动态内存压缩 | **未实现**产品能力；WindTerm 压缩属于官方性能声明，内部机制尚未证实 | 1/10/100 万行等明确规模下的内存/滚动/搜索/淘汰实验；根据瓶颈决定独立存储/压缩实现 | R4 | | 完整目标验收 | **待取证**：已有固定版本和局部观察，没有完整行为/性能对照矩阵 | 先在 Windows 11 x64 完整验收功能/行为/UI/性能及独立构建，再完成 Linux/macOS 全平台出口；重要差异和性能容差须有用户确认，Windows 通过不代表全平台完成；不依赖 WindTerm 二进制或专有资源 | R5 分平台出口 | diff --git a/experiments/measure/R54-CPU-DIAGNOSTIC.md b/experiments/measure/R54-CPU-DIAGNOSTIC.md new file mode 100644 index 00000000..49f54d10 --- /dev/null +++ b/experiments/measure/R54-CPU-DIAGNOSTIC.md @@ -0,0 +1,37 @@ +# R54：CPU 计量边界的独立诊断 + +本项接续 [#54](https://github.com/JavaZeroo/termforge/issues/54) 的首次 hosted 停止事件：原批次第24个位置按 v1 口径得到 5.410% 未归属活动估计，超过原5%环境护栏。原记录没有整个子进程及执行器CPU，无法事后区分自身区间外工作和其他guest活动。该事件、失败状态、[v1合同](R54-COST.md)及两批各112位置均保持；此诊断不补样、不重试旧批次、不判性能通过。 + +## 预声明问题与执行范围 + +按以下预测分别记录，而不是先假定护栏误报： + +1. 若自身区间外工作有贡献，完整子进程CPU会高于其内部连续测量区间CPU；差值包括启动、准备、oracle、输出和退出，不能仅凭差值细分各阶段。 +2. 若执行器有贡献，其同一外部区间的进程CPU会大于零。 +3. 扣除前两项后的余额仍可能包含其他guest活动、系统归账粒度和非原子采样边界；余额保留正负，不称为精确后台进程CPU。物理宿主争用和频率仍未知。 + +固定一次六位置序列为 `.3, .4, .4, .3, .3, .4`，全部使用原 `interactive` 22,000B 语料、chunk=1、observe=0、内部CPU/墙钟至少2秒及原正确性校验。沿用已审计的共同harness与两份无allocator包装EXE，不重新构建。只比较计量范围，**不计算A/B成本比或吞吐变化，也不运行空闲合格筛选**。允许在如实声明的共享宿主研究这些计数；这不把共享宿主升级为成本验收环境。 + +每个进程90秒期限，超时终止直接子进程并最多等待5秒；退出未确认、计数缺失、输出错误或来源变化立即停止，不补跑。固定核心EXE不派生进程，本工具只支持这个直接子进程边界，不宣称通用进程树计量或回收。诊断状态 `accounting_complete` 只说明六次记录/正确性/来源完整，始终 `performance_gate=not_evaluated`。 + +## 计数与证据 + +`GetSystemTimes` 先取全机计数，再取runner的 `process_time`，然后启动并等待直接子进程；退出后通过保留的进程句柄读取 `GetProcessTimes`，再取runner和全机末计数。全机计数只支持1–64逻辑CPU。两个采样端点不是原子操作，读取进程计数本身也有成本；诊断不声称无扰动。 + +每个观察保留全机原始前后计数、runner前后CPU、子进程创建/退出FILETIME及user/kernel/总CPU、原始stdout/stderr、内层区间和以下余额： + +- `inner_only_balance = guest_busy - inner_cpu`：本次插桩区间的内层单项扣除，不冒充旧批次的原值。 +- `child_outside_inner = child_lifetime_cpu - inner_cpu`。 +- `unassigned_balance = guest_busy - child_lifetime_cpu - runner_cpu`：保留负值，不据此修正成本或设新护栏。 + +Windows进程对象在外部仍持有句柄时可于退出后查询，依据 [GetProcessTimes](https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-getprocesstimes) 和[进程终止语义](https://learn.microsoft.com/en-us/windows/win32/procthread/terminating-a-process)。诊断限定Windows CPython3.12，通过Popen持有的句柄避免退出后按PID查询；该私有接口依赖及句柄生命周期按[CPython3.12.10源码](https://github.com/python/cpython/blob/v3.12.10/Lib/subprocess.py)核对，不能静默扩展到其他runtime。 + +入口要求已提交且干净的工具源码、显式环境声明、原计划及完整构建包，运行前后复核全部来源、编译产物及实际`--identity`。输出目录必须不存在；过程、失败和中断记录保留。原始本地路径和PID留local，公开时单独审核白名单、路径脱敏和原字节流哈希，不递归上传工作目录。 + +```powershell +python -m experiments.measure.r54_cpu_diagnostic ` + --plan <原plan-v1.json> --build-bundle <原构建包> ` + --environment <含host_kind与known_concurrent_work的JSON> --out <全新目录> +``` + +本次结果只回答声明宿主上的计量问题，不能补造旧hosted事件的原因。是否改变后续环境测量方法须先依据诊断另定协议，不能倒改原v1或用新口径将旧失败改绿。完整成本批次、allocator/projection及组件归因、数值预算提案与用户确认仍由 #54 接续。 diff --git a/experiments/measure/r54_cpu_diagnostic.py b/experiments/measure/r54_cpu_diagnostic.py new file mode 100644 index 00000000..6b04b424 --- /dev/null +++ b/experiments/measure/r54_cpu_diagnostic.py @@ -0,0 +1,236 @@ +"""Separate whole-child CPU accounting from R54's unchanged v1 cost batches.""" +from __future__ import annotations + +import argparse +import ctypes +from ctypes import wintypes +import json +import math +import os +from pathlib import Path +import subprocess +import sys +import time + +from experiments.measure.r54_batch import capture, sha, utc, write_json +from experiments.measure.r54_contract import CPU_SECONDS, ROOT, SOURCES, validate_sample +from experiments.measure.r54_environment import cpu_delta, cpu_snapshot, host_identity +from experiments.measure.r54_inputs import IDENTITY, git, load_builds, read, require + +PROTOCOL = dict(schema='termforge-r54-cpu-boundary-v1', + order=['baseline', 'candidate', 'candidate', 'baseline', 'baseline', 'candidate'], + route='interactive', chunk=1, observe=0, inner_cpu_seconds=CPU_SECONDS, + process_timeout_seconds=90, cleanup_timeout_seconds=5, + activity_acceptance_gate=False, performance_gate='not_evaluated', + purpose='separate whole-child, inner interval and runner CPU; no cost ratios', + residual_scope='other guest activity plus counter quantization and bracket mismatch; not exact attribution') + + +def partition_cpu(record, inner_cpu): + """Expose balances without labelling the signed residual external activity. + + Counter updates and non-atomic bracket endpoints can leave a negative + residual. Keep it visible; do not clamp, normalize or subtract it from cost. + """ + total = record['environment']['total_seconds'] + busy = record['environment']['busy_seconds'] + child = record['child_cpu']['total_seconds'] + runner = record['runner_cpu_seconds'] + if (any(not math.isfinite(value) or value < 0 for value in (total, busy, child, runner, inner_cpu)) + or total <= 0 or child < inner_cpu): + raise ValueError('inconsistent whole-child or CPU accounting') + residual = busy - child - runner + return dict(inner_only_balance_seconds=busy - inner_cpu, + inner_only_balance_percent=(busy - inner_cpu) * 100 / total, + child_outside_inner_seconds=child - inner_cpu, + runner_cpu_seconds=runner, unassigned_balance_seconds=residual, + unassigned_balance_percent=residual * 100 / total) + + +def exited_child_cpu(process): + """Borrow CPython's retained Windows process handle after wait, before close. + + The handle belongs to Popen. Querying by PID after exit would lose the + lifetime or risk a reused identity; retaining this handle avoids that race. + This private CPython dependency is limited to the pinned diagnostic runtime. + """ + if process.returncode is None: + raise ValueError('child must have exited before lifetime accounting') + creation, end, kernel, user = (wintypes.FILETIME() for _ in range(4)) + function = ctypes.WinDLL('kernel32', use_last_error=True).GetProcessTimes + function.argtypes = [wintypes.HANDLE] + [ctypes.POINTER(wintypes.FILETIME)] * 4 + function.restype = wintypes.BOOL + if not function(int(process._handle), ctypes.byref(creation), ctypes.byref(end), + ctypes.byref(kernel), ctypes.byref(user)): + raise ctypes.WinError(ctypes.get_last_error()) + ticks = lambda value: (value.dwHighDateTime << 32) | value.dwLowDateTime + return dict(creation_filetime=ticks(creation), exit_filetime=ticks(end), + kernel_seconds=ticks(kernel) / 10_000_000, + user_seconds=ticks(user) / 10_000_000, + total_seconds=(ticks(kernel) + ticks(user)) / 10_000_000) + + +def capture_cpu(command, output, timeout=90): + """Capture a direct child and its complete CPU lifetime, including exit. + + The fixed measurement executables do not spawn children. This helper does + not claim descendant accounting or cleanup. Stream bytes are never decoded + or rewritten here, including failed and timed-out executions. + """ + if os.name != 'nt' or sys.implementation.name != 'cpython' or sys.version_info[:2] != (3, 12): + raise ValueError('CPU diagnostic requires Windows CPython 3.12') + output = Path(output) + result = dict(command=command, started_utc=utc(), exit_code=None, timed_out=False, + child_cpu=None, clock='GetProcessTimes kernel plus user', + runtime=sys.version, accounting_scope='direct child only') + with (output / 'stdout.bin').open('xb') as stdout, (output / 'stderr.bin').open('xb') as stderr: + before = cpu_snapshot() + runner_before = time.process_time() + try: + # Popen.__exit__ waits without a bound. Keep the object alive for + # accounting, but use explicit bounded waits on failure instead. + child = subprocess.Popen(command, stdin=subprocess.DEVNULL, stdout=stdout, stderr=stderr) + result['child_pid'] = child.pid + try: + child.wait(timeout=timeout) + except subprocess.TimeoutExpired: + result.update(timed_out=True, error=f'direct child timed out after {timeout}s') + child.kill() + child.wait(timeout=PROTOCOL['cleanup_timeout_seconds']) + except BaseException: + child.kill() + child.wait(timeout=PROTOCOL['cleanup_timeout_seconds']) + raise + result['exit_code'] = child.returncode + result['child_cpu'] = exited_child_cpu(child) + except (OSError, subprocess.SubprocessError) as error: + result['error'] = f'{type(error).__name__}: {error}' + finally: + runner_after = time.process_time() + after = cpu_snapshot() + result['runner_cpu_seconds'] = runner_after - runner_before + result['runner_cpu_before_seconds'] = runner_before + result['runner_cpu_after_seconds'] = runner_after + result['environment'] = dict(cpu_delta(before, after), before=before, after=after) + result['finished_utc'] = utc() + result['streams'] = {name: dict(bytes=(output / name).stat().st_size, sha256=sha(output / name)) + for name in ('stdout.bin', 'stderr.bin')} + return result + + +def run_observations(commands, output, context): + """Process/artifact seam for the fixed six-observation diagnostic only.""" + require(set(commands) == set(SOURCES), 'fixed source pair required') + output = Path(output) + output.mkdir(parents=True, exist_ok=False) + write_json(output / 'context.json', context) + write_json(output / 'protocol.json', PROTOCOL) + manifest = dict(schema='termforge-r54-cpu-observations-v1', status='running', + started_utc=utc(), attempted=0, accounted_observations=0, + context_sha256=sha(output / 'context.json'), protocol_sha256=sha(output / 'protocol.json'), + performance_gate='not_evaluated') + try: + for sequence, variant in enumerate(PROTOCOL['order']): + item = dict(sequence=sequence, variant=variant, route=PROTOCOL['route'], chunk=PROTOCOL['chunk']) + directory = output / f'{sequence:03d}' + directory.mkdir() + record = dict(item=item, status='started') + manifest.update(attempted=sequence + 1, active_sequence=sequence) + write_json(output / 'manifest.json', manifest) + write_json(directory / 'record.json', record) + try: + command = list(commands[variant]) + [item['route'], str(item['chunk']), '0', str(CPU_SECONDS)] + before_sha = sha(command[0]) + if 'binding' in context: + require(before_sha == context['binding']['builds'][variant]['binary']['sha256'], + 'binary changed before observation') + record.update(capture_cpu(command, directory, PROTOCOL['process_timeout_seconds'])) + require(not record['timed_out'] and record['exit_code'] == 0 and 'error' not in record, + record.get('error', 'nonzero child exit')) + result = json.loads((directory / 'stdout.bin').read_bytes().decode('utf-8')) + validate_sample(result, item) + require(result['cpu_clock'] == 'GetProcessTimes kernel plus user', 'wrong child CPU clock') + require(sha(command[0]) == before_sha, 'binary changed during observation') + record['result'] = result + record['accounting'] = partition_cpu(record, result['interval_cpu_seconds']) + record['status'] = 'accounted' + except Exception as error: + record.update(status='failed', error=f'{type(error).__name__}: {error}') + finally: + write_json(directory / 'record.json', record) + require(record['status'] == 'accounted', record.get('error', 'observation incomplete')) + manifest['accounted_observations'] += 1 + manifest.update(status='accounting_complete', active_sequence=None) + except Exception as error: + manifest.update(status='failed', error=f'{type(error).__name__}: {error}') + except BaseException: + manifest['status'] = 'interrupted' + raise + finally: + manifest['finished_utc'] = utc() + write_json(output / 'manifest.json', manifest) + return manifest + + +def execute(plan, bundle, declaration, output): + """Audit retained builds; do not rebuild or repeat an old preflight/batch.""" + require(os.name == 'nt', 'Windows accounting required') + output = Path(output) + output.mkdir(parents=True, exist_ok=False) + manifest = dict(schema='termforge-r54-cpu-diagnostic-v1', status='preparing', started_utc=utc(), + performance_gate='not_evaluated') + try: + runner = dict(commit=git(ROOT, 'rev-parse', 'HEAD'), worktree_status=git(ROOT, 'status', '--porcelain'), + sources={name: sha(ROOT / name) for name in ( + 'experiments/measure/r54_cpu_diagnostic.py', 'experiments/measure/r54_environment.py', + 'experiments/measure/r54_batch.py', 'experiments/measure/r54_contract.py', + 'experiments/measure/r54_inputs.py', 'experiments/measure/run.py')}) + write_json(output / 'runner.json', runner) + require(not runner['worktree_status'], 'runner source must be committed and clean') + environment = read(declaration) + require(environment.get('host_kind') in ('shared_workstation', 'hosted_vm', 'dedicated_host') and + isinstance(environment.get('known_concurrent_work'), list), 'explicit environment declaration required') + write_json(output / 'environment.json', dict(host=host_identity(), declaration=environment, + scope='accounting diagnosis on declared host; no idle qualification or cost acceptance')) + binding = load_builds(plan, bundle) + write_json(output / 'binding.json', binding) + (output / 'plan-v1.json').write_bytes(Path(plan).read_bytes()) + for variant, command in binding['commands'].items(): + directory = output / (variant + '-identity') + directory.mkdir() + value = capture(command + ['--identity'], directory, timeout=10) + write_json(directory / 'record.json', value) + require(value['exit_code'] == 0 and read(directory / 'stdout.bin') == IDENTITY, 'actual build identity mismatch') + result = run_observations(binding['commands'], output / 'observations', + dict(evidence_kind='cpu_boundary_diagnostic', binding=binding, runner=runner)) + manifest['observations'] = result + require(result['status'] == 'accounting_complete', 'diagnostic stopped; no replacement observations') + require(load_builds(plan, bundle) == binding, 'build binding changed during diagnosis') + for name, digest in runner['sources'].items(): + require(sha(ROOT / name) == digest, 'runner changed during diagnosis') + manifest['status'] = 'accounting_complete' + except Exception as error: + manifest.update(status='failed', error=f'{type(error).__name__}: {error}') + except BaseException: + manifest['status'] = 'interrupted' + raise + finally: + manifest['finished_utc'] = utc() + write_json(output / 'manifest.json', manifest) + return manifest + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--plan', type=Path, required=True) + parser.add_argument('--build-bundle', type=Path, required=True) + parser.add_argument('--environment', type=Path, required=True) + parser.add_argument('--out', type=Path, required=True) + args = parser.parse_args() + result = execute(args.plan, args.build_bundle, args.environment, args.out) + print(json.dumps(result)) + raise SystemExit(0 if result['status'] == 'accounting_complete' else 1) + + +if __name__ == '__main__': + main() diff --git a/experiments/measure/results/2026-10-27-r54-cpu-boundary/README.md b/experiments/measure/results/2026-10-27-r54-cpu-boundary/README.md new file mode 100644 index 00000000..338d62ab --- /dev/null +++ b/experiments/measure/results/2026-10-27-r54-cpu-boundary/README.md @@ -0,0 +1,36 @@ +# CPU 计量边界：一次共享宿主诊断 + +按[预声明合同](../../R54-CPU-DIAGNOSTIC.md)，在共享Windows11宿主完成一次固定六位置诊断，所有进程退出0、输出和完整来源前后核验通过。**这不是原112位置成本批次，也没有性能通过结论。** 原hosted第24位置失败保留，当前结果不能反推其具体原因。 + +实际执行源码为本地提交 `ff0d7caca138b91683f16ce8d748eea6fbb289b7`，树 `796005cdcd0e330ae86d676ec5386125459d733d`；开始时工作树干净。运行时间为2026-10-27 12:17:43–12:18:00 UTC。完整工具源哈希在归档`runner.json`，发布版本中的执行代码与它逐文件一致；后续仅调整测试的runtime跳过边界及增加本记录。 + +环境为Windows11 build26200/UBR9711/25H2、Intel i7-9700、8逻辑/8物理CPU、CPython3.12.10、psutil7.2.2。明确共享宿主，未调整个人设置，未执行空闲资格筛选。复用[原构建记录](../2026-10-27-r54-runner/validation.json)中已审计的 `.3/.4` EXE；两份来源、harness、build命令/日志/覆盖、80/84文件身份、linker与实际`--identity`在运行前后重新校验。没有重建，二进制与旧hosted的两份构建不同。 + +下表均为CPU秒；余额比例的分母是全机总CPU区间，所有位置固定interactive/chunk1/observe0。它不报告吞吐或A/B成本比。 + +| 位置 | 核心 | 内部区间 | 完整子进程 | 区间外差值 | runner | 未归属余额 | 余额占比 | +|---|---|---:|---:|---:|---:|---:|---:| +| 0 | .3 | 2.125000 | 2.203125 | 0.078125 | 0 | 9.796875 | 49.920% | +| 1 | .4 | 2.140625 | 2.234375 | 0.093750 | 0.015625 | 8.859375 | 44.575% | +| 2 | .4 | 2.203125 | 2.296875 | 0.093750 | 0 | 6.656250 | 34.578% | +| 3 | .3 | 2.156250 | 2.265625 | 0.109375 | 0.015625 | 9.343750 | 47.611% | +| 4 | .3 | 2.125000 | 2.203125 | 0.078125 | 0 | 7.359375 | 38.231% | +| 5 | .4 | 2.031250 | 2.109375 | 0.078125 | 0.015625 | 9.812500 | 50.974% | + +此宿主确有可见的自身区间外CPU和runnerCPU，单扣内层区间不能准确区分这些部分。但完整扣除后的余额仍很大；原始计数和非原子采样可能有粒度差异，不把余额归因到某个后台进程，也不声称空闲、无扰动或物理宿主独占。下一步应在另行声明的独立hosted诊断中采用相同分量记录；不重复本地批次取绿，不调整原5%护栏，不将本诊断混入原正式样本。 + +相关measurement本地回归36/36（18.660s），新增真实进程/原字节产物边界及固定算例覆盖自身区间外CPU、超时、首错停止、不可覆盖与负余额。首次合成夹具使用venv重定向启动器，暴露了直接子进程与实际解释器的边界；改用实际解释器并核对子PID后验证通过，没有把派生进程CPU算进直接子进程。测试随后限定Windows CPython3.12、1–64逻辑CPU的native诊断；其他支持的环境仍可运行项目普通测试，并跳过这三个专属用例，跳过不表示计量通过。 + +runtime/CPU适用条件修正后的同组回归仍为36/36（17.069s）。实际执行代码与原测量源码哈希保持一致，没有重跑六位置诊断。 + +首次发布 `c76bab0` 的默认[PR](https://github.com/JavaZeroo/termforge/actions/runs/40504397868)/[push](https://github.com/JavaZeroo/termforge/actions/runs/40504336483)均通过（Windows46/46、Linux29/29）；候选[PR](https://github.com/JavaZeroo/termforge/actions/runs/40504397941)/[push](https://github.com/JavaZeroo/termforge/actions/runs/40504336888)的Linux measurement为41通过/4跳过，Windows因未安装`psutil`在导入新测试时失败。原失败保留；修正候选工作流安装仓库已有固定`benchmarks/requirements.txt`，不跳过缺少依赖的测试。干净Python环境复现同一缺包错误，安装后4个新测试通过（1.583s）。修订CI单独跟踪，不将原两条候选失败写成通过。 + +[归档](evidence.zip)为100053B，SHA256 `9bec0731e08d7f2ef99d140a9e56acbc01468814cc68fb01fbdcfe30eaa8e10c`。内含46份清单证据及`publication.json`，其中16份stdout/stderr按原字节保留（12份观察、4份身份）。路径改为别名，子PID移除；清单分别保留原始/公开大小和SHA，原元数据里的来源哈希不回填成脱敏后哈希。未上传EXE、进程清单、任意环境变量或私有文件；完整本地原件保留。 + +离线复核命令，不执行任何测量： + +```powershell +python experiments/measure/results/2026-10-27-r54-cpu-boundary/verify.py +``` + +完整受控批次、allocator/projection及组件归因、预算提案与用户确认仍未完成；#54、#34与R2保持开放。 diff --git a/experiments/measure/results/2026-10-27-r54-cpu-boundary/evidence.zip b/experiments/measure/results/2026-10-27-r54-cpu-boundary/evidence.zip new file mode 100644 index 00000000..0414e68a Binary files /dev/null and b/experiments/measure/results/2026-10-27-r54-cpu-boundary/evidence.zip differ diff --git a/experiments/measure/results/2026-10-27-r54-cpu-boundary/validation.json b/experiments/measure/results/2026-10-27-r54-cpu-boundary/validation.json new file mode 100644 index 00000000..20b89e3e --- /dev/null +++ b/experiments/measure/results/2026-10-27-r54-cpu-boundary/validation.json @@ -0,0 +1,453 @@ +{ + "schema": "termforge-r54-cpu-boundary-validation-v1", + "performance_gate": "not_evaluated", + "measured_source": { + "commit": "ff0d7caca138b91683f16ce8d748eea6fbb289b7", + "worktree_status": "", + "sources": { + "experiments/measure/r54_cpu_diagnostic.py": "3071ae52fbddc0e080b586a78837498bdc2105072471ac12c6a48d15f5eb032e", + "experiments/measure/r54_environment.py": "60375a50e83637d27210d5ac0ff958cb0191cb65782fe91e69a741655c1d51f2", + "experiments/measure/r54_batch.py": "b7e19a85797b6d31c22d32eb9878e1975bfe733fdb09aa289da70fe0680e5b68", + "experiments/measure/r54_contract.py": "08c744f3d78e23439fecd8cc01348fdc1673c5397e16738266774fb49fbc76fa", + "experiments/measure/r54_inputs.py": "6e5430681254df364c8abc6b662969ca00991027878708ecdd3ba6f4e96f4eed", + "experiments/measure/run.py": "5643194c95c710790bee3a10e825cd338626222e88a9e2af194ad905b0718852" + } + }, + "measured_tree": "796005cdcd0e330ae86d676ec5386125459d733d", + "baseline": "ed0c12597d3b80a8806b2c6c24032cffaacf0945", + "manifest": { + "schema": "termforge-r54-cpu-diagnostic-v1", + "status": "accounting_complete", + "started_utc": "2026-10-27T12:17:43.046171+00:00", + "performance_gate": "not_evaluated", + "observations": { + "schema": "termforge-r54-cpu-observations-v1", + "status": "accounting_complete", + "started_utc": "2026-10-27T12:17:44.816067+00:00", + "attempted": 6, + "accounted_observations": 6, + "context_sha256": "ad54a604709f4a2f28d2af4349de08ebccf1f30d0f2d2238e5381c3005fd2707", + "protocol_sha256": "c92098b46db730cfd7939163831682e5650ca7c4995da40d13980cf7b355410e", + "performance_gate": "not_evaluated", + "active_sequence": null, + "finished_utc": "2026-10-27T12:17:59.527911+00:00" + }, + "finished_utc": "2026-10-27T12:18:00.346147+00:00" + }, + "environment": { + "host": { + "platform": "Windows-11-10.0.26200-SP0", + "python": "3.12.10", + "psutil": "7.2.2", + "cpu_logical": 8, + "cpu_physical": 8, + "cpu": { + "ProcessorNameString": "Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz", + "Identifier": "Intel64 Family 6 Model 158 Stepping 13" + }, + "windows": { + "CurrentBuildNumber": "26200", + "UBR": 9711, + "DisplayVersion": "25H2" + } + }, + "declaration": { + "host_kind": "shared_workstation", + "known_concurrent_work": [ + "This agent completed the measurement contract tests before this diagnostic and starts no concurrent owned build or test.", + "The user workstation, operating-system services and other applications are not isolated or changed.", + "Guest-visible system CPU is recorded; physical-host contention and clock stability remain unobserved." + ], + "purpose": "One predeclared six-observation CPU-boundary diagnostic, not an idle-qualified cost batch." + }, + "scope": "accounting diagnosis on declared host; no idle qualification or cost acceptance" + }, + "builds": { + "baseline": { + "source": { + "commit": "65988f5f20f62294ff52fcac5b5b81e563f93f3b", + "version": "0.26.1-dev+termforge.3", + "states": { + "project": "11966868740884721388", + "interactive": "8035857386658397897" + } + }, + "binary": { + "bytes": 523264, + "sha256": "33cb90beebffc57f07aa993b8ea635eaf461b49d86c143939a41f6242c087b6d" + }, + "candidate_tree": "38f901ce2407d73ab7d4ae70987c294849bafa92", + "tracked_files": 80 + }, + "candidate": { + "source": { + "commit": "3d6d44d3b3e9a9a3e41da98e8711e9f19a211fd8", + "version": "0.26.1-dev+termforge.4", + "states": { + "project": "16178973312633948535", + "interactive": "6154007251599136798" + } + }, + "binary": { + "bytes": 579584, + "sha256": "d4d068be44a0a9ec733ad17f0323765ee38ffbf775e6efacb4632c1dce6b3f01" + }, + "candidate_tree": "2cce55fcf2d379440110bbbf8612117cce3edf64", + "tracked_files": 84 + } + }, + "observations": [ + { + "item": { + "sequence": 0, + "variant": "baseline", + "route": "interactive", + "chunk": 1 + }, + "exit_code": 0, + "timed_out": false, + "inner_cpu_seconds": 2.125, + "child_cpu": { + "creation_filetime": 134375770648262487, + "exit_filetime": 134375770672853118, + "kernel_seconds": 0.921875, + "user_seconds": 1.28125, + "total_seconds": 2.203125 + }, + "runner_cpu_seconds": 0.0, + "environment": { + "total_seconds": 19.625, + "busy_seconds": 12.0, + "busy_percent": 61.146496815286625, + "before": { + "user": 4467.78125, + "kernel_including_idle": 24590.65625, + "idle": 19905.25, + "system": 4685.40625, + "perf_counter_seconds": 3632.723437, + "utc": "2026-10-27T12:17:44.825276+00:00" + }, + "after": { + "user": 4474.515625, + "kernel_including_idle": 24603.546875, + "idle": 19912.875, + "system": 4690.671875, + "perf_counter_seconds": 3635.1845603, + "utc": "2026-10-27T12:17:47.286686+00:00" + } + }, + "accounting": { + "inner_only_balance_seconds": 9.875, + "inner_only_balance_percent": 50.318471337579616, + "child_outside_inner_seconds": 0.078125, + "runner_cpu_seconds": 0.0, + "unassigned_balance_seconds": 9.796875, + "unassigned_balance_percent": 49.9203821656051 + }, + "state_fnv1a64": "8035857386658397897" + }, + { + "item": { + "sequence": 1, + "variant": "candidate", + "route": "interactive", + "chunk": 1 + }, + "exit_code": 0, + "timed_out": false, + "inner_cpu_seconds": 2.140625, + "child_cpu": { + "creation_filetime": 134375770673032344, + "exit_filetime": 134375770697758213, + "kernel_seconds": 0.625, + "user_seconds": 1.609375, + "total_seconds": 2.234375 + }, + "runner_cpu_seconds": 0.015625, + "environment": { + "total_seconds": 19.875, + "busy_seconds": 11.109375, + "busy_percent": 55.89622641509434, + "before": { + "user": 4474.515625, + "kernel_including_idle": 24603.671875, + "idle": 19912.96875, + "system": 4690.703125, + "perf_counter_seconds": 3635.2004591, + "utc": "2026-10-27T12:17:47.302400+00:00" + }, + "after": { + "user": 4480.765625, + "kernel_including_idle": 24617.296875, + "idle": 19921.734375, + "system": 4695.5625, + "perf_counter_seconds": 3637.6748693, + "utc": "2026-10-27T12:17:49.776942+00:00" + } + }, + "accounting": { + "inner_only_balance_seconds": 8.96875, + "inner_only_balance_percent": 45.125786163522015, + "child_outside_inner_seconds": 0.09375, + "runner_cpu_seconds": 0.015625, + "unassigned_balance_seconds": 8.859375, + "unassigned_balance_percent": 44.575471698113205 + }, + "state_fnv1a64": "6154007251599136798" + }, + { + "item": { + "sequence": 2, + "variant": "candidate", + "route": "interactive", + "chunk": 1 + }, + "exit_code": 0, + "timed_out": false, + "inner_cpu_seconds": 2.203125, + "child_cpu": { + "creation_filetime": 134375770697951230, + "exit_filetime": 134375770722075739, + "kernel_seconds": 0.828125, + "user_seconds": 1.46875, + "total_seconds": 2.296875 + }, + "runner_cpu_seconds": 0.0, + "environment": { + "total_seconds": 19.25, + "busy_seconds": 8.953125, + "busy_percent": 46.50974025974026, + "before": { + "user": 4480.796875, + "kernel_including_idle": 24617.390625, + "idle": 19921.8125, + "system": 4695.578125, + "perf_counter_seconds": 3637.6917606, + "utc": "2026-10-27T12:17:49.793919+00:00" + }, + "after": { + "user": 4485.46875, + "kernel_including_idle": 24631.96875, + "idle": 19932.109375, + "system": 4699.859375, + "perf_counter_seconds": 3640.1066225, + "utc": "2026-10-27T12:17:52.208557+00:00" + } + }, + "accounting": { + "inner_only_balance_seconds": 6.75, + "inner_only_balance_percent": 35.064935064935064, + "child_outside_inner_seconds": 0.09375, + "runner_cpu_seconds": 0.0, + "unassigned_balance_seconds": 6.65625, + "unassigned_balance_percent": 34.577922077922075 + }, + "state_fnv1a64": "6154007251599136798" + }, + { + "item": { + "sequence": 3, + "variant": "baseline", + "route": "interactive", + "chunk": 1 + }, + "exit_code": 0, + "timed_out": false, + "inner_cpu_seconds": 2.15625, + "child_cpu": { + "creation_filetime": 134375770722276046, + "exit_filetime": 134375770746664698, + "kernel_seconds": 0.75, + "user_seconds": 1.515625, + "total_seconds": 2.265625 + }, + "runner_cpu_seconds": 0.015625, + "environment": { + "total_seconds": 19.625, + "busy_seconds": 11.625, + "busy_percent": 59.23566878980892, + "before": { + "user": 4485.5, + "kernel_including_idle": 24632.0625, + "idle": 19932.171875, + "system": 4699.890625, + "perf_counter_seconds": 3640.1245048, + "utc": "2026-10-27T12:17:52.226226+00:00" + }, + "after": { + "user": 4491.75, + "kernel_including_idle": 24645.4375, + "idle": 19940.171875, + "system": 4705.265625, + "perf_counter_seconds": 3642.5663847, + "utc": "2026-10-27T12:17:54.668146+00:00" + } + }, + "accounting": { + "inner_only_balance_seconds": 9.46875, + "inner_only_balance_percent": 48.2484076433121, + "child_outside_inner_seconds": 0.109375, + "runner_cpu_seconds": 0.015625, + "unassigned_balance_seconds": 9.34375, + "unassigned_balance_percent": 47.611464968152866 + }, + "state_fnv1a64": "8035857386658397897" + }, + { + "item": { + "sequence": 4, + "variant": "baseline", + "route": "interactive", + "chunk": 1 + }, + "exit_code": 0, + "timed_out": false, + "inner_cpu_seconds": 2.125, + "child_cpu": { + "creation_filetime": 134375770746872014, + "exit_filetime": 134375770770988460, + "kernel_seconds": 0.578125, + "user_seconds": 1.625, + "total_seconds": 2.203125 + }, + "runner_cpu_seconds": 0.0, + "environment": { + "total_seconds": 19.25, + "busy_seconds": 9.5625, + "busy_percent": 49.675324675324674, + "before": { + "user": 4491.78125, + "kernel_including_idle": 24645.53125, + "idle": 19940.25, + "system": 4705.28125, + "perf_counter_seconds": 3642.5841586, + "utc": "2026-10-27T12:17:54.685918+00:00" + }, + "after": { + "user": 4497.65625, + "kernel_including_idle": 24658.90625, + "idle": 19949.9375, + "system": 4708.96875, + "perf_counter_seconds": 3644.9979968, + "utc": "2026-10-27T12:17:57.098922+00:00" + } + }, + "accounting": { + "inner_only_balance_seconds": 7.4375, + "inner_only_balance_percent": 38.63636363636363, + "child_outside_inner_seconds": 0.078125, + "runner_cpu_seconds": 0.0, + "unassigned_balance_seconds": 7.359375, + "unassigned_balance_percent": 38.23051948051948 + }, + "state_fnv1a64": "8035857386658397897" + }, + { + "item": { + "sequence": 5, + "variant": "candidate", + "route": "interactive", + "chunk": 1 + }, + "exit_code": 0, + "timed_out": false, + "inner_cpu_seconds": 2.03125, + "child_cpu": { + "creation_filetime": 134375770771152026, + "exit_filetime": 134375770795208267, + "kernel_seconds": 0.953125, + "user_seconds": 1.15625, + "total_seconds": 2.109375 + }, + "runner_cpu_seconds": 0.015625, + "environment": { + "total_seconds": 19.25, + "busy_seconds": 11.9375, + "busy_percent": 62.01298701298701, + "before": { + "user": 4497.65625, + "kernel_including_idle": 24659.03125, + "idle": 19950.03125, + "system": 4709.0, + "perf_counter_seconds": 3645.0124583, + "utc": "2026-10-27T12:17:57.114619+00:00" + }, + "after": { + "user": 4503.953125, + "kernel_including_idle": 24671.984375, + "idle": 19957.34375, + "system": 4714.640625, + "perf_counter_seconds": 3647.4196989, + "utc": "2026-10-27T12:17:59.521405+00:00" + } + }, + "accounting": { + "inner_only_balance_seconds": 9.90625, + "inner_only_balance_percent": 51.46103896103896, + "child_outside_inner_seconds": 0.078125, + "runner_cpu_seconds": 0.015625, + "unassigned_balance_seconds": 9.8125, + "unassigned_balance_percent": 50.97402597402598 + }, + "state_fnv1a64": "6154007251599136798" + } + ], + "formal_cost_samples": 0, + "old_hosted_event_cause": "unknown", + "archive": { + "bytes": 100053, + "sha256": "9bec0731e08d7f2ef99d140a9e56acbc01468814cc68fb01fbdcfe30eaa8e10c", + "inventory_files": 46, + "byte_exact_streams": 16, + "path_prefixes_aliased": true, + "child_pid_omitted": true + }, + "local_tests": { + "before_measurement": { + "count": 36, + "seconds": 18.66 + }, + "after_test_runtime_boundary_fix": { + "count": 36, + "seconds": 17.069, + "log_sha256": "e3ab51e002dd2446be8ee34943895825e433197f5d7bff6338b4cf5c3763edf7" + } + }, + "limits": [ + "Shared host, no idle qualification and no performance ratios.", + "Residual includes possible other guest activity, counter quantization and non-atomic endpoints.", + "New instrumentation does not recreate missing whole-child counters for the old hosted failure.", + "Full controlled batches, wrapper/projection and component attribution, budget proposal and user confirmation remain incomplete." + ], + "initial_ci": { + "source": "c76bab0281843fca298af8b0d9ba521247ac745a", + "all_attempts": 1, + "default_pr": { + "run": 40504397868, + "status": "success", + "windows": 46, + "linux": 29 + }, + "default_push": { + "run": 40504336483, + "status": "success", + "windows": 46, + "linux": 29 + }, + "candidate_pr": { + "run": 40504397941, + "status": "failure", + "windows_failure": "ModuleNotFoundError: No module named psutil", + "linux_passed": 41, + "linux_skipped": 4 + }, + "candidate_push": { + "run": 40504336888, + "status": "failure", + "windows_failure": "ModuleNotFoundError: No module named psutil", + "linux_passed": 41, + "linux_skipped": 4 + }, + "fix": "Install the existing pinned benchmarks/requirements.txt in candidate CI; no measurement source or observation change." + } +} diff --git a/experiments/measure/results/2026-10-27-r54-cpu-boundary/verify.py b/experiments/measure/results/2026-10-27-r54-cpu-boundary/verify.py new file mode 100644 index 00000000..f60f9e5a --- /dev/null +++ b/experiments/measure/results/2026-10-27-r54-cpu-boundary/verify.py @@ -0,0 +1,73 @@ +"""Recheck retained synthetic evidence without extracting files or launching a cost run.""" +import hashlib +import json +from pathlib import Path +import sys +import zipfile + +ROOT = Path(__file__).resolve().parents[4] +sys.path.insert(0, str(ROOT)) +from experiments.measure.r54_contract import validate_sample + + +def verify(): + path = Path(__file__).with_name('evidence.zip') + assert hashlib.sha256(path.read_bytes()).hexdigest() == '9bec0731e08d7f2ef99d140a9e56acbc01468814cc68fb01fbdcfe30eaa8e10c' + with zipfile.ZipFile(path) as archive: + read = lambda name: json.loads(archive.read(name)) + inventory = read('publication.json')['files'] + assert len(archive.namelist()) == 47 + assert len({row['path'] for row in inventory}) == 46 + assert set(archive.namelist()) == {row['path'] for row in inventory} | {'publication.json'} + for row in inventory: + data = archive.read(row['path']) + assert len(data) == row['published_bytes'] + assert hashlib.sha256(data).hexdigest() == row['published_sha256'] + if row['treatment'] == 'byte_exact_owned_synthetic_stream': + assert row['published_sha256'] == row['original_sha256'] + assert read('manifest.json')['status'] == 'accounting_complete' + manifest = read('observations/manifest.json') + assert manifest['attempted'] == manifest['accounted_observations'] == 6 + assert manifest['performance_gate'] == 'not_evaluated' + builds = read('binding.json')['builds'] + prior = json.loads((path.parent.parent / '2026-10-27-r54-runner/validation.json').read_text(encoding='utf-8')) + for variant in ('baseline', 'candidate'): + assert builds[variant]['binary'] == prior['builds'][variant]['binary'] + assert builds[variant]['source']['commit'] == prior['builds'][variant]['source'] + for index, variant in enumerate(('baseline', 'candidate', 'candidate', 'baseline', 'baseline', 'candidate')): + prefix = f'observations/{index:03d}/' + row = read(prefix + 'record.json') + assert row['item'] == dict(sequence=index, variant=variant, route='interactive', chunk=1) + assert row['status'] == 'accounted' and row['exit_code'] == 0 and not row['timed_out'] + assert 'child_pid' not in row + for name in ('stdout.bin', 'stderr.bin'): + data = archive.read(prefix + name) + assert len(data) == row['streams'][name]['bytes'] + assert hashlib.sha256(data).hexdigest() == row['streams'][name]['sha256'] + raw = read(prefix + 'stdout.bin') + assert raw == row['result'] + validate_sample(raw, row['item']) + env, child = row['environment'], row['child_cpu'] + before, after = env['before'], env['after'] + busy = after['user'] + after['system'] - before['user'] - before['system'] + total = busy + after['idle'] - before['idle'] + assert abs(busy - env['busy_seconds']) < 1e-8 + assert abs(total - env['total_seconds']) < 1e-8 + assert child['total_seconds'] == child['kernel_seconds'] + child['user_seconds'] + assert child['exit_filetime'] > child['creation_filetime'] + runner = row['runner_cpu_after_seconds'] - row['runner_cpu_before_seconds'] + assert runner == row['runner_cpu_seconds'] + accounted = row['accounting'] + inner = raw['interval_cpu_seconds'] + assert child['total_seconds'] >= inner + assert accounted['child_outside_inner_seconds'] == child['total_seconds'] - inner + assert accounted['inner_only_balance_seconds'] == busy - inner + assert accounted['runner_cpu_seconds'] == runner + assert accounted['unassigned_balance_seconds'] == busy - child['total_seconds'] - runner + assert abs(accounted['unassigned_balance_percent'] - accounted['unassigned_balance_seconds'] * 100 / total) < 1e-8 + return dict(files=47, hashed_evidence=46, byte_exact_streams=16, observations=6, + oracles_passed=6, accepted_cost_samples=0, performance_gate='not_evaluated') + + +if __name__ == '__main__': + print(json.dumps(verify())) diff --git a/experiments/measure/test_r54_cpu_diagnostic.py b/experiments/measure/test_r54_cpu_diagnostic.py new file mode 100644 index 00000000..d97220ac --- /dev/null +++ b/experiments/measure/test_r54_cpu_diagnostic.py @@ -0,0 +1,98 @@ +"""Process/artifact boundary tests; synthetic CPU work is not a core benchmark.""" +import json +import os +from pathlib import Path +import sys +import tempfile +import unittest + + +class AccountingTests(unittest.TestCase): + def test_own_interval_excluded_cpu_is_separated_and_negative_balance_is_retained(self): + from experiments.measure.r54_cpu_diagnostic import partition_cpu + record = dict(environment=dict(total_seconds=10, busy_seconds=3), + child_cpu=dict(total_seconds=2.25), runner_cpu_seconds=.125) + value = partition_cpu(record, 2) + self.assertEqual(value['inner_only_balance_seconds'], 1) + self.assertEqual(value['child_outside_inner_seconds'], .25) + self.assertEqual(value['unassigned_balance_seconds'], .625) + self.assertEqual(value['unassigned_balance_percent'], 6.25) + record['environment']['busy_seconds'] = 2.25 + self.assertEqual(partition_cpu(record, 2)['unassigned_balance_seconds'], -.125) + with self.assertRaises(ValueError): + partition_cpu(record, 3) + for invalid in (float('nan'), float('inf'), -1): + record['runner_cpu_seconds'] = invalid + with self.assertRaises(ValueError): + partition_cpu(record, 2) + + +def native_environment_supported(): + if os.name != 'nt' or sys.implementation.name != 'cpython' or sys.version_info[:2] != (3, 12): + return False + import psutil + return 0 < (psutil.cpu_count() or 0) <= 64 + + +@unittest.skipUnless(native_environment_supported(), 'native diagnostic requires Windows CPython 3.12, 1..64 CPUs') +class ProcessAccountingTests(unittest.TestCase): + def test_lifetime_cpu_includes_work_outside_child_reported_interval(self): + from experiments.measure.r54_cpu_diagnostic import capture_cpu + code = '''import json,os,time +def burn(seconds): + start=time.process_time() + while time.process_time()-start < seconds: + pass +burn(.15) +start=time.process_time() +burn(.05) +inner=time.process_time()-start +burn(.15) +print(json.dumps({'inner_cpu':inner,'pid':os.getpid()})) +''' + with tempfile.TemporaryDirectory() as folder: + # A Windows venv executable is a redirector which starts another + # process. Use the actual interpreter to test the direct-child seam. + result = capture_cpu([sys._base_executable, '-c', code], Path(folder), timeout=5) + self.assertEqual(result['exit_code'], 0, result) + self.assertNotIn('error', result) + child = json.loads((Path(folder) / 'stdout.bin').read_bytes()) + self.assertEqual(result['child_pid'], child['pid']) + self.assertGreaterEqual(result['child_cpu']['total_seconds'] - child['inner_cpu'], .29) + self.assertGreater(result['child_cpu']['exit_filetime'], result['child_cpu']['creation_filetime']) + self.assertGreater(result['environment']['total_seconds'], 0) + self.assertGreaterEqual(result['runner_cpu_seconds'], 0) + + def test_timeout_retains_partial_streams_and_exited_child_accounting(self): + from experiments.measure.r54_cpu_diagnostic import capture_cpu + code = "import os,time; os.write(1,b'partial\\xff'); os.write(2,b'error\\xfe'); time.sleep(10)" + with tempfile.TemporaryDirectory() as folder: + result = capture_cpu([sys._base_executable, '-c', code], Path(folder), timeout=1) + self.assertTrue(result['timed_out']) + self.assertIsNotNone(result['exit_code']) + self.assertGreater(result['child_cpu']['exit_filetime'], 0) + self.assertEqual((Path(folder) / 'stdout.bin').read_bytes(), b'partial\xff') + self.assertEqual((Path(folder) / 'stderr.bin').read_bytes(), b'error\xfe') + + def test_bad_output_stops_diagnostic_without_overwrite_or_cost_summary(self): + from experiments.measure.r54_cpu_diagnostic import run_observations + command = [sys._base_executable, '-c', "import os; os.write(1,b'bad\\xff')"] + with tempfile.TemporaryDirectory() as folder: + output = Path(folder) / 'observations' + context = {'evidence_kind': 'synthetic_process_fixture'} + result = run_observations({name: command for name in ('baseline', 'candidate')}, output, context) + self.assertEqual(result['status'], 'failed') + self.assertEqual(result['attempted'], 1) + self.assertEqual(result['accounted_observations'], 0) + self.assertEqual((output / '000/stdout.bin').read_bytes(), b'bad\xff') + self.assertFalse((output / '001').exists()) + self.assertFalse((output / 'summary.json').exists()) + self.assertEqual(result['performance_gate'], 'not_evaluated') + original = (output / 'manifest.json').read_bytes() + with self.assertRaises(FileExistsError): + run_observations({name: command for name in ('baseline', 'candidate')}, output, context) + self.assertEqual((output / 'manifest.json').read_bytes(), original) + + +if __name__ == '__main__': + unittest.main()