You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
dvm: a dispatch tick can hand a job to a worker that handle_exit then clears #1096
Found by the round-7 static review of #870 (PR #1094). Pre-existing at rewrite/main — the file appears in that diff only because its one find_by_filters call site now goes through collect_records.
The mechanism
crates/nostream-services/src/workers/dvm_orchestrator.rs:206 — kill_worker clears the child slot before the old child's Exit is read. A dispatch tick arriving in that window can hand a job to a replacement worker, which handle_exit then clears when it processes the old child's exit. The job is accepted and lost, with no error to the client and no record that it was dispatched.
Why it is filed rather than fixed here
It has nothing to do with page slots or completeness hints; it is a lifecycle race in the DVM worker supervisor. Widening the #870 job to cover it would make a protocol-PR review a supervision-PR review, which is how a finding becomes two unreviewed changes.
What a fix needs
The slot must not be observable as free until the old child's Exit has been consumed — either hold the job behind the exit, or tag each dispatch with the generation it was handed to so handle_exit clears only its own generation. The test needs a deterministic way to lose the race: a double whose Exit is delivered after the next dispatch tick, asserting the job is delivered to a worker that is still alive.
Two more mechanisms in the same worker, from Greptile's round-8 static pass on #1094 — filed here rather than as separate issues because one pass over the worker's lifecycle should fix all three.
drive_worker stops listening once stdout closes (dvm_orchestrator.rs:607). After the child's stdout ends, the loop exits and the code waits only for process exit, so the killed signal it was supposed to honour after a job timeout or a shutdown is never observed. A child that closes stdout but keeps running stays alive while the relay starts a replacement.
dispatch_next can never reach newer jobs (dvm_orchestrator.rs:262). It examines the oldest ten pending rows, and that list includes PickedUp jobs assign_worker cannot claim. Ten abandoned claims at the front — the shape a crash leaves — make every later tick re-read the same ten, so newer submitted jobs are never dispatched. Query claimable rows separately, page past claimed ones, or recover abandoned claims on a timer.
All three are pre-existing on rewrite/main; they reached the static record because #1094 touches this file's one find_by_filters call site.
A fourth mechanism in the same file, from the round-9 static review of #870 (PR #1094) — pre-existing at rewrite/main68be6f3, and the PR's only hunk in this file is at line 302.
A restarted supervisor run reuses a spent orchestrator.dvm_specs (dvm_orchestrator.rs:648-662) builds one Arc<DvmOrchestrator> per worker outside the spec's closure and moves the Arc in:
let orchestrator = Arc::new(DvmOrchestrator::new(..));
installed.push(WorkerSpec::new("dvm-orchestrator",Box::new(move |shutdown| Box::pin(Arc::clone(&orchestrator).run(shutdown)))));
run consumes the event receiver on entry (:534-540, .lock().take(), else-branch return Ok(())). The first run takes it; if that run ends — a panic inside the loop, or any restart — the supervisor's next call finds None, returns Ok(()), and a clean exit is what the supervisor reads as "finished, stop restarting". Every DVM job then silently stops until the daemon itself restarts.
The three mechanisms already recorded here are all "the job is lost"; this one is "the worker never comes back", which is why it needs its own fix: build a fresh orchestrator inside the closure (or make run take the receiver as an argument rather than consuming a stored one), with cleanup for the previous child and its pending jobs. A test wants a supervisor double that restarts a worker whose first run returns, then asserts the second run still dispatches.
Activity
chappie-daemon commentedon Nov 9, 2026
Two more mechanisms in the same worker, from Greptile's round-8 static pass on #1094 — filed here rather than as separate issues because one pass over the worker's lifecycle should fix all three.
drive_workerstops listening once stdout closes (dvm_orchestrator.rs:607). After the child's stdout ends, the loop exits and the code waits only for process exit, so thekilledsignal it was supposed to honour after a job timeout or a shutdown is never observed. A child that closes stdout but keeps running stays alive while the relay starts a replacement.dispatch_nextcan never reach newer jobs (dvm_orchestrator.rs:262). It examines the oldest ten pending rows, and that list includesPickedUpjobsassign_workercannot claim. Ten abandoned claims at the front — the shape a crash leaves — make every later tick re-read the same ten, so newer submitted jobs are never dispatched. Query claimable rows separately, page past claimed ones, or recover abandoned claims on a timer.All three are pre-existing on
rewrite/main; they reached the static record because #1094 touches this file's onefind_by_filterscall site.chappie-daemon commentedon Nov 9, 2026
A fourth mechanism in the same file, from the round-9 static review of #870 (PR #1094) — pre-existing at
rewrite/main68be6f3, and the PR's only hunk in this file is at line 302.A restarted supervisor run reuses a spent orchestrator.
dvm_specs(dvm_orchestrator.rs:648-662) builds oneArc<DvmOrchestrator>per worker outside the spec's closure and moves theArcin:runconsumes the event receiver on entry (:534-540,.lock().take(), else-branchreturn Ok(())). The first run takes it; if that run ends — a panic inside the loop, or any restart — the supervisor's next call findsNone, returnsOk(()), and a clean exit is what the supervisor reads as "finished, stop restarting". Every DVM job then silently stops until the daemon itself restarts.The three mechanisms already recorded here are all "the job is lost"; this one is "the worker never comes back", which is why it needs its own fix: build a fresh orchestrator inside the closure (or make
runtake the receiver as an argument rather than consuming a stored one), with cleanup for the previous child and its pending jobs. A test wants a supervisor double that restarts a worker whose first run returns, then asserts the second run still dispatches.