Repository navigation
feat(services): port the services layer and the worker model (#866) - #1054
Conversation
🦋 Changeset detectedLatest commit: 78df655 The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
[Critical impact] The latest changes appear safe to merge; the remaining import-memory concern is non-blocking. Findings
|
Tor: the bootstrap now returns a `HiddenServiceBootstrap` that owns the control connection, and the daemon holds it until the drain. `ADD_ONION` carries no `Flags=Detach`, so quitting at startup killed the onion; the task now QUITs at shutdown instead. The key is staged at a temp path and renamed, and its mode is set on the open handle so a pre-cutover 0644 file is narrowed. NWC: replies correlate on the reply event's `e` tag, which is what NIP-47 and `@getalby/sdk` use, instead of an `id` the payload never carries. The relay transport compiles with rustls, so a `wss://` relay connects. Workers: a panicked worker is mapped back to its index through `JoinSet::join_next_with_id` and the task-id map, so it takes the same restart path as an error. The DVM child write is bounded and kills a child that stops reading, instead of parking the loop that reads timeouts and the drain. Web of trust: follow lists are deduplicated, so one member's repeated `p` tag cannot satisfy `minimum_followers` alone, and `ensure_built` spawns the rebuild so a cancelled relay handler cannot leave `building` set forever. Payments: `UserPatch`'s flags are `FieldUpdate<bool>` and the Postgres merge skips an absent one, so creating an invoice no longer clears an admission or a vanish flag. A month-end payment notification rolls into the month after the next, as `new Date(y, m + 1, d)` does, instead of expiring on arrival. Imports: replaceable versions are grouped by their conflict key before the batch write (Postgres refuses to update one row twice in a statement), the array reader enforces the array's own grammar (`[e,]`, `[e]garbage`), and the CLI streams progress and line errors to an injected writer as they happen. Finding 13 (SSRF on the NIP-05 fetch) is not fixed; build.md records what remains and claims no coverage for it.
The NIP-05 fetch built its URL from a stored identifier and dialled it, so a domain pointing at loopback, RFC1918, link-local, multicast or bettercloud metadata space was fetched. The check could not live in the shared client: the payment processors and notification channels are the operator's to point at a private address, and a global block would break a self-hosted lnbits or lnurl endpoint. The NIP-05 fetch already builds its own client per request for the redirect policy, so it now also installs GuardedResolver there. Resolution drops every non-public address and fails the lookup when nothing public is left, which keeps SNI and the Host header on the original name and connects only to the address set that was vetted. IPv4-mapped IPv6 forms are matched against the IPv4 rule. The parsed initial URL goes through the same predicate the redirect policy uses before anything is resolved, and describe_transport_error keeps reqwest's source chain so the refusal reaches the verification outcome instead of a bare "error sending request". The capture has the same gap (src/utils/nip05.ts) and no resolver to close it with, so this is a deliberate divergence. Tests inject a resolver that answers from a table, so none needs DNS.
| async fn expect_trailing_whitespace(&mut self) -> Result<(), ServicesError> { | ||
| loop { | ||
| self.skip_whitespace(); |
There was a problem hiding this comment.
expect_trailing_whitespace keeps every chunk after the closing bracket in buffer. Unlike the element-reading loop, it never calls compact(). A valid JSON file with a large tail of blank lines therefore uses memory proportional to that tail and can exhaust memory during an otherwise streaming import. Discard consumed bytes before reading the next chunk.
The round-5 NWC fix correlated replies on the `e` tag but left `pubkey` a field the relay writes, so a hostile relay could replay an earlier genuine ciphertext under the current request and reach the admission gate on a settled lookup. A reply is now accepted only when its id is the one its own fields hash to and the wallet's signature verifies over it, on top of the pubkey and `e`-tag checks. The onion key narrowing only ran on the write path, which an existing key never takes: a key the TypeScript daemon left at 0644 was adopted at that mode for the life of the installation. ensure_hidden_service narrows an adopted key as it reads it, and the test now drives that path instead of the branch that already worked. HiddenServiceBootstrap is #[must_use] and warns on drop while the control connection is still held, so the lifetime invariant fails loudly instead of leaving a green Enabled log over a dead onion. The import takes the stream main hands it rather than locking stdout inside the library, and a write failure is an error rather than a success with nothing delivered; R40 and A22 record the caller-supplied sink. The replaceable conflict key and merge predicate now live in storage, where upsert_many discharges them for every backend, so the import no longer restates the rule and the key stops allocating. Also: the month-end helper takes the day's first existing hour instead of falling back to now across a midnight DST skip, the follow dedup borrows its keys, and the wot guard test discriminates its predicate.
Ports the services layer and the worker model from the TypeScript capture into a new
nostream-servicescrate, wires it throughnostreamd, and gives the importer a CLI. Job023-services-worker-model, from the factory; issue #866.What is on the branch
Supervisordrives the four worker types the capture has today (client, maintenance, relay-monitor, dvm-orchestrator) as tokio tasks with the same lifecycle guarantees:ShutdownStatedrain with a deadline, respawn on panic, per-worker event-store resolution.WORKER_COUNTfalls through on aNaNinstead of forking zero workers.ServiceContext,ServiceStores, anHttpClientseam whoseReqwestClientis also the NIP-05 fetch seam, the six payment processors plus the null one, the invoice lifecycle (pending → completed/expired, the 404-on-lookup rule, the local-midnight expiry), the payments service, the operator-notification dispatcher with its outbox, delivery log and HMAC-signed HTTP channel, the maintenance and DVM orchestrators, the Tor control client, the wot graph service, and the JSONL importer.SECRETgate, worker-count and event-backend resolution,WotGraphServicein place ofUnavailableWotGraph, supervisor boot,relay.restartedenqueue, Tor outcome logging at drain.nostream-cli importwith its own argument parser, help text, progress and summary output.0010-services-worker-model-and-cache.md(cache decision, worker count, respawn, the Tor port spec, the transaction gap),CLAUDE.mdupdates, the remaining services stubs relabelled to [Rewrite P2] nostream-relay: wire handlers, strategies, rate limits #865/[Rewrite P5] Admin console parity + admin MCP server + CLI port #867.Acceptance
cargo test --workspace --all-features→ 1092 passed, 0 failed, 4 ignored at78df655(base 951). Every CIRust Checkscontext is green at that head, includingStorage parity against Postgres, the op that was failing until round 7 fixed it; pluscargo check -p nostream-storage --no-default-features --all-targets.Seven review rounds: rounds 1–2 gated by the code lens, round 3 ran all six lenses and returned three CHANGES, round 4 returned six PASS, round 5 turned up thirteen Greptile findings plus three CHANGES (security ×2, architecture ×3, code), round 6 confirmed all of them fixed but the architecture lens found one defect the round itself created, which exhausted the budget and escalated. The operator extended the job; round 7 fixed that defect and returned code PASS and architecture PASS.
build.mdcarries the per-criterion evidence table for all 65 acceptance criteria.Two criteria are judged, not measured, and the spec says so: the 100k-event LMDB checksum belongs to open #864 (there is no LMDB backend and no corpus), and four rows are marked
[judged: Cameri].What the approver escalated
Two approvers escalated this branch, on different conditions, and both stand:
Round 4 —
schema-or-dependency. Every approve condition held (its own oracle run, all six lenses PASS, a cleanmerge-tree, all 65 criteria checked); the escalation is the harness rule that a diff adding dependencies and touching money/access goes to a person:reqwest0.12 (default-features = false,rustls-tls),aes0.8,cbc0.1, plus tokio'sfs/processfeatures. Rationale and alternatives: ADR 0010 §A3.PaymentsService::confirm_invoiceadmits a payer whenamount_paid_msats >= admission_fee. Payments are disabled by default with the null processor and the daemon is pre-cutover, but the surface lands here.Round 7 —
money-access-identity. The same subject, unresolved by any recorded decision:confirm_invoice's admission rule lands on the knowingly non-transactional write order (spec A9), and AC61, AC62 and AC65 are[judged: Cameri]by the spec's own text.The spec's escalate-marked assumptions ride with it: A1 (embedded cache, no Redis), A9 (three ordered writes instead of the capture's single transaction — the one knowingly weakened guarantee), A11 (Tor port spec targets
127.0.0.1), A12 (subscriptions deferred, no issue owns them), A17 (compressed.jsonl.gzimports refused).Follow-ups this PR does not close
None outstanding in
tor.rs. Both items an earlier revision of this body listed were closed in round 6: the drain no longer truncates the key in place (temp file plus rename,crates/nostream-services/src/tor.rs:433-456), and the0600narrowing now runs on the read path too, so a pre-cutover key at the same path is narrowed rather than left at 0644. The round-6 code lens confirmed both by inversion.spec.mdalso carries citation ranges that the oracle flagged and a reader confirmed stale (lnurl/opennode/nwc processor ranges, the operator-notification range, the storage cache stub range, and.rsfor a.tspath). No criterion turns on them.Closing
Closes #866is decoration here — this PR targetsrewrite/main, not the default branch, so the merge does not close the issue. Whoever merges closes #866 with the merge sha.