Repository navigation
Description
The storage seam and the LMDB event store landed in job 024 (68be6f3a), including the metadata tables' placement recorded in docs/rewrite/decisions/0011-lmdb-backend.md. #864 closed with three slices still unbuilt — NIP-50 search, the pg-to-lmdb migration tool and the benchmark rig — and none of them had an issue of its own. This is the search one.
Recon corrected three of this issue's original premises (job 028, measured at b42f833), so they are removed rather than left to mislead:
- The LMDB path does not silently ignore
search. With the NIP-50 gate on it refuses:StorageError::NotImplemented { module: "storage::lmdb::nip50_search" },crates/nostream-storage/src/lmdb/events.rs:1028-1048. The silent pass-through exists only with the gate off. So this job defines a behaviour where there is a refusal, it does not fill a silent gap. - The fixture already has the search row — script entry 21, 4 expected rows — and no corpus addition is needed. It is excluded from the LMDB arm by index in
crates/nostream-storage/tests/parity_lmdb.rs:43(EXCLUDED: [21, 48, 50, 112, 115]). - The capture disagrees with itself, so it is a reference and not one behaviour: SQL
to_tsvector @@ plainto_tsqueryrankedts_rank DESC, event_id ASCwith a 500-row default, versus an in-memory matcher doing lowercase all-terms substrings. The fixture recorded the SQL one.
Scope: scan-time filtering with set parity. The rule is applied where records are already examined, and the fixture's set is the authority for the gate-on path. That means the token rule (all query terms present as tokens), not Filter::matches's substring rule: measured, substring yields 5 rows where the fixture expects 4. With the gate off, the Postgres path applies SubstringTerms (postgres/repository.rs:72-80), so LMDB implementing the substring rule there is parity and its current silence is a divergence.
What decides it: entry 21 replayed in the LMDB arm — red at base on the refusal, green after — with the fixture's own count assertion (EXCLUDED.len() + REPLAYED, and the marker line that prints the list) still holding. Plus the 500-row cap, and the gate-off arm applying substring terms.
What it does not claim: ordering. All four expected rows tie on term frequency and the fixture's order is ts_rank proximity, so a scan reproduces neither the order nor the boundary. That criterion is judged, and the PR body's first paragraph says the set matches while the order is implementation-defined.
Out of scope: nostream-cli migrate pg-to-lmdb, the benchmark rig, and ranked search — that last one is #1110, which needs an inverted index and a rebuild path the LMDB store does not have.
Cost: a job-level expectation recorded in job.json, not the #864 row, which is 4× overdrawn by job 024 alone (estimates.md records the measurement).
Found while closing out 027; corrected by recon on 2026-11-11.
Activity
chappie-daemon commentedon Nov 11, 2026
Scope reversed on 2026-11-11, before the first commit — recorded rather than quietly edited.
I scoped this job to a scan-time filter with set parity only, and filed #1110 for the index, ranking and rebuild. Job 028's spec then came back built for the larger shape, with 22 criteria through two adversarial passes: the first round named six implementations that pass undetected (a stale-key index never cleaned on delete, a lazy build-on-first-search, a marker-written-first rebuild, a verdict fed truncated terms, an unconditional rebuild, tf+id ranking), and the second named three more — including a
page-only router thatcount's limited path bypasses atevents.rs:803, which would have shipped a count disagreeing with its page.I reverted my own scope rather than the spec. The validated spec is the expensive artifact, and a 30-line scan followed by a second job in the same files would spend more in total than one job with the index; two adversarial passes over a storage-format change are worth more than two sequential slices. So the job builds the index, the
search-index-versionmarker and the open-time rebuild.Conditions attached, so the change cannot arrive unrecorded: an ADR in
docs/rewrite/decisions/for the 30th database, the key shape and the rebuild, in the same change; the ordering divergence (limited search ranks by span then id where Postgres keepscreated_at DESC) as the PR body's first paragraph; and the residuals in the PR body's known-limits section. #1110 closes against this job if the ADR and the rebuild land, and stays open for whatever does not.This corrects my Telegram report of "A, set parity, ordering judged", which is no longer what the job builds.
chappie-daemon commentedon Nov 11, 2026
Final scope, after three reversals — this is the decision, and the branch state proves it.
028 ships the index build (design B) at
8916b38: a 30th named databaseevents_by_termmaintained inwrite_eventandhard_delete, asearch-index-versionmarker written last, an open-time rebuild in batches, ADR 0012, and ranking by(min cover span, event_id).The scan-time design was also built — green at
6b50424onfactory/028a-nip50-scan, 1253 tests, red-first proven — and it is preserved rather than deleted. It lost on one measurement: it asserts the fixture's set and marks ordering judged, while(min cover span, event_id)reproduces the recorded order and set. B is therefore strictly more parity, and the earlier argument that "the index buys performance, not parity" was wrong exactly where it mattered.The flip-flop is in the record on purpose. Earlier comments here and on #1110 describe the scan-time scope, then the index, then the scan again; each was decided on evidence available at the time, and two of those messages never reached the job at all (see pi-extensions#63 — a dispatch to a mid-turn member is not injected until its turn ends, so three rows were answered hours late or not at all).
Cost
7.50, both builds left in the spans rather than netted out. Out of scope now: nothing from the original issue body; ranked search is in this job, and #1110 keeps only what goes beyond it.chappie-daemon commentedon Nov 11, 2026
Follow-ups filed from the PR's review round: #1111 (second in-process handle can lose postings during a rebuild — contract sentence is in ADR 0012 D9, the failing test is not) and #1112 (one-term search proposes every posting sharing the term's first bytes). Both are known limits in the PR body, not defects in it.