Skip to content

storage: NIP-50 search over the LMDB store — the FTS index, parity-tested against the corpus #1109

Open

Description

@chappie-daemon

The storage seam and the LMDB event store landed in job 024 (68be6f3a), including the metadata tables' placement recorded in docs/rewrite/decisions/0011-lmdb-backend.md. #864 closed with three slices still unbuilt — NIP-50 search, the pg-to-lmdb migration tool and the benchmark rig — and none of them had an issue of its own. This is the search one.

Recon corrected three of this issue's original premises (job 028, measured at b42f833), so they are removed rather than left to mislead:

  • The LMDB path does not silently ignore search. With the NIP-50 gate on it refuses: StorageError::NotImplemented { module: "storage::lmdb::nip50_search" }, crates/nostream-storage/src/lmdb/events.rs:1028-1048. The silent pass-through exists only with the gate off. So this job defines a behaviour where there is a refusal, it does not fill a silent gap.
  • The fixture already has the search row — script entry 21, 4 expected rows — and no corpus addition is needed. It is excluded from the LMDB arm by index in crates/nostream-storage/tests/parity_lmdb.rs:43 (EXCLUDED: [21, 48, 50, 112, 115]).
  • The capture disagrees with itself, so it is a reference and not one behaviour: SQL to_tsvector @@ plainto_tsquery ranked ts_rank DESC, event_id ASC with a 500-row default, versus an in-memory matcher doing lowercase all-terms substrings. The fixture recorded the SQL one.

Scope: scan-time filtering with set parity. The rule is applied where records are already examined, and the fixture's set is the authority for the gate-on path. That means the token rule (all query terms present as tokens), not Filter::matches's substring rule: measured, substring yields 5 rows where the fixture expects 4. With the gate off, the Postgres path applies SubstringTerms (postgres/repository.rs:72-80), so LMDB implementing the substring rule there is parity and its current silence is a divergence.

What decides it: entry 21 replayed in the LMDB arm — red at base on the refusal, green after — with the fixture's own count assertion (EXCLUDED.len() + REPLAYED, and the marker line that prints the list) still holding. Plus the 500-row cap, and the gate-off arm applying substring terms.

What it does not claim: ordering. All four expected rows tie on term frequency and the fixture's order is ts_rank proximity, so a scan reproduces neither the order nor the boundary. That criterion is judged, and the PR body's first paragraph says the set matches while the order is implementation-defined.

Out of scope: nostream-cli migrate pg-to-lmdb, the benchmark rig, and ranked search — that last one is #1110, which needs an inverted index and a rebuild path the LMDB store does not have.

Cost: a job-level expectation recorded in job.json, not the #864 row, which is 4× overdrawn by job 024 alone (estimates.md records the measurement).

Found while closing out 027; corrected by recon on 2026-11-11.

Activity

  1. added
    rewritenostream 4.0 Rust/LMDB rewrite
    rustRust implementation work
    lmdbLMDB / strfry-style storage backend
    factoryBuilt by the software factory
    on Nov 10, 2026
  2. chappie-daemon commented on Nov 11, 2026

    @chappie-daemon
    CollaboratorAuthor

    Scope reversed on 2026-11-11, before the first commit — recorded rather than quietly edited.

    I scoped this job to a scan-time filter with set parity only, and filed #1110 for the index, ranking and rebuild. Job 028's spec then came back built for the larger shape, with 22 criteria through two adversarial passes: the first round named six implementations that pass undetected (a stale-key index never cleaned on delete, a lazy build-on-first-search, a marker-written-first rebuild, a verdict fed truncated terms, an unconditional rebuild, tf+id ranking), and the second named three more — including a page-only router that count's limited path bypasses at events.rs:803, which would have shipped a count disagreeing with its page.

    I reverted my own scope rather than the spec. The validated spec is the expensive artifact, and a 30-line scan followed by a second job in the same files would spend more in total than one job with the index; two adversarial passes over a storage-format change are worth more than two sequential slices. So the job builds the index, the search-index-version marker and the open-time rebuild.

    Conditions attached, so the change cannot arrive unrecorded: an ADR in docs/rewrite/decisions/ for the 30th database, the key shape and the rebuild, in the same change; the ordering divergence (limited search ranks by span then id where Postgres keeps created_at DESC) as the PR body's first paragraph; and the residuals in the PR body's known-limits section. #1110 closes against this job if the ADR and the rebuild land, and stays open for whatever does not.

    This corrects my Telegram report of "A, set parity, ordering judged", which is no longer what the job builds.

  3. chappie-daemon commented on Nov 11, 2026

    @chappie-daemon
    CollaboratorAuthor

    Final scope, after three reversals — this is the decision, and the branch state proves it.

    028 ships the index build (design B) at 8916b38: a 30th named database events_by_term maintained in write_event and hard_delete, a search-index-version marker written last, an open-time rebuild in batches, ADR 0012, and ranking by (min cover span, event_id).

    The scan-time design was also built — green at 6b50424 on factory/028a-nip50-scan, 1253 tests, red-first proven — and it is preserved rather than deleted. It lost on one measurement: it asserts the fixture's set and marks ordering judged, while (min cover span, event_id) reproduces the recorded order and set. B is therefore strictly more parity, and the earlier argument that "the index buys performance, not parity" was wrong exactly where it mattered.

    The flip-flop is in the record on purpose. Earlier comments here and on #1110 describe the scan-time scope, then the index, then the scan again; each was decided on evidence available at the time, and two of those messages never reached the job at all (see pi-extensions#63 — a dispatch to a mid-turn member is not injected until its turn ends, so three rows were answered hours late or not at all).

    Cost 7.50, both builds left in the spans rather than netted out. Out of scope now: nothing from the original issue body; ranked search is in this job, and #1110 keeps only what goes beyond it.

  4. chappie-daemon commented on Nov 11, 2026

    @chappie-daemon
    CollaboratorAuthor

    Follow-ups filed from the PR's review round: #1111 (second in-process handle can lose postings during a rebuild — contract sentence is in ADR 0012 D9, the failing test is not) and #1112 (one-term search proposes every posting sharing the term's first bytes). Both are known limits in the PR body, not defects in it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    factoryBuilt by the software factorylmdbLMDB / strfry-style storage backendrewritenostream 4.0 Rust/LMDB rewriterustRust implementation work

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions