Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
e120443
feat(storage): make the lmdb feature the backend's slot and retire th…
chappie-daemon Nov 6, 2026
544630b
feat(storage): implement the nine repositories over lmdb
chappie-daemon Nov 6, 2026
391ee8d
test(storage): replay the parity fixture through a shared nine-trait …
chappie-daemon Nov 6, 2026
b53c646
docs(storage): record the lmdb backend and add its oracle step
chappie-daemon Nov 6, 2026
509a8fc
build(storage): lock the lmdb feature's heed and uuid edges
chappie-daemon Nov 6, 2026
c66fa9f
docs(storage): mark each lmdb ceiling where a reader meets it
chappie-daemon Nov 6, 2026
ebaa971
docs(storage): mark the stream's snapshot ceiling at its type
chappie-daemon Nov 6, 2026
47e79d7
fix(storage): prune only the created-at scan and reconcile every batch
chappie-daemon Nov 6, 2026
871ca6c
fix(storage): key the delivery-log outbox index by entry id
chappie-daemon Nov 6, 2026
6f461fe
fix(storage): compare hex criteria as the range builder seeks them
chappie-daemon Nov 6, 2026
a4a503f
refactor(storage): drop the unreachable lmdb data-directory error
chappie-daemon Nov 6, 2026
05ef862
test(storage): claim the limited page, batch, tag and outbox-key rules
chappie-daemon Nov 6, 2026
6d774ff
docs(storage): correct the reconciliation note in ADR 0010
chappie-daemon Nov 6, 2026
c06b5d2
test(storage): claim the retention batch ceiling and its classification
chappie-daemon Nov 6, 2026
75b7ea1
test(storage): claim the empty criterion family
chappie-daemon Nov 6, 2026
0b2d4b1
refactor(storage): give the lmdb key codecs one home
chappie-daemon Nov 6, 2026
9656faa
perf(storage): bound a read to its page and report the map's fill
chappie-daemon Nov 6, 2026
731dcb1
docs(storage): correct the read bound and the value layout in ADR 0010
chappie-daemon Nov 6, 2026
85793c1
refactor(storage): let the tag range be the proof the key carries
chappie-daemon Nov 6, 2026
60fe1bf
docs(storage): mark the ranges a page cannot stop inside
chappie-daemon Nov 6, 2026
f260d9c
test(storage): pin the reconciliation range to the keys it holds
chappie-daemon Nov 6, 2026
db5e164
docs(storage): say how a tags range proves its family
chappie-daemon Nov 6, 2026
f748967
docs(storage): state the tag key's exact length bound
chappie-daemon Nov 6, 2026
760c8c4
fix(storage): compare a hex prefix against its own field
chappie-daemon Nov 6, 2026
7a7e45e
fix(storage): move the map usage report off the caller's thread
chappie-daemon Nov 6, 2026
61a6bd4
docs(storage): give the stream's page bound its filters
chappie-daemon Nov 6, 2026
514ae25
docs(storage): say what the range dedupe covers
chappie-daemon Nov 6, 2026
94d6599
fix(storage): encode dedup keys without escaping and refuse NUL text
chappie-daemon Nov 6, 2026
3add554
perf(storage): bound a filter's ranges and a stream's id scan
chappie-daemon Nov 6, 2026
fbc99a9
test(storage): cover an ids reconciliation and a NUL target id
chappie-daemon Nov 6, 2026
1168af0
docs(storage): record the key bounds and the stream's ids path
chappie-daemon Nov 6, 2026
a4064e1
docs(storage): mark the id stream's whole-list read
chappie-daemon Nov 6, 2026
71162c3
docs(storage): say what the page bound counts
chappie-daemon Nov 6, 2026
1388c56
fix(storage): give a page one accumulator and bound a prefix id scan
chappie-daemon Nov 6, 2026
ff7b616
docs(storage): say which ids read holds the request's list
chappie-daemon Nov 6, 2026
c04467f
test(storage): pin an unlimited page over two tag values
chappie-daemon Nov 6, 2026
6d1372b
perf(storage): keep a page as a heap so an offer costs a comparison
chappie-daemon Nov 7, 2026
1454c1d
docs(storage): record the author and kind composition the plan performs
chappie-daemon Nov 7, 2026
9006fec
test(storage): quote the corpus the cost test measures
chappie-daemon Nov 7, 2026
b12e647
test(storage): hold the counting lock in the page-holds test
chappie-daemon Nov 7, 2026
51f2fe1
docs(storage): record the schema contract the store implements
chappie-daemon Nov 7, 2026
6f5b2e4
docs(storage): point the NIP-50 gate note at what the ADR holds
chappie-daemon Nov 7, 2026
b426fc5
docs(storage): correct two ADR references that point at nothing
chappie-daemon Nov 7, 2026
db95029
docs(storage): cite the sqlx migration tree by its full path
chappie-daemon Nov 7, 2026
2e348bc
fix(storage): reduce a user patch's flags as the postgres merge does
chappie-daemon Nov 8, 2026
631255c
fix(storage): keep one row per replaceable address in an lmdb batch
chappie-daemon Nov 8, 2026
861f171
fix(storage): decide an lmdb replacement's merge from the header
chappie-daemon Nov 8, 2026
dc6db6b
docs(storage): number the lmdb decision record 0011
chappie-daemon Nov 8, 2026
eb44918
fix(storage): match postgres on lmdb replays, bulk merges, tag counts…
chappie-daemon Nov 8, 2026
4d72873
fix(storage): bound lmdb metadata reads and refuse a replayed dvm job
chappie-daemon Nov 8, 2026
fd159c5
fix(storage): bound the lmdb claim and the delivery-log index key
chappie-daemon Nov 8, 2026
50d09ea
fix(storage): count an lmdb tag walk by the id in its key
chappie-daemon Nov 8, 2026
08a7563
refactor(storage): pass the merge predicate its created-at and id pairs
chappie-daemon Nov 8, 2026
c6a2381
refactor(storage): give the text-column NUL rule one owner
chappie-daemon Nov 8, 2026
fb73314
docs(storage): make the lmdb decision record match the code
chappie-daemon Nov 8, 2026
4e5cd43
fix(storage): hold an lmdb count's ids only for the rows it matches
chappie-daemon Nov 8, 2026
785e1ea
test(storage): count the decode an lmdb count no longer makes
chappie-daemon Nov 8, 2026
94a9815
docs(storage): qualify the lmdb record's residency and ordering claims
chappie-daemon Nov 8, 2026
a98ad60
fix(storage): bound an lmdb count's id set and check it before the ma…
chappie-daemon Nov 8, 2026
f31011f
docs(storage): record the count's id cap and what it costs
chappie-daemon Nov 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .changeset/storage-lmdb-backend.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
---
---
5 changes: 4 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,10 @@ cargo test -p nostream-storage --test migrations --test parity -- --ignored --te
`PARITY_ADMIN_URL` names the administrative database (CI sets
`postgres://postgres:postgres@localhost:5432/postgres`); the targets create, migrate and
drop their own scratch databases. `scripts/storage-parity.sh` runs both plus the
TypeScript Cucumber suite against the sqlx-migrated schema — the complete local oracle.
TypeScript Cucumber suite against the sqlx-migrated schema — the complete local oracle. The
LMDB arm of that oracle needs no database: `cargo test -p nostream-storage --test parity_lmdb
-- --nocapture` replays the same fixture on a temporary store and runs inside the fifth
validation command above.

- **`--workspace`, never a single crate.** A crate-local `cargo test -p` proves nothing
about a workspace lint that another member trips.
Expand Down
13 changes: 13 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ tracing = "0.1"
tracing-opentelemetry = "0.34"
tracing-subscriber = { version = "0.3", features = ["env-filter", "fmt"] }
url = "2"
uuid = { version = "1", features = ["v4"] }

[workspace.lints.rust]
# `deny` rather than `forbid`: heed's LMDB `EnvOpenOptions::open` is an `unsafe fn`
Expand Down
17 changes: 11 additions & 6 deletions crates/nostream-storage/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,26 +5,31 @@ edition.workspace = true
rust-version.workspace = true
license.workspace = true
repository.workspace = true
description = "Repository traits, backend selection and the Postgres storage backend"
description = "Repository traits, backend selection and the Postgres and LMDB storage backends"

[features]
# The Postgres backend and its driver. On by default because Postgres is the reference
# implementation; a build that only needs the trait seam (the LMDB follow-up) can turn it
# off and lose the sqlx link entirely.
default = ["postgres"]
# Both backends are on by default: Postgres is the reference implementation and the LMDB
# backend is what the rewrite exists to provide. A build that only needs the trait seam
# turns both off and links no driver at all.
default = ["postgres", "lmdb"]
postgres = ["dep:sqlx"]
# The LMDB backend, its environment and its row encodings. `uuid` is only used to mint
# the canonical text ids the outbox and delivery-log columns generate.
lmdb = ["dep:heed", "dep:uuid", "dep:tracing"]

[dependencies]
chrono.workspace = true
futures-core.workspace = true
heed.workspace = true
heed = { workspace = true, optional = true }
nostream-config.workspace = true
nostream-core.workspace = true
serde.workspace = true
serde_json.workspace = true
sqlx = { workspace = true, optional = true }
thiserror.workspace = true
tokio.workspace = true
tracing = { workspace = true, optional = true }
uuid = { workspace = true, optional = true }

[dev-dependencies]
serde_yaml.workspace = true
Expand Down
136 changes: 50 additions & 86 deletions crates/nostream-storage/src/backend.rs
Original file line number Diff line number Diff line change
@@ -1,47 +1,33 @@
//! Backend selection and the two backends. Mirrors `src/utils/event-store.ts` (the selection
//! half) and the storage boundary in `src/@types/adapters.ts`.
//! Backend selection and the LMDB environment. Mirrors `src/utils/event-store.ts` (the
//! selection half) and the storage boundary in `src/@types/adapters.ts`.

use std::path::{Path, PathBuf};

use heed::types::Str;
use nostream_core::EventStoreBackend;

use crate::error::StorageError;
use heed::{Database, Env, EnvOpenOptions};

/// The storage backends the rewrite can be configured with (`src/constants/base.ts`
/// `EventStoreBackend`).
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum Backend {
Postgres,
Lmdb,
}
#[cfg(feature = "lmdb")]
use heed::{Env, EnvOpenOptions};

impl Backend {
/// The settings spelling of this backend.
pub const fn as_str(self) -> &'static str {
match self {
Self::Postgres => "postgres",
Self::Lmdb => "lmdb",
}
}
#[cfg(feature = "lmdb")]
use crate::error::StorageError;
#[cfg(feature = "lmdb")]
use crate::lmdb::{LMDB_MAX_DBS, LMDB_MAX_READERS};

/// Parses a settings value.
pub fn parse(value: &str) -> Option<Self> {
match value {
"postgres" => Some(Self::Postgres),
"lmdb" => Some(Self::Lmdb),
_ => None,
}
}
}
/// The directory one backend's data lives in, under the root a caller supplies.
/// `src/@types/adapters.ts` names the LMDB store as a directory of its own beside the
/// database it replaces.
pub const LMDB_DIR: &str = "lmdb";

impl std::fmt::Display for Backend {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter.write_str(self.as_str())
/// Resolves the data directory under `root` for one backend. A backend that keeps its data
/// somewhere other than a directory of its own answers `None`.
pub fn data_dir(root: &Path, backend: EventStoreBackend) -> Option<PathBuf> {
match backend {
EventStoreBackend::Lmdb => Some(root.join(LMDB_DIR)),
EventStoreBackend::Postgres => None,
}
}

/// An opened LMDB environment with one named database of string keys and values.
/// An opened LMDB environment, with the named databases the backend creates.
///
/// The constructor is the crate's single `unsafe` boundary. heed 0.20.5 marks
/// `EnvOpenOptions::open` unsafe because an LMDB environment memory-maps a file shared with
Expand All @@ -51,34 +37,36 @@ impl std::fmt::Display for Backend {
/// and `Error::BadOpenOptions` when they differ (heed-0.20.5 `src/env.rs`, `OPENED_ENV`).
/// heed's documented preconditions, and where each stands today:
///
/// - **No long-lived transactions** — upheld by the code: `put`/`get` open a transaction,
/// use it, and commit/drop it before returning; nothing holds one across an await.
/// - **No long-lived transactions** — upheld by the code: every repository call opens a
/// transaction, uses it, and commits or drops it inside one `spawn_blocking` closure;
/// nothing holds one across an await.
/// - **No aborting the process with an active transaction** — relied on the operator for
/// SIGKILL-class exits; the Phase-2 daemon drain path must ensure a clean shutdown
/// SIGKILL-class exits; the daemon drain path must ensure a clean shutdown
/// (SIGINT/SIGTERM drain through `ShutdownState` does not abort mid-transaction).
/// - **No network filesystems** — relied on the operator today: `open` accepts any path and
/// only `create_dir_all` before mapping; the Phase-2 backend factory must refuse a remote
/// location before this slice is wired into settings.
/// - **No network filesystems** — relied on the operator: `open` accepts any path and only
/// `create_dir_all` before mapping; the daemon's backend factory must refuse a remote
/// location before this environment is wired into settings.
/// - **`NO_LOCK` requires self-managed concurrency; a broken lock file means
/// synchronization issues and possible UB** — split responsibility. The code guarantees
/// its half: no env flags are passed (`EnvOpenOptions::new()` only), so `NO_LOCK` is
/// never in play and this crate never reads, writes or removes the lock file. The
/// operator must supply the rest: the data directory's lock file must not be broken
/// (stale locks left by a crashed process on a filesystem without proper flock support,
/// manual deletion, two processes on different hosts sharing a volume — LMDB cannot
/// recover from any of those, per heed's contract).
/// operator must supply the rest: the data directory's lock file must not be broken.
/// - **One `map_size` per path for the life of the process** — upheld by heed + propagated:
/// heed's per-process `OPENED_ENV` registry returns the same `Env` for a second open with
/// equal options and `Error::BadOpenOptions` when they differ, and `open` surfaces that
/// error to the caller rather than opening a second environment.
/// error to the caller rather than opening a second environment. `Env::resize` is unsafe
/// as well, so the map is sized once and never grown.
#[cfg(feature = "lmdb")]
#[derive(Debug, Clone)]
pub struct LmdbEnv {
env: Env,
path: PathBuf,
}

#[cfg(feature = "lmdb")]
impl LmdbEnv {
/// Opens (creating if needed) an LMDB environment at `path` with `map_size` bytes.
/// Opens (creating if needed) an LMDB environment at `path` with `map_size` bytes, room
/// for the backend's named databases and its reader slots.
pub fn open(path: impl Into<PathBuf>, map_size: usize) -> Result<Self, StorageError> {
let path = path.into();
std::fs::create_dir_all(&path).map_err(|source| StorageError::CreateDirectory {
Expand All @@ -87,40 +75,26 @@ impl LmdbEnv {
})?;

// # Safety: the preconditions on `LmdbEnv` above, each marked upheld or relied-on:
// no env flags (LMDB's locking runs), transactions never escape the call, and
// heed's per-process dedup turns a second open into the same Env or
// BadOpenOptions. The network-filesystem precondition rests on the operator
// passing a local path; see the list above.
// no env flags (LMDB's locking runs), transactions never escape the closure they are
// opened in, and heed's per-process dedup turns a second open into the same Env or
// BadOpenOptions. The network-filesystem precondition rests on the operator passing a
// local path; see the list above.
#[allow(unsafe_code)]
let env =
unsafe { EnvOpenOptions::new().map_size(map_size).open(&path) }.map_err(|source| {
StorageError::OpenLmdb {
path: path.clone(),
source,
}
})?;
let env = unsafe {
EnvOpenOptions::new()
.map_size(map_size)
.max_dbs(LMDB_MAX_DBS)
.max_readers(LMDB_MAX_READERS)
.open(&path)
}
.map_err(|source| StorageError::OpenLmdb {
path: path.clone(),
source,
})?;
Ok(Self { env, path })
}

/// Writes one value under one key, committing immediately.
pub fn put(&self, key: &str, value: &str) -> Result<(), StorageError> {
let mut transaction = self.env.write_txn()?;
let database: Database<Str, Str> = self.env.create_database(&mut transaction, None)?;
database.put(&mut transaction, key, value)?;
transaction.commit()?;
Ok(())
}

/// Reads one value; `Ok(None)` means the key is absent.
pub fn get(&self, key: &str) -> Result<Option<String>, StorageError> {
let transaction = self.env.read_txn()?;
let Some(database) = self.env.open_database::<Str, Str>(&transaction, None)? else {
return Ok(None);
};
Ok(database.get(&transaction, key)?.map(str::to_owned))
}

/// The environment handle, for the Phase-2 repository implementations.
/// The environment handle, for the repository implementations.
pub fn env(&self) -> &Env {
&self.env
}
Expand All @@ -130,13 +104,3 @@ impl LmdbEnv {
&self.path
}
}

/// Resolves the database path under `root` for one backend.
///
/// Phase 1 stub — job 005; will pick the per-backend data directory; see `src/@types/adapters.ts`.
pub fn data_dir(root: &Path, backend: Backend) -> Result<PathBuf, StorageError> {
let _ = (root, backend);
Err(StorageError::NotImplemented {
module: "storage::backend::data_dir",
})
}
7 changes: 7 additions & 0 deletions crates/nostream-storage/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -49,17 +49,24 @@ pub enum StorageError {
ValueOutOfRange { column: &'static str, value: String },

/// An LMDB environment or transaction failed.
#[cfg(feature = "lmdb")]
#[error("lmdb operation failed: {0}")]
Lmdb(#[from] heed::Error),

/// An LMDB environment could not be opened at the requested path.
#[cfg(feature = "lmdb")]
#[error("failed to open the lmdb environment at {path}: {source}")]
OpenLmdb {
path: std::path::PathBuf,
#[source]
source: heed::Error,
},

/// An LMDB blocking task was cancelled before it returned (the runtime is shutting down).
#[cfg(feature = "lmdb")]
#[error("the lmdb task did not finish")]
LmdbTaskCancelled,

/// A value must be valid UTF-8 to be stored through the string codecs.
#[error("lmdb values must be valid utf-8")]
NotUtf8,
Expand Down
27 changes: 18 additions & 9 deletions crates/nostream-storage/src/lib.rs
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
//! `nostream-storage` owns the persistence seam: the nine repository traits, the streaming
//! query surface, the Postgres backend with its embedded migrations, the backend selection
//! and the LMDB environment.
//! query surface, the Postgres backend with its embedded migrations, the LMDB backend, the
//! backend selection and the LMDB environment.
//!
//! It owns these TypeScript paths from the capture commit
//! (`242a3001d77149e4d70151ec03060c4ef44a1547`):
//!
//! - `src/repositories/` (9 files, ported to `src/postgres/`)
//! - `src/repositories/` (9 of the 10 files, ported to `src/postgres/` and `src/lmdb/`)
//! - `src/database/client.ts`, `src/database/transaction.ts` (`src/postgres/mod.rs`)
//! - the schema: `migrations/` (37 knex migrations, ported to `migrations/*.sql`)
//! - `src/utils/event-store.ts` (the mirror of `src/utils/hidden-content-cache.ts` is the
Expand All @@ -16,21 +16,30 @@
//! - `src/factories/event-repository-factory.ts`
//!
//! What it deliberately does **not** own: the relay handlers that call these traits, the
//! payments service, and the LMDB backend. The cache is not a cache server: no Redis client
//! is added and `src/cache/client.ts` / `src/adapters/redis-adapter.ts` are not ported — the
//! decision and what would overturn it are ADR
//! `0010-services-worker-model-and-cache.md`. `LmdbEnv`'s environment slice and
//! `backend::data_dir` remain labelled stubs; the Postgres backend is complete.
//! payments service, the daemon's backend factory and the `nostream-cli migrate pg-to-lmdb`
//! tool. The cache is not a cache server: no Redis client is added and
//! `src/cache/client.ts` / `src/adapters/redis-adapter.ts` are not ported — the decision and
//! what would overturn it are ADR `0010-services-worker-model-and-cache.md`. Both backends are
//! complete: Postgres against the 37-migration schema, and LMDB against the same column
//! contract for 9 of the capture's 10 repositories — `user_subscriptions` has a TypeScript
//! repository and no trait yet, and `events.first_seen` is stored but never indexed
//! (`docs/rewrite/decisions/0011-lmdb-backend.md`).

pub mod backend;
pub mod error;
pub mod event_store;
#[cfg(feature = "lmdb")]
pub mod lmdb;
#[cfg(feature = "postgres")]
pub mod postgres;
pub mod records;
pub mod repositories;

pub use backend::{Backend, LmdbEnv};
#[cfg(feature = "lmdb")]
pub use backend::LmdbEnv;
pub use backend::data_dir;
pub use error::StorageError;
#[cfg(feature = "lmdb")]
pub use lmdb::{LmdbBackend, MapUsage};
#[cfg(feature = "postgres")]
pub use postgres::{PostgresBackend, PostgresTransaction};
Loading
Loading