[user properties 2/2] Keep user properties out of the vector store

[edwinyyyu]

Purpose of the change

The event backend wrote every property of an event into its vector record and mapped the caller's whole filter onto the vector store, so a user key a deployment never declared was both stored and filtered there: on Qdrant and Milvus as unindexed payload a filtered query scans for. With #1670 a tenant can no longer declare indexes; this closes the other half, so a tenant's properties cannot shape what the vector store stores or scans either. The segment store already holds every property and already receives the whole filter for the context windows, so the vector side only duplicated work the segment store does anyway.

The vector record now carries the keys the collection declares: EventMemory's reserved timestamp, and the _-prefixed system properties an adapter stamps on the event, which after #1670 are exactly the collection's schema (collection.config.indexed_properties_schema). The vector store is queried with the conjuncts of the filter that name only such fields; a conjunct is dropped whole when any field under it is a user property, so dropping only ever widens the vector search, and the segment store narrows it back on the windows. User keys never reach the vector store.

filter_fields joins the filter parser: every field name a tree addresses. Tests: a user property stays on the segment and off the record; a user-property conjunct reaches only the segment store; a user property under an OR leaves the vector search unfiltered; filter_fields names every field under every node.

This is the routing half of what was #1628 on speedkick (its store-side half, the stores rejecting undeclared keys, is deferred to after #1627); it is re-derived onto #1736's collection shape from the same commit, where the segment store, not the vector record, maps a derivative to its segment.

Stack

21 open PRs: one independent PR, and the vector store tree of short parallel branches. Every PR's GitHub base is main, since the branches are in a fork and a pull request can target only this repository's branches; the on column gives the order the PRs build on each other instead. A stacked PR's diff on GitHub includes the PRs under it until they merge.

Independent of the vector store tree, directly on main:

#	PR	change	on
—	#1624	Make no memory request create a project	main

The vector store tree. Each PR builds on the one in its on column; PRs on the same parent are parallel branches and do not depend on each other. #1631 is closed, superseded by #1733–#1736, which hold its changes split in four, with review changes since. #1670 and #1702 sit beneath #1627, whose code depends on them. Until the PRs under it merge, their changes show in a stacked PR's diff.

#	PR	change	on
[vector store scale-out 1/6]	#1671 (merged)	Remove custom sharding from the Qdrant store (port of #1654)	main
[vector store scale-out 2/6]	#1733	Answer vector store queries with record UUIDs and scores, and refuse invalid inputs	main
[vector store scale-out 3/6]	#1734	Arbitrate vector store collections in a SQL registry, with an incarnation per collection life and a purge	#1733
[vector store scale-out 4/6]	#1735	Move the Qdrant store onto the collection registry	#1734
[vector store scale-out 5/6]	#1736	Move the Milvus store onto the collection registry, against a Milvus server	#1735
—	#1631 (closed)	Superseded by #1733–#1736, which hold its changes split in four, with review changes since	—
[user properties 1/2]	#1670	Remove per-project filterable properties (port of #1606)	#1736
[user properties 2/2]	#1702 (this PR)	Keep user properties out of the vector store	#1670
[vector store scale-out 6/6]	#1627	Make a vector store one collection, with string-keyed partitions	#1702
[session storage 1/2]	#1622	Create a session's storage with the session, never on a request	#1627
[session storage 2/2]	#1625	Remove open-or-create from both stores, and close from the segment store	#1622
[search results]	#1663	Score every vector search by cosine similarity, and name scores for it (port of #1598's cosine half)	#1627
[sqlite store fixes 1/7]	#1460	Publish vector index files atomically (but not durably)	#1663
[sqlite store fixes 2/7]	#1469	Never reuse a row id in SQLiteVectorStore	#1460
[sqlite store fixes 3/7]	#1672	Own the search engine's concurrency in the store, not in each engine (port of #1612)	#1469
[sqlite store fixes 4/7]	#1673	Serialize a partition's writes so the engine sees them in order (port of #1607)	#1672
[sqlite store fixes 5/7]	#1674	Refuse a pending row replay cannot honor, instead of dropping it (port of #1608)	#1673
[sqlite store fixes 6/7]	#1675	Take SQLite's write lock at BEGIN, not at the first write (port of #1609)	#1674
[sqlite store fixes 7/7]	#1676	Give every write a fresh row id, so a key names one version (port of #1610)	#1675
[qdrant options]	#1618	Let a deployment tune a Qdrant collection's HNSW, optimizers and quantization	#1663
[declared schema 1/2]	#1628	Make a vector store filter only on the properties it declares	#1663
[declared schema 2/2]	#1616	Close the filter union, and make negation the complement on every backend	#1628

This PR is its one commit, 23761359f, stacked on #1670. #1627 is stacked on it.

Verification

At every commit from #1736's head b3baee60e to this PR's head 23761359f (this PR and the PRs under it), on 2026-10-02: ruff check and ruff format --check clean; ty check clean as CI runs it (uv run --frozen --all-extras ty check --project packages/server); the server suite without integration tests passes, 2104 tests at this PR's head. The integration tests of the vector stores, the resource manager, episodic memory and semantic storage, against PostgreSQL 16, Qdrant 1.19.1 and Milvus 2.6.24 in containers, pass at the heads of #1627 (603), #1625 (575), #1676 (600), #1618 (600) and #1616 (670).

🤖 Generated with Claude Code

https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn