[TheodoreSpeaks]

Summary

Background jobs, async exports, and a systemic performance pass for Tables.

Background row deletes + table_jobs

• "Select all" deletes (filter + optional deselections) send { filter, excludeRowIds } and delete in keyset-paginated background batches; rows inserted mid-job are spared via a created_at cutoff.
• New table_jobs table (one row per job, type = import | delete | export | backfill). Concurrency gate: partial-unique index on (table_id) WHERE status='running' AND type <> 'export' — one writing job per table; read-only exports run concurrently.
• Read-path mask: while a delete job runs, every read (grid, exports, search) hides the doomed rows, so mid-job reads match the eventual result.

Jobs on trigger.dev

• Import, delete, export, and output-column backfill run as trigger.dev tasks (apps/sim/background/) with an in-process runDetached fallback — jobs survive deploys/restarts.
• Backfill is hybrid: ≤500 completed runs inline (response returns consistent), above that a background job.

CSV export

• Hybrid by size: ≤10k rows keep the synchronous stream; larger tables run an async export job → storage → presigned download (auto-downloads for the initiating session).
• Unified jobs tray (imports + exports) with aggregate state icon, live progress, cancel, and re-download.

Performance (measured on a 1M-row table in a 12M-row shared relation)

Path	Before	After
1M-row delete	~25 min	seconds/page — (table_id, id) index (0231)
1M-row export	~30 min	~1 min — keyset pagination (OFFSET was O(N²))
Grid infinite scroll / drains	OFFSET per page	(order_key, id) cursor seek
Filtered count (ILIKE)	11–12.7s	0.6s
Sorted view, per page	9.7s + disk spills	0.76s
Cmd+F search	75s	2s
Equality filters / $in	326ms–1.1s	17ms — tenant-scoped GIN (0232)
Unique-constraint check (every write)	3.5s	<1s

Root cause for most of these: jsonb predicates are unestimatable, so the planner picked parallel seq scans over the entire shared relation (every tenant's rows). Two mechanisms fix it:

• withSeqscanOff (lib/table/planner.ts): transaction-scoped SET LOCAL enable_seqscan = off around the queries with unindexable predicates (ILIKE/range/lower()/sort keys).
• Migration 0232: replaces the cross-tenant GIN on data with btree_gin (table_id, data jsonb_path_ops) — containment lookups resolve tenant-scoped inside the index (and the index is smaller, 529MB vs 694MB on dev).
• Autovacuum tuned to 2% churn on the shared relation (0231) so one tenant's mass delete doesn't degrade reads for days.

Fit & finish

• Error toasts surface the real failure (drizzle wraps DB errors; routes now classify on the root cause) — e.g. "Row limit exceeded — this table is capped at 10,000 rows" instead of "Failed to insert row".
• Export tray icon appears immediately on kickoff; tray restored on the tables list page.
• Import batches 5k rows (bind-param bounded), delete pages 10k.

Migrations

• 0231_table_jobs_and_keyset: table_jobs + import-state data migration + (table_id, id) index + autovacuum tuning + import_* column drops.
• 0232_tenant_scoped_data_gin: btree_gin extension + tenant-scoped containment index (replaces user_table_rows_data_gin_idx).

Test plan

• bunx vitest run lib/table app/api/table app/api/v1 hooks/queries (395 tests)
• bun run lint:check, tsc --noEmit, bun run check:api-validation
• Manual: 1M-row table — select-all delete, async export + download, filtered/sorted scrolls, Cmd+F, unique-column inserts; trigger.dev path with TRIGGER_DEV_ENABLED=true; EXPLAIN ANALYZE before/after for each perf claim.

🤖 Generated with Claude Code