Skip to content

Messages sent right after the WebSocket opens are silently dropped (REQ gets no EOSE, EVENT gets no OK) #1044

Closed

Description

@vitorpamplona

A REQ or EVENT that a client sends as soon as its WebSocket opens is often lost. The relay sends nothing back: no EOSE or CLOSED for the REQ, no OK for the EVENT. The client can’t tell that it was lost, so a subscription stays silent and a publish waits until it times out. Every client sends its first REQs and EVENTs on open, and a reconnecting client sends all of them at once, so this affects a large share of connections.

Cause

WebSocketServerAdapter.onConnection (src/adapters/web-socket-server-adapter.ts) awaits the connection rate-limit check before it creates the WebSocketAdapter:

if (await isRateLimited(remoteAddress, currentSettings)) { … }
this.webSocketsAdapters.set(client, this.createWebSocketAdapter([client, req, this]))

The WebSocketAdapter constructor is what attaches client.on('message', …). isRateLimited reads its counters from the rate limiter’s store (Redis by default), so the await takes a round trip. A frame that arrives in the meantime is emitted by ws to no listener, and Node’s EventEmitter drops it without an error. Whether a connection loses its first messages depends on whether the client’s frame or the store’s reply arrives first, which is why only some connections are affected.

The connection rate limits are on in default-settings.yaml (limits.connection.rateLimits), so a default install has the race.

To reproduce

nostream 3.11.0 (ghcr.io/cameri/nostream:latest) with Postgres and Redis, default settings, relay on port 7784. This Node 22 script opens connections one after another and sends a REQ delayMs after each open:

// node first-msg.mjs ws://127.0.0.1:7784 0 10
const [url, delayMs = '0', n = '6'] = process.argv.slice(2)
function once(i) {
return new Promise((resolve) => {
const ws = new WebSocket(url)
const t0 = Date.now()
let answered = null
ws.onopen = () => setTimeout(() => ws.send(JSON.stringify(['REQ', 'r' + i, { kinds: [1], limit: 1 }])), Number(delayMs))
ws.onmessage = (m) => { const msg = JSON.parse(m.data); if (!answered && ['EOSE', 'CLOSED', 'NOTICE'].includes(msg[0])) answered = ${msg[0]} after ${Date.now() - t0} ms }
setTimeout(() => { ws.close(); resolve(conn ${i}: ${answered ?? 'UNANSWERED'}) }, 3000)
})
}
for (let i = 0; i < Number(n); i++) console.log(await once(i))

SetupREQs answeredDefault settings, REQ sent on open2 of 6, and 7 of 10 in a second run (no connection was rate-limited or terminated)Default settings, REQ sent 1 s after open4 of 4Client’s address added to limits.connection.ipWhitelist, so the check returns without touching Redis8 of 8With the fix in the linked PR, default settings10 of 10EVENTs are affected the same way: in a test harness publishing on fresh connections, 2 of 6 EVENTs got no OK and were never stored.

Expected behavior

Every message the client sends after the socket opens is handled, or answered with a CLOSED/NOTICE/OK saying why not.

Environment

•	nostream 3.11.0, Docker image ghcr.io/cameri/nostream:latest, Postgres 16, Redis 7
•	Clients: a plain Node 22 WebSocket client (above) and Quartz’s Kotlin client

Activity

  1. cameri commented on Nov 3, 2026

    @cameri
    Owner

    @vitorpamplona thanks for the detailed issue

  2. chappie-daemon commented on Nov 4, 2026

    @chappie-daemon
    Collaborator

    Independent reproduction on a second deployment, with one extra measurement that separates the cause from look-alike symptoms.

    Environment. Two different instances, both default settings with connection rate limits on: the reporter's relay at 3.11.0 and a separate container running 3.8.0. The 3.8.0 instance answered with the same script from the issue, unchanged, against ws://<host>:8008:

    conn 0: EOSE after 23 ms      conn 4: EOSE after 5 ms
    conn 1: EOSE after 12 ms      conn 5: UNANSWERED
    conn 2: EOSE after 5 ms       conn 6: UNANSWERED
    conn 3: EOSE after 11 ms      conn 7: EOSE after 10 ms
    

    A second pass of 10 connections lost 1. Control with the REQ delayed 1000 ms after onopen: 4 of 4 answered, every time.

    Dropped frame, live socket. For each connection I sent a REQ immediately and a second REQ on the same socket 600 ms later, and recorded which subids came back:

    conn 6: first(immediate)=LOST  second(+600ms)=answered
    

    On the connection that lost its first REQ, the same socket answered a REQ 600 ms later — no reconnect, no second handshake. That rules out the two explanations which look identical from a client's side:

    • a rate-limit rejection, which would have called client.terminate() and left nothing to answer on;
    • a slow query, which would have produced a late EOSE rather than none.

    What remains is a socket that was open and served later frames, with the earliest frame emitted before anything was listening for it.

    The code still has the ordering. origin/main at 3.12.0, src/adapters/web-socket-server-adapter.ts line 137:

    if (await isRateLimited(remoteAddress, currentSettings)) {

    precedes this.webSocketsAdapters.set(client, this.createWebSocketAdapter([client, req, this])), and the adapter's constructor is what attaches client.on('message', …).

    No fix is proposed here — this comment adds measurements only.

  3. Ferryx349 commented on Nov 4, 2026

    @Ferryx349
    Collaborator

    @vitorpamplona Thanks again, we’ve implemented the fix in #1045.

    Could you please try it with your reproduction against 3.12.1 / the latest image, and let us know if it works for you now? Specifically, we’d like to confirm that a REQ sent immediately on open always gets an EOSE/CLOSED, and an EVENT gets an OK. Rate-limited connections should still be terminated as before.

    Any remaining failures would be very helpful.

  4. vitorpamplona commented on Nov 4, 2026

    @vitorpamplona
    Author

    Yes, the fix works.

  5. Ferryx349 commented on Nov 4, 2026

    @Ferryx349
    Collaborator

    8. Yes, the fix works.

    Thank You!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions