You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[BUG] - Payments taking a long time to clear, or don't clear #342
Hi guys - thanks for making your tool available to the Nostr community!
We'd like to report an issue we have observed with the 'paid relay' feature, which we have deployed to relay.nquiz.io
When we do a fresh deploy, everything works as it should.
But, after some time of usage, generating multiple invoices, of which some are paid, some not, some expired, some not - the relay stops processing payments. So, even though the user has paid the lightning invoice, the relay can't figure it out and the user doesn't get admitted (/invoices page never says "payment successful", QR code remains on screen)
After we notice that this is happening, we perform the following hack:
@nQuiz We are going to need more information to understand what is causing the issue.
Which payment processor are you using?
Can you provide us with a count of the number of invoices by status? e.g. SELECT status, COUNT(*) FROM invoices GROUP BY status
Please enable debug logging when this is happening and provide the logs while attempting to pay an invoice. You can enable debug logging by setting the DEBUG environment variable to worker:*. Make sure to disable debug logging once you are done testing.
I dug into this one a bit. Pretty sure the issue isn’t the payments themselves, it’s the poller window.
getPendingInvoices just grabs a fixed window of ten, oldest first. The maintenance worker hits it every 60s with no paging at all, so it keeps staring at the exact same ten rows every single tick. Once you’ve got ten that never resolve, nothing newer ever gets looked at.
I ran the worker against a local DB with twelve stale invoices and three fresh ones, processor just reporting everything still unpaid:
tick 1: old-00 old-01 old-02 ... old-09
tick 5: old-00 old-01 old-02 ... old-09
invoice states after 5 ticks: pending=15
The three new ones never even get fetched.
As for why they stick around forever: the only place the worker actually retires an invoice is in the catch block, and it needs both a 404 from the processor and isExpiredInvoice coming back true. That second check only passes when the invoice has a real expiry date. expires_at is nullable though, and the Zebedee mapping leaves it null whenever the response doesn’t carry an expiry. Same worker, same setup, but with the processor 404ing on two of them:
So a 404 only cleans up invoices that actually have an expiry date. The null ones just sit there forever. Get ten of those and the poller is stuck, which matches exactly what @nQuiz described: works fine right after a fresh deploy, then dies later.
@cameri I’m guessing @nQuiz’s payment processor is LNURL or Zebedee, since that null expiry comes straight out of its mapping and Zebedee is the default anyway. Status count should show ten or more stuck in pending. Debug logs would just keep saying “found 10 pending invoices” every tick with the same ten ids, never the one that’s actually getting paid.
select status, count(*) from invoices group by status;
selectcount(*) from invoices where status ='pending'and expires_at is null;
There are two fixes for it:
The starvation stops if the offset advances on each tick and wraps around, so every
pending invoice gets a turn. Worth keeping the window itself narrow, since it's
doubling as a rate limiter and plain paging would hammer the processor API.
The pile-up stops if invoices get retired off their own expiry date instead of only
when the processor 404s, which also clears out whatever is already stuck on relays
running today.
Either one by itself should get payments clearing again. Happy to op3n a PR for whichever approach you’d rather go with.
Activity
cameri commentedon Jan 12, 2024
@nQuiz We are going to need more information to understand what is causing the issue.
Which payment processor are you using?
Can you provide us with a count of the number of invoices by status? e.g.
SELECT status, COUNT(*) FROM invoices GROUP BY statusPlease enable debug logging when this is happening and provide the logs while attempting to pay an invoice. You can enable debug logging by setting the
DEBUGenvironment variable toworker:*. Make sure to disable debug logging once you are done testing.Anshumancanrock commentedon Aug 28, 2026
I dug into this one a bit. Pretty sure the issue isn’t the payments themselves, it’s the poller window.
getPendingInvoicesjust grabs a fixed window of ten, oldest first. The maintenance worker hits it every 60s with no paging at all, so it keeps staring at the exact same ten rows every single tick. Once you’ve got ten that never resolve, nothing newer ever gets looked at.I ran the worker against a local DB with twelve stale invoices and three fresh ones, processor just reporting everything still unpaid:
The three new ones never even get fetched.
As for why they stick around forever: the only place the worker actually retires an invoice is in the catch block, and it needs both a 404 from the processor and
isExpiredInvoicecoming back true. That second check only passes when the invoice has a real expiry date.expires_atisnullablethough, and the Zebedee mapping leaves it null whenever the response doesn’t carry an expiry. Same worker, same setup, but with the processor 404ing on two of them:So a 404 only cleans up invoices that actually have an expiry date. The null ones just sit there forever. Get ten of those and the poller is stuck, which matches exactly what @nQuiz described: works fine right after a fresh deploy, then dies later.
@cameri I’m guessing @nQuiz’s payment processor is LNURL or Zebedee, since that null expiry comes straight out of its mapping and Zebedee is the default anyway. Status count should show ten or more stuck in pending. Debug logs would just keep saying “found 10 pending invoices” every tick with the same ten ids, never the one that’s actually getting paid.
There are two fixes for it:
The starvation stops if the offset advances on each tick and wraps around, so every
pending invoice gets a turn. Worth keeping the window itself narrow, since it's
doubling as a rate limiter and plain paging would hammer the processor API.
The pile-up stops if invoices get retired off their own expiry date instead of only
when the processor 404s, which also clears out whatever is already stuck on relays
running today.
Either one by itself should get payments clearing again. Happy to op3n a PR for whichever approach you’d rather go with.
cameri commentedon Sep 7, 2026
Thank you @Anshumancanrock !