Repository navigation
The cpython github merge webhook to bugs.python.org is broken #301
Description
The github webhook that sends cpython merge updates to individual issues on bpo is failing (https://bugs.python.org/pull_request). I'm guessing it may have something to do with the recent upgrade and move of bpo to a different system. I don't have time to track it down myself at the moment but this is a pretty critical item as we are losing documentation in bpo of pushed merges.
Activity
Mariatta commentedon Dec 23, 2018
GitMate.io thinks possibly related issues are python/devguide#4 (Migrate over issues from bugs.python.org), #28 (Add links back to bugs.python.org), #15 (Shut down Rietveld instance on bugs.python.org), #292 (Links to www.bugs.python.org), and python/devguide#79 (https://docs.python.org/devguide/ is broken?).
Mariatta commentedon Dec 23, 2018
Webhook are returning 400 error from bpo.
Mariatta commentedon Dec 23, 2018
I don't have access to bpo server log. I don't know who would be able to look into this.
ned-deily commentedon Dec 24, 2018
Thanks, @Mariatta. @ezio-melotti and/or @ewdurbin have access, I believe.
ewdurbin commentedon Dec 24, 2018
Fairly clear error message:
ERROR:root:Missing SECRET_KEY environment variable set!Appears configuration is the only non ini file based config in the roundup instance, and was missed.
I'm working to resolve it now and will replay all backlogged web hooks after its restored.
ewdurbin commentedon Dec 24, 2018
https://bitbucket.org/python/roundup/commits/59d6d4284aea7df30344758fe391478a47609c2e resolves, it's deployed and hooks are flowing. I'm trying to determine how to mass replay the hooks now.
ewdurbin commentedon Dec 24, 2018
I've submitted a support request to GitHub... it appears our only option for redelivery of the failed hooks is to click the buttons in the GitHub UI (or script something to do so)... but not too fast! if you click them too fast you'll get rate limited :)
ned-deily commentedon Dec 24, 2018
Thanks for getting it working again, @ewdurbin! But, yikes, I hope you don't have to push every button!
ewdurbin commentedon Dec 24, 2018
Welp, clicked and tapped and it’s done. 😅
Gonna leave this issue open until we confirm things are flowing/behaving as expected.
ewdurbin commentedon Dec 26, 2018
Got confirmation from GitHub that there is no automated way to mass redeliver failed hooks.
ewdurbin commentedon Dec 26, 2018
Okay, looks like we had some kerfuffle in delivery over the past few days.
To address this I've refactored each roundup instance on the new host to run its own gunicorn stack and gracefully reload to maintain availability: python/psf-salt@ebacc0a
Will continue to watch for a few days :)
ned-deily commentedon Dec 31, 2018
Thanks, @ewdurbin! I think the merge messages are now showing up as expected. But there is still at least one remaining transition issue: the multiple identical PR links showing up in b.p.o issues as referenced above.