A deploy restart cuts off requests in flight: drain them issue #233 closed: completed andymccall opened this · 2 comments
andymccall set labels bug ·
Code done in the commit above: SIGTERM drains (up to 30 s, event streams ended), and the hub serves a systemd-held socket when given one. To finish this on Earth, oxygenhub.socket and the service changes in docs/deploy.md (Restarts without errors) need installing; until then a restart still refuses connections for the second or two between processes. Left open for that step.
andymccall referenced by change c92a99490cba on main ·
andymccall referenced by change 529c703db24d on main ·
Done on Earth (2026-10-17, Andy's go-ahead): oxygenhub.socket on 127.0.0.1:7300 plus a drop-in (Requires/After the socket, TimeoutStopSec=40). The log says 'http on the socket systemd holds'; a restart while /health was asked 10 times a second answered all 120 with 200. Together with the drain (5a7a1c24) and the bounded exit (a42a5831), deploys are now invisible.
closed by change 9aee52d392a6 ·
Andy, 2026-10-16 09:25: a deploy restarted the hub while his metalgear2 proposal page was reloading; Cloudflare showed 502. systemctl stop takes about a second and drops what is running. Fix: on SIGTERM stop accepting, finish requests in flight (axum graceful shutdown, a bound such as 60 s, longer than the slowest page), then exit; Cloudflare retries the tunnel meanwhile. Later: socket handoff for no gap at all.