Runners ride out a hub restart issue #49
closed: completed andymccall opened this · 0 comments
andymccall
A job's log batches, report and output uploads are retried only 1+2+4 s and only on 5xx; a refused or dropped connection (the hub restarting, cloudflared answering 502/530) is not retried, so a hub deploy mid-job loses log lines and can fail a job's uploads. Retry transport errors and 5xx with backoff for about 3 minutes, and keep unsent log lines for the next batch. Then hub deploys need not wait for idle runners.
A job's log batches, report and output uploads are retried only 1+2+4 s and only on 5xx; a refused or dropped connection (the hub restarting, cloudflared answering 502/530) is not retried, so a hub deploy mid-job loses log lines and can fail a job's uploads. Retry transport errors and 5xx with backoff for about 3 minutes, and keep unsent log lines for the next batch. Then hub deploys need not wait for idle runners.