Leases and drain: a runner that leaves hands its job back issue #76
open andymccall opened this · 0 comments
andymccall
A job held by a runner is a lease, renewed by the runner's log and heartbeat; when it lapses the job is requeued (as today, after a grace). New: drain, one message in the runner contract for every planned departure: SIGTERM, a node drain, a spot reclaim notice, a phone taken off charge or off the allowed network. The runner then finishes the job when the rest is shorter than the time it has, else saves its checkpoints (issue #77) and releases the lease with drained; the job is requeued at once, ahead of new work. Design: docs/handover.md §6.
A job held by a runner is a lease, renewed by the runner's log and heartbeat; when it lapses the job is requeued (as today, after a grace). New: drain, one message in the runner contract for every planned departure: SIGTERM, a node drain, a spot reclaim notice, a phone taken off charge or off the allowed network. The runner then finishes the job when the rest is shorter than the time it has, else saves its checkpoints (issue #77) and releases the lease with
drained; the job is requeued at once, ahead of new work. Design: docs/handover.md §6.