Jobs stay 'running' after their run ends, and count against the runner issue #71 closed: completed andymccall opened this · 2 comments
andymccall set labels gap ·
Closed by hand on Earth (backup: data/hub.db.before-stale-jobs): jobs 39, 798, 944 and 988 set to error with the reason, and their runs 9, 186, 224 and 235 to failed. earth-1 now shows 0 running. The timeout fix is still needed so this doesn't recur.
Job 1534 (metalgear run issue #63, on plex-1) was another: plex ran it at 15:32, its clone failed when Earth went down, and its finish report got HTTP 502; the runner retried for 7 s and dropped it. Closed by hand (backup hub.db.before-job-1534). Besides the hub-side timeout, the runner should keep a job's result until the hub accepts it, retrying with backoff for as long as it takes.
andymccall referenced by change 070a7b64b368 on main ·
closed by change eef677c48cd1 ·
flow_jobs 798, 944 and 988 (runs 186, 224, 235) have been 'running' on earth-1 for 1.5 to 2.8 days, though their runs show as finished. Admin → Runners and /api/v1/runners count them, so earth-1 shows 3 running while it runs nothing. (Job 39 of run 9 is the same, from before runner ids.)
The job timeout (flow.job_timeout_minutes, 60) should end a running job whose runner has stopped reporting it, mark it errored with the reason, and close its run. A runner that restarts should also tell the hub which jobs it still holds, so the hub can end the rest at once.