Runner: one job's slow fetch holds every job on the runner issue #63
closed: completed andymccall opened this · 0 comments
andymccall
A runner prepares workspaces under one global lock (PREPARE), held from the first fetch to the workspace image. On 2026-10-16 the hub's fmt job logged 819 s between 'fetching andymccall/oxygenhub' and its next line, waiting behind metalgear2 jobs whose repository lives in a GCS bucket (oxygenhub#234), and a job token expired before one fetch finished (HTTP 401, job 5121). Fetches and exports should be locked per repository; only the cache pool and imaging need the runner-wide lock, and only a first clone (its token in the process environment) needs a clone lock. Each job's log should say how long it waited and how long fetching and exporting took.
A runner prepares workspaces under one global lock (PREPARE), held from the first fetch to the workspace image. On 2026-10-16 the hub's fmt job logged 819 s between 'fetching andymccall/oxygenhub' and its next line, waiting behind metalgear2 jobs whose repository lives in a GCS bucket (oxygenhub#234), and a job token expired before one fetch finished (HTTP 401, job 5121). Fetches and exports should be locked per repository; only the cache pool and imaging need the runner-wide lock, and only a first clone (its token in the process environment) needs a clone lock. Each job's log should say how long it waited and how long fetching and exporting took.