Plan: cancelling a running job, and concurrency:
Status: ready to execute. Written to be carried out start to finish by
one agent without further questions. Steps 1–2 (cancellation reaching the
runner) stand alone and are worth committing on their own first; steps 3–7
(concurrency:) build on them. Two commits.
Why
- Cancel doesn’t cancel a running job.
Workflows.cancel/1marks the run’s queued and running jobscancelledin the database and stops there. forgejo-runner learns of a cancellation only from the reply to its periodicUpdateTask(internal/pkg/report/reporter.go,ReportState: if the reply’sstate.resultisRESULT_CANCELLEDorRESULT_FAILURE, it cancels the job’s context). But while a job runs the runner reportsRESULT_UNSPECIFIED, andRunnerController.update_task/2maps that to"running"(map_result/1) and writes it back withWorkflows.update_job_status/3— so the job flips fromcancelledback torunning, the reply saysRESULT_UNSPECIFIED, and the runner carries on to the end. The run page says “cancelled” while the job keeps deploying. Nothing inWorkflowJob.status_changeset/3stops a terminal job from becoming non-terminal again. - Fail-fast leaves running legs running.
Workflows.maybe_fail_fast/1cancels onlyqueued/blockedsiblings (“killing a mid-flight job loses its logs for no gain”). GitHub cancels in-progress legs too; once step 1 exists, cancellation keeps the logs (the runner still uploads them). concurrency:is ignored. Nothing inlib/reads it (grepconcurrency). A deploy workflow withconcurrency: {group: deploy-prod, cancel-in-progress: false}runs two deploys at once when two pushes land close together; a PR workflow withcancel-in-progress: trueburns runners on superseded commits. Both are silent.
Semantics to match (GitHub, “Control the concurrency of workflows and jobs”)
concurrencyat the workflow top level or on a job: either a string (the group) or{group: <string>, cancel-in-progress: <bool>}. Both values may be expressions (${{ … }}).- Within a group (scoped to the repository): at most one running and
one pending run (or job). A new member arriving while one is running:
cancel-in-progress: true→ the running member and any pending member are cancelled; the new one starts;- otherwise → the new one waits as pending; an already pending member is cancelled (replaced by the newer one).
- When the running member finishes, the pending one starts.
- Workflow-level groups are decided when the run is created; job-level
groups when the job would otherwise be queued (it can use
matrixandneeds).
Where things are
| Runner’s view | lib/git_gud_web/controllers/runner_controller.ex update_task/2, map_result/1, result_atom/1 |
| Job lifecycle | lib/git_gud/workflows.ex update_job_status/3, cancel/1, maybe_fail_fast/1, maybe_complete_run/1, create_run/2, create_job_leg/3, resolve_dependents/1, do_claim_next_job/2 (claims status == "queued" only) |
| Schemas | lib/git_gud/workflows/workflow_job.ex (@valid_statuses, status_changeset/3, terminal?/1), workflow_run.ex |
| Stale job reaper | lib/git_gud/workflows/workers/stale_job_reaper.ex — make sure it doesn’t resurrect cancelled jobs |
| UI | lib/git_gud_web/live/workflow_live/show.ex, index.ex, lib/git_gud_web/live/ci_components.ex (status badges) |
| Tests | test/git_gud/workflows/rerun_cancel_test.exs, matrix_dispatch_test.exs, test/git_gud_web/controllers/runner_controller_test.exs |
Steps
1. A terminal job stays terminal; the runner is told
- In
update_task/2: if the job is already terminal (WorkflowJob.terminal?/1) and the runner reportsRESULT_UNSPECIFIED(still running), don’t update its status; reply withresult_atom(job.status)—RESULT_CANCELLEDfor a cancelled job, which makes the runner stop. Still apply step states and log rows (they are the job’s real history). - When the runner then sends its final result for a job the server
already cancelled, keep
cancelled(the server’s decision wins) but record the step states; replyRESULT_CANCELLED. - Guard at the schema too:
WorkflowJob.status_changeset/3refuses a transition from a terminal status torunning/queued/blockedunless givenforce: true;rerun_job/2(the one legitimate path back) passes it. Grep for everystatus_changeset(caller and check each. StaleJobReapermust only touchrunningjobs (check it).
2. Fail-fast cancels running legs
maybe_fail_fast/1: include running siblings. They become cancelled
in the database; step 1 makes their runners stop. Update the comment that
explains why it didn’t. Commit steps 1–2.
3. A small expression evaluator for groups
Group names are almost always built from a few context values:
${{ github.workflow }}-${{ github.ref }},
${{ github.event.pull_request.number || github.ref }},
${{ github.head_ref || github.run_id }}, ${{ inputs.env }},
${{ matrix.target }}. Add GitGud.Workflows.Expr with
interpolate(string, contexts) :: {:ok, String.t()} | :error that
replaces each ${{ … }} with the evaluated value, supporting: property
paths (a.b.c, a['b']) into contexts (string-keyed maps: github,
inputs, vars, matrix, needs), string/number/boolean/null literals,
||, &&, ==, !=, !, parentheses, and format('{0}-{1}', …).
Values stringify as GitHub does (null → "", booleans true/false).
Anything else (other functions, unknown syntax) → :error. Keep it a
recursive-descent parser in one module; no new dependency.
Contexts are built from what the run already stores: github = the same
keys the runner context gets (RunnerController.build_context_struct/4
— extract the map-building half into a public function in
GitGud.Workflows or a new GitGud.Workflows.Context module, so the
runner and the evaluator can’t disagree), inputs from the event payload
if workflow-runner-context.md has been
executed (else %{}), vars from GitGud.Secrets.resolve_for/1‘s vars,
matrix = the job’s combination, needs = the job’s needs_snapshot.
If evaluation returns :error, the run/job proceeds without
concurrency control, and the reason is logged at warning level and shown
on the run (see step 6). Never block a run on an expression we couldn’t
read.
4. Store the group
Migration (mix ecto.gen.migration add_concurrency_to_workflow_runs_and_jobs):
concurrency_group :string and cancel_in_progress :boolean, default: false on both workflow_runs and workflow_jobs, plus an index on
(repository_id, concurrency_group) for runs; for jobs index
(concurrency_group) and join through the run for the repository. Run it
on dev and test.
New job status waiting (held by concurrency, not claimable): add to
WorkflowJob.@valid_statuses, not terminal, and to the UI badge helpers.
maybe_complete_run/1 treats it as in flight.
5. Workflow-level concurrency
In create_run/2 (inside its transaction), read
workflow_doc["concurrency"]; evaluate the group with the run’s
contexts. If a group results:
- lock the group’s members (
SELECT … FOR UPDATEon runs of this repo in this group with statusqueued/running, plus runs with any jobwaiting), then:cancel-in-progress→Workflows.cancel/1each of them (after commit: do the cancels after the transaction returns, collecting the ids inside it, to avoid nestingcancel/1’s own transaction);- else → cancel any member that has not started (all its jobs
waiting), and if one is running, create this run’s initially-queued jobs aswaitinginstead ofqueued.
- When a run reaches a terminal state (
maybe_complete_run/1andcancel/1), release its group: the oldest run in the group whose jobs arewaitinggets them moved toqueued(orblockedwhen they haveneeds, matchingcreate_job_leg/3’s rule).
6. Job-level concurrency
Same rules per job, evaluated when the job would become queued (in
create_job_leg/3 for jobs without needs, and in
resolve_dependents/1 when promoting), with matrix and needs in the
contexts. The group’s members are jobs (any repository-scoped run) in
queued/running/waiting. Release in update_job_status/3 when the
job becomes terminal.
On the run page, show a waiting job as “Waiting for ” and a
cancelled-by-concurrency job with that reason: add a nullable
status_reason :string to workflow_jobs in the same migration and set
it in steps 5–6.
7. Payload and docs
Make sure RunnerPayload.encode/1 does not pass concurrency to the
runner (act ignores it today; it should never decide). README CI list:
concurrency (workflow and job level, cancel-in-progress), the supported
expression subset, and that an unreadable group means no concurrency
control.
Tests
- Runner protocol (
runner_controller_test.exs): claim a job, cancel its run, thenUpdateTaskwithRESULT_UNSPECIFIED→ replyRESULT_CANCELLED, job stillcancelled; a finalRESULT_SUCCESSafter cancel → job stayscancelled. rerun_job/2still revives a cancelled job.- Fail-fast: a running sibling leg is cancelled.
Exprunit tests: each pattern listed in step 3;||fallbacks with null and empty string (GitHub: empty string is falsy); unknown function →:error.- Workflow concurrency (on-disk dispatch,
path_dispatch_test.exsstyle, withconcurrency: {group: deploy, cancel-in-progress: false}): first push → jobs queued; second → jobs waiting; third → second run cancelled, third waiting; completing the first (driveupdate_job_status/3) → third queued. cancel-in-progress: true: second push cancels the first run (its running job iscancelled, and its runner would be told — assert viaupdate_taskas above).- Group
${{ github.workflow }}-${{ github.ref }}: pushes to two different branches don’t interact. - Job-level group with
${{ matrix.target }}: legs with the same target serialize, different targets don’t. - An unparseable group (
${{ toJSON(github) }}) → runs normally, warning logged.
Done when
mix testpasses; no new compile warnings (comparemix compile --forcewarning lists before/after);mix formatonly touched files.- Two commits in the repository’s message style: cancellation, then concurrency.
- Not pushed.