10.4 KiB · text History 95a46eb

Plan: cancelling a running job, and concurrency:

Status: ready to execute. Written to be carried out start to finish by one agent without further questions. Steps 1–2 (cancellation reaching the runner) stand alone and are worth committing on their own first; steps 3–7 (concurrency:) build on them. Two commits.

Why

  1. Cancel doesn’t cancel a running job. Workflows.cancel/1 marks the run’s queued and running jobs cancelled in the database and stops there. forgejo-runner learns of a cancellation only from the reply to its periodic UpdateTask (internal/pkg/report/reporter.go, ReportState: if the reply’s state.result is RESULT_CANCELLED or RESULT_FAILURE, it cancels the job’s context). But while a job runs the runner reports RESULT_UNSPECIFIED, and RunnerController.update_task/2 maps that to "running" (map_result/1) and writes it back with Workflows.update_job_status/3 — so the job flips from cancelled back to running, the reply says RESULT_UNSPECIFIED, and the runner carries on to the end. The run page says “cancelled” while the job keeps deploying. Nothing in WorkflowJob.status_changeset/3 stops a terminal job from becoming non-terminal again.
  2. Fail-fast leaves running legs running. Workflows.maybe_fail_fast/1 cancels only queued/blocked siblings (“killing a mid-flight job loses its logs for no gain”). GitHub cancels in-progress legs too; once step 1 exists, cancellation keeps the logs (the runner still uploads them).
  3. concurrency: is ignored. Nothing in lib/ reads it (grep concurrency). A deploy workflow with concurrency: {group: deploy-prod, cancel-in-progress: false} runs two deploys at once when two pushes land close together; a PR workflow with cancel-in-progress: true burns runners on superseded commits. Both are silent.

Semantics to match (GitHub, “Control the concurrency of workflows and jobs”)

  • concurrency at the workflow top level or on a job: either a string (the group) or {group: <string>, cancel-in-progress: <bool>}. Both values may be expressions (${{ … }}).
  • Within a group (scoped to the repository): at most one running and one pending run (or job). A new member arriving while one is running:
    • cancel-in-progress: true → the running member and any pending member are cancelled; the new one starts;
    • otherwise → the new one waits as pending; an already pending member is cancelled (replaced by the newer one).
  • When the running member finishes, the pending one starts.
  • Workflow-level groups are decided when the run is created; job-level groups when the job would otherwise be queued (it can use matrix and needs).

Where things are

Runner’s view lib/git_gud_web/controllers/runner_controller.ex update_task/2, map_result/1, result_atom/1
Job lifecycle lib/git_gud/workflows.ex update_job_status/3, cancel/1, maybe_fail_fast/1, maybe_complete_run/1, create_run/2, create_job_leg/3, resolve_dependents/1, do_claim_next_job/2 (claims status == "queued" only)
Schemas lib/git_gud/workflows/workflow_job.ex (@valid_statuses, status_changeset/3, terminal?/1), workflow_run.ex
Stale job reaper lib/git_gud/workflows/workers/stale_job_reaper.ex — make sure it doesn’t resurrect cancelled jobs
UI lib/git_gud_web/live/workflow_live/show.ex, index.ex, lib/git_gud_web/live/ci_components.ex (status badges)
Tests test/git_gud/workflows/rerun_cancel_test.exs, matrix_dispatch_test.exs, test/git_gud_web/controllers/runner_controller_test.exs

Steps

1. A terminal job stays terminal; the runner is told

  • In update_task/2: if the job is already terminal (WorkflowJob.terminal?/1) and the runner reports RESULT_UNSPECIFIED (still running), don’t update its status; reply with result_atom(job.status) — RESULT_CANCELLED for a cancelled job, which makes the runner stop. Still apply step states and log rows (they are the job’s real history).
  • When the runner then sends its final result for a job the server already cancelled, keep cancelled (the server’s decision wins) but record the step states; reply RESULT_CANCELLED.
  • Guard at the schema too: WorkflowJob.status_changeset/3 refuses a transition from a terminal status to running/queued/blocked unless given force: true; rerun_job/2 (the one legitimate path back) passes it. Grep for every status_changeset( caller and check each.
  • StaleJobReaper must only touch running jobs (check it).

2. Fail-fast cancels running legs

maybe_fail_fast/1: include running siblings. They become cancelled in the database; step 1 makes their runners stop. Update the comment that explains why it didn’t. Commit steps 1–2.

3. A small expression evaluator for groups

Group names are almost always built from a few context values: ${{ github.workflow }}-${{ github.ref }}, ${{ github.event.pull_request.number || github.ref }}, ${{ github.head_ref || github.run_id }}, ${{ inputs.env }}, ${{ matrix.target }}. Add GitGud.Workflows.Expr with interpolate(string, contexts) :: {:ok, String.t()} | :error that replaces each ${{ … }} with the evaluated value, supporting: property paths (a.b.c, a['b']) into contexts (string-keyed maps: github, inputs, vars, matrix, needs), string/number/boolean/null literals, ||, &&, ==, !=, !, parentheses, and format('{0}-{1}', …). Values stringify as GitHub does (null → "", booleans true/false). Anything else (other functions, unknown syntax) → :error. Keep it a recursive-descent parser in one module; no new dependency.

Contexts are built from what the run already stores: github = the same keys the runner context gets (RunnerController.build_context_struct/4 — extract the map-building half into a public function in GitGud.Workflows or a new GitGud.Workflows.Context module, so the runner and the evaluator can’t disagree), inputs from the event payload if workflow-runner-context.md has been executed (else %{}), vars from GitGud.Secrets.resolve_for/1‘s vars, matrix = the job’s combination, needs = the job’s needs_snapshot.

If evaluation returns :error, the run/job proceeds without concurrency control, and the reason is logged at warning level and shown on the run (see step 6). Never block a run on an expression we couldn’t read.

4. Store the group

Migration (mix ecto.gen.migration add_concurrency_to_workflow_runs_and_jobs): concurrency_group :string and cancel_in_progress :boolean, default: false on both workflow_runs and workflow_jobs, plus an index on (repository_id, concurrency_group) for runs; for jobs index (concurrency_group) and join through the run for the repository. Run it on dev and test.

New job status waiting (held by concurrency, not claimable): add to WorkflowJob.@valid_statuses, not terminal, and to the UI badge helpers. maybe_complete_run/1 treats it as in flight.

5. Workflow-level concurrency

In create_run/2 (inside its transaction), read workflow_doc["concurrency"]; evaluate the group with the run’s contexts. If a group results:

  • lock the group’s members (SELECT … FOR UPDATE on runs of this repo in this group with status queued/running, plus runs with any job waiting), then:
    • cancel-in-progress → Workflows.cancel/1 each of them (after commit: do the cancels after the transaction returns, collecting the ids inside it, to avoid nesting cancel/1’s own transaction);
    • else → cancel any member that has not started (all its jobs waiting), and if one is running, create this run’s initially-queued jobs as waiting instead of queued.
  • When a run reaches a terminal state (maybe_complete_run/1 and cancel/1), release its group: the oldest run in the group whose jobs are waiting gets them moved to queued (or blocked when they have needs, matching create_job_leg/3’s rule).

6. Job-level concurrency

Same rules per job, evaluated when the job would become queued (in create_job_leg/3 for jobs without needs, and in resolve_dependents/1 when promoting), with matrix and needs in the contexts. The group’s members are jobs (any repository-scoped run) in queued/running/waiting. Release in update_job_status/3 when the job becomes terminal.

On the run page, show a waiting job as “Waiting for ” and a cancelled-by-concurrency job with that reason: add a nullable status_reason :string to workflow_jobs in the same migration and set it in steps 5–6.

7. Payload and docs

Make sure RunnerPayload.encode/1 does not pass concurrency to the runner (act ignores it today; it should never decide). README CI list: concurrency (workflow and job level, cancel-in-progress), the supported expression subset, and that an unreadable group means no concurrency control.

Tests

  • Runner protocol (runner_controller_test.exs): claim a job, cancel its run, then UpdateTask with RESULT_UNSPECIFIED → reply RESULT_CANCELLED, job still cancelled; a final RESULT_SUCCESS after cancel → job stays cancelled.
  • rerun_job/2 still revives a cancelled job.
  • Fail-fast: a running sibling leg is cancelled.
  • Expr unit tests: each pattern listed in step 3; || fallbacks with null and empty string (GitHub: empty string is falsy); unknown function → :error.
  • Workflow concurrency (on-disk dispatch, path_dispatch_test.exs style, with concurrency: {group: deploy, cancel-in-progress: false}): first push → jobs queued; second → jobs waiting; third → second run cancelled, third waiting; completing the first (drive update_job_status/3) → third queued.
  • cancel-in-progress: true: second push cancels the first run (its running job is cancelled, and its runner would be told — assert via update_task as above).
  • Group ${{ github.workflow }}-${{ github.ref }}: pushes to two different branches don’t interact.
  • Job-level group with ${{ matrix.target }}: legs with the same target serialize, different targets don’t.
  • An unparseable group (${{ toJSON(github) }}) → runs normally, warning logged.

Done when

  • mix test passes; no new compile warnings (compare mix compile --force warning lists before/after); mix format only touched files.
  • Two commits in the repository’s message style: cancellation, then concurrency.
  • Not pushed.