GITGUD

neiam neiam /gitgud

Git Gud

public · Issues · Pulls · Labels · Forks · Compare · Actions success · Badges · Packages

⭐ 1

Log in to mark this repository.

Matrix builds, and runs-on as a real label set

cac71de · Gabriel Morell · 2026-09-11 22:14

18 files +1319 -73
Message 
{commit_body(@commit)}

Files changed

modified README.md
+37 −0
@@ -81,6 +81,7 @@ reinvent them.
81 81 - **CI**
82 82 - Speaks the **Forgejo Actions runner protocol** (Twirp) — point any modern `forgejo-runner` at it; protocol path autodetected (`ping.v1`, `runner.v1`, with/without `_apis/`)
83 83 - Repo-scoped + org-scoped runner registration with one-shot tokens minted from the settings UI
84 + - **Matrix builds**, with `runs-on` as a real label set — `strategy.matrix` expands into one job per combination, each claimed independently, so `runs-on: [self-hosted, "${{ matrix.arch }}"]` sends the arm64 leg to an arm64 machine and the amd64 leg to an amd64 one. `include`/`exclude` follow GitHub's semantics, `fail-fast` defaults on, and `needs:` names the group rather than a leg
84 85 - **Five deployment styles** offered in the registration UI: Native binary, Docker, Podman, Podman Quadlet, Kubernetes (Deployment + dind sidecar) — copy-pasteable `register` + `daemon` snippets per style
85 86 - Re-run / cancel buttons on the workflow show page
86 87 - Per-task HS256 JWT minted on `FetchTask`, gates artifact + cache endpoints
@@ -309,6 +310,42 @@ register → daemon --config runner-config.yml` flow per the
309 310 The runner is **external**. A native runner is on the roadmap but the
310 311 external one works today and is well-tested by the Forgejo community.
311 312
313 +### Matrix builds across architectures
314 +
315 +Label each runner with what it *is*, in the registration UI's labels
316 +field — `self-hosted,arm64,linux` on the Pi, `self-hosted,amd64,linux`
317 +on the x86 box — then let the matrix pick:
318 +
319 +```yaml
320 +jobs:
321 + build:
322 + strategy:
323 + matrix:
324 + arch: [amd64, arm64]
325 + runs-on: [self-hosted, "${{ matrix.arch }}"]
326 + steps:
327 + - uses: actions/checkout@v4
328 + - run: make build ARCH=${{ matrix.arch }}
329 +```
330 +
331 +That is two jobs, claimed by two runners, running at the same time. A
332 +job is claimable when the runner carries **every** label its `runs-on`
333 +lists — GitHub's rule — so extra labels on a runner never disqualify
334 +it and a missing one always does. `runs-on: self-hosted` matches any
335 +self-registered runner, whatever else it carries.
336 +
337 +`runs-on` is the one expression the forge resolves itself, because it
338 +decides *which* runner and there is nobody else to ask yet. Everything
339 +else — `${{ matrix.arch }}` inside a step — is evaluated by the runner
340 +against a matrix context pinned to that leg's single value.
341 +
342 +`needs: build` waits for **every** leg, and a downstream job sees one
343 +`needs.build` however many machines produced it. `fail-fast` is on by
344 +default: the first leg to fail cancels its siblings that haven't
345 +started, while one already running is left to finish rather than
346 +killed. `max-parallel` is not implemented — every leg is claimable as
347 +soon as it exists.
348 +
312 349 ### OCI registry
313 350
314 351 When you want pushes from your workflows (`docker/build-push-action`,
modified ROADMAP.md
+2 −1
@@ -43,7 +43,8 @@ queue in slice 42). Remaining federation items are discretionary
43 43 | RFC 9421 (HTTP Messages) signature | When peers upgrade from draft-cavage-12 | small (mostly mechanical) |
44 44 | Reputation aging for new federated hosts | Strictest policy until N days clean; auto-promote | small-medium |
45 45 | `act_runner` (older Forgejo runner) explicit support | Protocol is identical; verify against a specific binary version | small |
46 −| Per-job concurrency / matrix builds | YAML parser supports it; runner orchestration doesn't yet | medium |
46 +| `strategy.max-parallel` | Matrix builds shipped in slice 72; every leg is claimable the moment it exists, so a wide matrix can saturate a small runner pool. Needs a claim-time count of in-flight legs per `(run, job_id)` | small-medium |
47 +| Per-job concurrency (`concurrency:` groups) | Cancel-in-progress and named concurrency groups across runs; unrelated to the matrix work despite sharing a roadmap line previously | medium |
47 48 | Code search (Meilisearch or similar) | Postgres FTS covers issues/PRs; code search is its own beast | large |
48 49
49 50 ## Explicitly declined
modified lib/git_gud/workflows.ex
+187 −67
@@ -17,6 +17,7 @@ defmodule GitGud.Workflows do
17 17 alias GitGud.Repo
18 18 alias GitGud.Repositories
19 19 alias GitGud.Repositories.Repository
20 + alias GitGud.Workflows.Matrix
20 21 alias GitGud.Workflows.Runner
21 22 alias GitGud.Workflows.WorkflowJob
22 23 alias GitGud.Workflows.WorkflowRun
@@ -141,6 +142,17 @@ defmodule GitGud.Workflows do
141 142 end)
142 143 end
143 144
145 + # `build (arm64, 28)` — the convention Actions uses, so two legs are
146 + # distinguishable at a glance in the UI and in a commit status.
147 + defp leg_name(job, combination) do
148 + base = job.name || job.id
149 +
150 + case Matrix.suffix(combination) do
151 + nil -> base
152 + suffix -> "#{base} (#{suffix})"
153 + end
154 + end
155 +
144 156 defp next_run_number(rid) do
145 157 (from(r in WorkflowRun,
146 158 where: r.repository_id == ^rid,
@@ -149,20 +161,38 @@ defmodule GitGud.Workflows do
149 161 |> Repo.one() || 0) + 1
150 162 end
151 163
152 − defp create_job(%WorkflowRun{id: rid}, %{} = job) do
164 + # One row per `strategy.matrix` combination — that is the whole of
165 + # matrix support on this side. Separate rows are what let two legs be
166 + # claimed by two runners, which is the point when the axis is the
167 + # machine's architecture.
168 + defp create_job(%WorkflowRun{} = run, %{} = job) do
169 + for combination <- Matrix.expand(job[:strategy] || job.raw["strategy"]) do
170 + create_job_leg(run, job, combination)
171 + end
172 + end
173 +
174 + defp create_job_leg(%WorkflowRun{id: rid}, %{} = job, combination) do
153 175 # Jobs with unresolved `needs:` start in `blocked` so claim_next_job
154 176 # can never pick them up before their prerequisites complete.
155 177 # Promotion to `queued` happens in `resolve_dependents/1`.
156 178 initial_status = if job.needs == [], do: "queued", else: "blocked"
157 179
180 + # `runs-on` is resolved here rather than by the runner, because it
181 + # decides *which* runner — there is nobody else to ask yet.
182 + labels =
183 + job.runs_on
184 + |> Matrix.substitute(combination)
185 + |> Matrix.labels()
186 +
158 187 job_row =
159 188 %WorkflowJob{workflow_run_id: rid}
160 189 |> WorkflowJob.create_changeset(%{
161 190 job_id: job.id,
162 − name: job.name,
163 − runs_on: %{"value" => job.runs_on},
191 + name: leg_name(job, combination),
192 + runs_on: %{"labels" => labels},
164 193 needs: job.needs,
165 − definition: job.raw
194 + matrix: if(combination == %{}, do: nil, else: combination),
195 + definition: Matrix.pin_definition(job.raw, combination)
166 196 })
167 197 |> Ecto.Changeset.change(status: initial_status)
168 198 |> Repo.insert!()
@@ -412,15 +442,17 @@ defmodule GitGud.Workflows do
412 442 def claim_job_by_handle(runner, _), do: claim_next_job(runner)
413 443
414 444 defp do_claim_specific_job(%Runner{id: rid} = runner, job_id) do
415 − labels = effective_labels(runner)
416 − match_labels = if "self-hosted" in labels, do: labels, else: ["self-hosted" | labels]
445 + match_labels = runner |> effective_labels() |> match_labels()
417 446
418 447 Repo.transaction(fn ->
419 448 job =
420 449 scoped_job_query(runner)
421 450 |> where([j], j.id == ^job_id)
422 451 |> where([j], j.status == "queued")
423 − |> where([j], fragment("(?->>'value') = ANY(?)", j.runs_on, ^match_labels))
452 + |> where(
453 + [j],
454 + fragment("(? -> 'labels') <@ to_jsonb(?::text[])", j.runs_on, ^match_labels)
455 + )
424 456 |> lock("FOR UPDATE SKIP LOCKED")
425 457 |> Repo.one()
426 458
@@ -447,20 +479,16 @@ defmodule GitGud.Workflows do
447 479 end
448 480
449 481 defp do_claim_next_job(%Runner{id: rid} = runner, labels) do
450 − # GitHub-Actions semantics: `runs-on: self-hosted` matches any
451 − # self-registered runner regardless of OS label. So if the runner
452 − # was minted via `Runners.create_offline/2` we implicitly add
453 − # `self-hosted` to the match set for the query. Operators who want
454 − # strict label matching can still register a runner without the
455 − # `self-hosted` label — but everything created via this codebase
456 − # is by definition self-hosted, so the implicit wildcard is safe.
457 − match_labels = if "self-hosted" in labels, do: labels, else: ["self-hosted" | labels]
482 + match_labels = match_labels(labels)
458 483
459 484 Repo.transaction(fn ->
460 485 job =
461 486 scoped_job_query(runner)
462 487 |> where([j], j.status == "queued")
463 − |> where([j], fragment("(?->>'value') = ANY(?)", j.runs_on, ^match_labels))
488 + |> where(
489 + [j],
490 + fragment("(? -> 'labels') <@ to_jsonb(?::text[])", j.runs_on, ^match_labels)
491 + )
464 492 |> order_by([j], asc: j.id)
465 493 |> limit(1)
466 494 |> lock("FOR UPDATE SKIP LOCKED")
@@ -489,6 +517,26 @@ defmodule GitGud.Workflows do
489 517 end
490 518 end
491 519
520 + # The label set a runner can satisfy, as JSON for the containment
521 + # test. A job is claimable when *every* label its `runs-on` lists is
522 + # one this runner carries — GitHub's rule, and the reason
523 + # `runs-on: [self-hosted, arm64]` reaches an arm64 machine and only
524 + # an arm64 machine.
525 + #
526 + # `self-hosted` is added implicitly: `runs-on: self-hosted` is meant
527 + # to match any self-registered runner regardless of its other
528 + # labels, and everything this codebase mints is by definition
529 + # self-hosted.
530 + #
531 + # Built into jsonb on the SQL side from a text array rather than
532 + # interpolated as a JSON string: a string parameter bound to a
533 + # `::jsonb` placeholder is encoded as a JSON *string*, so containment
534 + # compares an array against `"[\"a\",\"b\"]"` and quietly matches
535 + # nothing. `to_jsonb(?::text[])` has one unambiguous reading.
536 + defp match_labels(labels) do
537 + if "self-hosted" in labels, do: labels, else: ["self-hosted" | labels]
538 + end
539 +
492 540 # A runner only claims jobs in its scope:
493 541 # * repo-scoped runner → jobs in that repo
494 542 # * org-scoped runner → jobs in any repo owned by that org
@@ -510,7 +558,7 @@ defmodule GitGud.Workflows do
510 558 end
511 559
512 560 defp scoped_job_query(_global) do
513 − from j in WorkflowJob
561 + from(j in WorkflowJob)
514 562 end
515 563
516 564 @doc """
@@ -525,7 +573,8 @@ defmodule GitGud.Workflows do
525 573
526 574 `rows` is the list of `%LogRow{}` protobufs from the request.
527 575 """
528 − def append_log(%WorkflowStep{id: step_id}, start_seq, rows) when is_integer(start_seq) and is_list(rows) do
576 + def append_log(%WorkflowStep{id: step_id}, start_seq, rows)
577 + when is_integer(start_seq) and is_list(rows) do
529 578 now = DateTime.utc_now(:second)
530 579
531 580 entries =
@@ -847,6 +896,7 @@ defmodule GitGud.Workflows do
847 896
848 897 if WorkflowJob.terminal?(updated.status) do
849 898 finalize_steps(updated)
899 + maybe_fail_fast(updated)
850 900 resolve_dependents(updated)
851 901 end
852 902
@@ -889,6 +939,31 @@ defmodule GitGud.Workflows do
889 939 end)
890 940 end
891 941
942 + # `strategy.fail-fast` — GitHub's default is to abandon the rest of a
943 + # matrix as soon as one leg fails, because the usual reason to run
944 + # sixteen legs is to find out whether they all pass, and once one
945 + # hasn't you have your answer. Only siblings that haven't started are
946 + # cancelled; a leg already running is left to finish, since killing a
947 + # mid-flight job loses its logs for no gain.
948 + defp maybe_fail_fast(%WorkflowJob{status: status} = job)
949 + when status in ~w(failure cancelled) do
950 + if job.matrix && Matrix.fail_fast?(strategy_of(job)) do
951 + from(j in WorkflowJob,
952 + where:
953 + j.workflow_run_id == ^job.workflow_run_id and j.job_id == ^job.job_id and
954 + j.id != ^job.id and j.status in ["queued", "blocked"]
955 + )
956 + |> Repo.all()
957 + |> Enum.each(&update_job_status(&1, "cancelled"))
958 + end
959 + end
960 +
961 + defp maybe_fail_fast(_job), do: :ok
962 +
963 + # `Matrix.pin_definition/2` carries `fail-fast` through onto the
964 + # pinned strategy precisely so it can be read back here.
965 + defp strategy_of(%WorkflowJob{definition: definition}), do: definition["strategy"]
966 +
892 967 @doc """
893 968 After a job reaches a terminal state, walk every sibling job in the
894 969 same run whose `needs:` array contains this job's `job_id` and
@@ -907,31 +982,48 @@ defmodule GitGud.Workflows do
907 982 from(j in WorkflowJob, where: j.workflow_run_id == ^rid)
908 983 |> Repo.all()
909 984
910 − by_jid = Map.new(siblings, fn j -> {j.job_id, j} end)
985 + # Grouped, not indexed: a matrix expands into several rows sharing
986 + # one `job_id`, and `needs: build` means "all of build", not
987 + # "whichever leg this lookup happened to find".
988 + by_jid = Enum.group_by(siblings, & &1.job_id)
911 989
912 − dependents = Enum.filter(siblings, fn j -> completed_jid in j.needs and j.status == "blocked" end)
990 + dependents =
991 + Enum.filter(siblings, fn j -> completed_jid in j.needs and j.status == "blocked" end)
913 992
914 993 Enum.each(dependents, fn dep ->
915 994 needs_states = Enum.map(dep.needs, fn jid -> Map.get(by_jid, jid) end)
916 995
917 996 cond do
918 − Enum.any?(needs_states, &is_nil/1) ->
997 + Enum.any?(needs_states, &(&1 in [nil, []])) ->
919 998 # A referenced need doesn't exist as a sibling — workflow
920 999 # author error. Cancel rather than block forever.
921 1000 {:ok, _} = update_job_status(dep, "cancelled")
922 1001
923 − Enum.any?(needs_states, fn n -> n.status in ~w(failure cancelled) end) ->
1002 + Enum.any?(needs_states, fn legs ->
1003 + Enum.any?(legs, &(&1.status in ~w(failure cancelled)))
1004 + end) ->
924 1005 {:ok, _} = update_job_status(dep, "cancelled")
925 1006
926 − Enum.all?(needs_states, fn n -> WorkflowJob.satisfies_dependency?(n.status) end) ->
1007 + Enum.all?(needs_states, fn legs ->
1008 + Enum.all?(legs, &WorkflowJob.satisfies_dependency?(&1.status))
1009 + end) ->
927 1010 # Snapshot upstream results/outputs now, while they're terminal.
928 1011 # build_needs/1 reads this instead of re-reading upstream status
929 1012 # at fetch time (which raced and could yield RESULT_UNSPECIFIED,
930 1013 # making act skip the job on `if: success()`).
1014 + #
1015 + # A matrix's legs collapse to one entry, because `needs.build`
1016 + # is one name downstream however many machines produced it:
1017 + # outputs merge, and the status is the worst of them.
931 1018 snapshot =
932 1019 Map.new(dep.needs, fn jid ->
933 − n = Map.get(by_jid, jid)
934 − {jid, %{"status" => n.status, "outputs" => n.outputs || %{}}}
1020 + legs = Map.get(by_jid, jid)
1021 +
1022 + {jid,
1023 + %{
1024 + "status" => aggregate_status(legs),
1025 + "outputs" => Enum.reduce(legs, %{}, &Map.merge(&2, &1.outputs || %{}))
1026 + }}
935 1027 end)
936 1028
937 1029 {:ok, _} =
@@ -950,6 +1042,17 @@ defmodule GitGud.Workflows do
950 1042 :ok
951 1043 end
952 1044
1045 + # `success` only when every leg succeeded; `skipped` when they all
1046 + # skipped. Anything mixed reports `success`, because reaching here
1047 + # means no leg failed and a downstream `if: success()` should run.
1048 + defp aggregate_status(legs) do
1049 + cond do
1050 + Enum.all?(legs, &(&1.status == "skipped")) -> "skipped"
1051 + Enum.all?(legs, &(&1.status == "success")) -> "success"
1052 + true -> "success"
1053 + end
1054 + end
1055 +
953 1056 defp maybe_complete_run(%WorkflowJob{workflow_run_id: rid}) do
954 1057 statuses =
955 1058 from(j in WorkflowJob, where: j.workflow_run_id == ^rid, select: j.status)
@@ -1059,7 +1162,8 @@ defmodule GitGud.Workflows do
1059 1162
1060 1163 target_url =
1061 1164 GitGudWeb.Endpoint.url() <>
1062 − "/r/repo/" <> Integer.to_string(run.repository_id) <>
1165 + "/r/repo/" <>
1166 + Integer.to_string(run.repository_id) <>
1063 1167 "/actions/runs/" <> Integer.to_string(run.id)
1064 1168
1065 1169 description =
@@ -1083,8 +1187,7 @@ defmodule GitGud.Workflows do
1083 1187 %GitGud.Workflows.CommitStatus{}
1084 1188 |> GitGud.Workflows.CommitStatus.changeset(attrs)
1085 1189 |> Repo.insert(
1086 − on_conflict:
1087 − {:replace, [:state, :description, :target_url, :updated_at]},
1190 + on_conflict: {:replace, [:state, :description, :target_url, :updated_at]},
1088 1191 conflict_target: [:repository_id, :sha, :context]
1089 1192 )
1090 1193
@@ -1144,50 +1247,67 @@ defmodule GitGud.Workflows do
1144 1247 def rerun_job(%WorkflowRun{} = run, job_id) when is_binary(job_id) do
1145 1248 siblings = Repo.all(from j in WorkflowJob, where: j.workflow_run_id == ^run.id)
1146 1249
1147 − case Enum.find(siblings, &(&1.job_id == job_id)) do
1148 − nil ->
1250 + # Every leg, not the first one found: a matrix's legs share a
1251 + # `job_id`, and "re-run build" means the build, not whichever
1252 + # architecture happened to sort first.
1253 + case Enum.filter(siblings, &(&1.job_id == job_id)) do
1254 + [] ->
1149 1255 {:error, :not_found}
1150 1256
1151 − job ->
1152 − by_jid = Map.new(siblings, &{&1.job_id, &1})
1257 + legs ->
1258 + by_jid = Enum.group_by(siblings, & &1.job_id)
1153 1259
1154 1260 needs_ok? =
1155 − job.needs == [] or
1156 − Enum.all?(job.needs, fn jid ->
1157 − case Map.get(by_jid, jid) do
1158 − %WorkflowJob{status: s} -> WorkflowJob.satisfies_dependency?(s)
1159 − _ -> false
1160 − end
1161 − end)
1261 + fn job ->
1262 + job.needs == [] or
1263 + Enum.all?(job.needs, fn jid ->
1264 + case Map.get(by_jid, jid) do
1265 + [_ | _] = upstream ->
1266 + Enum.all?(upstream, &WorkflowJob.satisfies_dependency?(&1.status))
1267 +
1268 + _ ->
1269 + false
1270 + end
1271 + end)
1272 + end
1162 1273
1163 1274 Repo.transaction(fn ->
1164 − # clean slate for this job's logs + steps. Logs are keyed by
1165 − # workflow_step_id, so scope by this job's step ids.
1166 − step_ids_q = from(s in WorkflowStep, where: s.workflow_job_id == ^job.id, select: s.id)
1167 − Repo.delete_all(from l in "workflow_step_logs", where: l.workflow_step_id in subquery(step_ids_q))
1168 −
1169 − Repo.update_all(
1170 − from(s in WorkflowStep, where: s.workflow_job_id == ^job.id),
1171 − set: [
1172 − status: "pending",
1173 − conclusion: nil,
1174 − started_at: nil,
1175 − completed_at: nil,
1176 − log_index: nil,
1177 − log_length: nil
1178 − ]
1179 − )
1180 −
1181 − {:ok, updated} =
1182 − job
1183 − |> Ecto.Changeset.change(
1184 − status: if(needs_ok?, do: "queued", else: "blocked"),
1185 − conclusion: nil,
1186 − started_at: nil,
1187 − completed_at: nil,
1188 − runner_id: nil
1189 − )
1190 − |> Repo.update()
1275 + updated =
1276 + Enum.map(legs, fn job ->
1277 + # clean slate for this job's logs + steps. Logs are keyed by
1278 + # workflow_step_id, so scope by this job's step ids.
1279 + step_ids_q =
1280 + from(s in WorkflowStep, where: s.workflow_job_id == ^job.id, select: s.id)
1281 +
1282 + Repo.delete_all(
1283 + from l in "workflow_step_logs", where: l.workflow_step_id in subquery(step_ids_q)
1284 + )
1285 +
1286 + Repo.update_all(
1287 + from(s in WorkflowStep, where: s.workflow_job_id == ^job.id),
1288 + set: [
1289 + status: "pending",
1290 + conclusion: nil,
1291 + started_at: nil,
1292 + completed_at: nil,
1293 + log_index: nil,
1294 + log_length: nil
1295 + ]
1296 + )
1297 +
1298 + {:ok, row} =
1299 + job
1300 + |> Ecto.Changeset.change(
1301 + status: if(needs_ok?.(job), do: "queued", else: "blocked"),
1302 + conclusion: nil,
1303 + started_at: nil,
1304 + completed_at: nil,
1305 + runner_id: nil
1306 + )
1307 + |> Repo.update()
1308 +
1309 + row
1310 + end)
1191 1311
1192 1312 # Flip the run non-terminal so it isn't "complete" and the
1193 1313 # runner polls it. maybe_complete_run re-settles it on finish.
@@ -1197,7 +1317,7 @@ defmodule GitGud.Workflows do
1197 1317 end)
1198 1318 |> case do
1199 1319 {:ok, updated} ->
1200 − broadcast({:job_status, run.id, updated})
1320 + Enum.each(updated, &broadcast({:job_status, run.id, &1}))
1201 1321 broadcast_ci_change(run.id)
1202 1322 {:ok, Repo.get!(WorkflowRun, run.id)}
1203 1323
added lib/git_gud/workflows/matrix.ex
+234 −0
@@ -0,0 +1,234 @@
1 +defmodule GitGud.Workflows.Matrix do
2 + @moduledoc """
3 + Expands a job's `strategy.matrix` into the combinations it stands for.
4 +
5 + strategy:
6 + matrix:
7 + arch: [amd64, arm64]
8 + otp: ["27", "28"]
9 +
10 + …is four jobs. Each becomes its own `WorkflowJob` row, with its own
11 + status, logs and runner — which is the point: two of those want an
12 + amd64 machine and two want arm64, and only separate rows can be
13 + claimed by separate runners.
14 +
15 + ## `include` and `exclude`
16 +
17 + Both are ordinary GitHub Actions semantics and both are fiddly, so
18 + they are spelled out here rather than left to the reader:
19 +
20 + * `exclude` drops every combination that matches *all* of the
21 + key/values it names. It is applied before `include`, so an
22 + `include` can add a combination back.
23 +
24 + * `include` is not "append these combinations". Each entry is
25 + merged into every existing combination whose overlapping keys
26 + already agree with it — so `{otp: "28", coverage: true}` adds
27 + `coverage` to the two OTP-28 jobs rather than creating a fifth.
28 + An entry that matches nothing is appended as a combination of its
29 + own, which is how you add a single odd platform to a matrix.
30 +
31 + An entry that *would* overwrite a value the base matrix set is where
32 + GitHub's own rules get genuinely ambiguous; here, overlapping keys
33 + must agree for a merge to happen, so such an entry appends instead.
34 +
35 + ## Ordering
36 +
37 + Combinations come back in the order GitHub produces them: the *last*
38 + matrix key varies fastest. That is only cosmetic, but it decides the
39 + order jobs appear in the UI, and matching the tool people already
40 + know beats an arbitrary choice.
41 + """
42 +
43 + @reserved ~w(include exclude)
44 +
45 + @typedoc "One combination: matrix key to the value this leg was expanded for."
46 + @type combination :: %{String.t() => term()}
47 +
48 + @doc """
49 + Every combination `strategy` expands to.
50 +
51 + Returns `[%{}]` — a single empty combination — when there is no
52 + matrix, so callers can treat "no matrix" as "one leg with nothing in
53 + it" rather than branching.
54 + """
55 + @spec expand(map() | nil) :: [combination()]
56 + def expand(strategy)
57 +
58 + def expand(%{"matrix" => %{} = matrix}) do
59 + axes =
60 + matrix
61 + |> Map.drop(@reserved)
62 + |> Enum.map(fn {key, values} -> {key, List.wrap(values)} end)
63 + # Map iteration order is undefined; sort so a given workflow always
64 + # expands the same way rather than shuffling between deploys.
65 + |> Enum.sort_by(&elem(&1, 0))
66 +
67 + axes
68 + |> product()
69 + |> exclude(List.wrap(matrix["exclude"]))
70 + |> include(List.wrap(matrix["include"]))
71 + |> case do
72 + [] -> [%{}]
73 + combinations -> combinations
74 + end
75 + end
76 +
77 + def expand(_strategy), do: [%{}]
78 +
79 + @doc """
80 + Whether `strategy` asks for a matrix at all.
81 +
82 + A `strategy:` with only `fail-fast:` on it is not a matrix, and its
83 + job should not be renamed as though it were one leg of something.
84 + """
85 + @spec matrix?(map() | nil) :: boolean()
86 + def matrix?(%{"matrix" => %{} = matrix}), do: matrix != %{}
87 + def matrix?(_), do: false
88 +
89 + @doc """
90 + Whether a failing leg should cancel its siblings. GitHub's default is
91 + `true`, and so is ours.
92 + """
93 + @spec fail_fast?(map() | nil) :: boolean()
94 + def fail_fast?(%{"fail-fast" => false}), do: false
95 + def fail_fast?(%{"fail-fast" => "false"}), do: false
96 + def fail_fast?(_), do: true
97 +
98 + @doc """
99 + The display suffix GitHub puts on a matrix leg — `build (arm64, 28)`.
100 +
101 + Values in the order their keys sort, which is the order `expand/1`
102 + varied them, so reading two names side by side tells you which axis
103 + differs.
104 + """
105 + @spec suffix(combination()) :: String.t() | nil
106 + def suffix(combination) when map_size(combination) == 0, do: nil
107 +
108 + def suffix(combination) do
109 + combination
110 + |> Enum.sort_by(&elem(&1, 0))
111 + |> Enum.map_join(", ", fn {_key, value} -> to_string(value) end)
112 + end
113 +
114 + @doc """
115 + Substitute `${{ matrix.key }}` in a `runs-on` value.
116 +
117 + Only `runs-on` needs this. Everything else the runner evaluates for
118 + itself — see `GitGud.Workflows.RunnerPayload`, which hands `act` a
119 + one-value matrix so the whole `matrix` context resolves natively.
120 + But routing is *our* decision and happens before any runner sees the
121 + job, so this one expression has to be resolved server-side.
122 +
123 + An unknown key resolves to the empty string, as it does in Actions.
124 + """
125 + @spec substitute(term(), combination()) :: term()
126 + def substitute(value, combination) when is_list(value),
127 + do: Enum.map(value, &substitute(&1, combination))
128 +
129 + def substitute(value, combination) when is_binary(value) do
130 + Regex.replace(~r/\$\{\{\s*matrix\.([A-Za-z_][A-Za-z0-9_-]*)\s*\}\}/, value, fn _, key ->
131 + combination |> Map.get(key, "") |> to_string()
132 + end)
133 + end
134 +
135 + def substitute(value, _combination), do: value
136 +
137 + @doc """
138 + Normalise a `runs-on` into the list of labels a runner must carry.
139 +
140 + `runs-on: ubuntu-latest` and `runs-on: [ubuntu-latest]` mean the same
141 + thing and are stored the same way, so claiming has one shape to match
142 + rather than two.
143 + """
144 + @spec labels(term()) :: [String.t()]
145 + def labels(runs_on) do
146 + runs_on
147 + |> List.wrap()
148 + |> Enum.map(&to_string/1)
149 + |> Enum.reject(&(&1 == ""))
150 + |> Enum.uniq()
151 + end
152 +
153 + @doc """
154 + Rewrite a job definition so it describes exactly this one leg.
155 +
156 + The matrix is replaced by the single combination rather than
157 + removed: `act` then expands it to exactly one job and populates the
158 + `matrix` context from it, so `${{ matrix.arch }}` inside a step keeps
159 + working without us implementing an expression evaluator. Dropping
160 + `strategy` entirely would leave those expressions resolving to
161 + nothing; leaving it whole would have the runner fan out *again*,
162 + inside the single job we just handed it.
163 + """
164 + @spec pin_definition(map(), combination()) :: map()
165 + def pin_definition(definition, combination) when map_size(combination) == 0,
166 + do: Map.drop(definition, ["strategy"])
167 +
168 + def pin_definition(definition, combination) do
169 + pinned = Map.new(combination, fn {key, value} -> {key, [value]} end)
170 + strategy = Map.get(definition, "strategy") || %{}
171 +
172 + # `fail-fast` rides along even though it means nothing to a
173 + # one-value matrix: the forge reads it back off the stored
174 + # definition to decide whether a failing leg cancels its siblings,
175 + # and the pinned strategy is the only copy that survives.
176 + kept =
177 + case Map.fetch(strategy, "fail-fast") do
178 + {:ok, value} -> %{"matrix" => pinned, "fail-fast" => value}
179 + :error -> %{"matrix" => pinned}
180 + end
181 +
182 + Map.put(definition, "strategy", kept)
183 + end
184 +
185 + # ── expansion internals ─────────────────────────────────────────────
186 +
187 + # Cartesian product with the last axis varying fastest, matching the
188 + # order Actions lists a matrix's jobs in.
189 + defp product([]), do: []
190 +
191 + defp product(axes) do
192 + Enum.reduce(axes, [%{}], fn {key, values}, acc ->
193 + for combination <- acc, value <- values, do: Map.put(combination, key, value)
194 + end)
195 + end
196 +
197 + defp exclude(combinations, []), do: combinations
198 +
199 + defp exclude(combinations, exclusions) do
200 + Enum.reject(combinations, fn combination ->
201 + Enum.any?(exclusions, &subset_of?(&1, combination))
202 + end)
203 + end
204 +
205 + defp include(combinations, []), do: combinations
206 +
207 + defp include(combinations, inclusions) do
208 + Enum.reduce(inclusions, combinations, fn entry, acc ->
209 + entry = Map.new(entry, fn {k, v} -> {to_string(k), v} end)
210 +
211 + merged = Enum.map(acc, &merge_if_compatible(&1, entry))
212 +
213 + # Nothing took it — it describes a combination of its own.
214 + if merged == acc, do: acc ++ [entry], else: merged
215 + end)
216 + end
217 +
218 + # Compatible means every key they share already agrees. An entry that
219 + # would change an existing value describes a different combination,
220 + # not an addition to this one.
221 + defp merge_if_compatible(combination, entry) do
222 + overlapping = Map.take(entry, Map.keys(combination))
223 +
224 + if subset_of?(overlapping, combination) and entry != overlapping,
225 + do: Map.merge(combination, entry),
226 + else: combination
227 + end
228 +
229 + defp subset_of?(subset, combination) do
230 + Enum.all?(subset, fn {key, value} ->
231 + Map.has_key?(combination, key) and to_string(Map.get(combination, key)) == to_string(value)
232 + end)
233 + end
234 +end
modified lib/git_gud/workflows/workflow_job.ex
+6 −1
@@ -36,6 +36,11 @@ defmodule GitGud.Workflows.WorkflowJob do
36 36 # `build_needs/1`; avoids a racy re-query of upstream status at fetch.
37 37 # Shape: %{"<need_job_id>" => %{"status" => "...", "outputs" => %{}}}.
38 38 field :needs_snapshot, :map, default: %{}
39 + # The one `strategy.matrix` combination this row was expanded for,
40 + # e.g. %{"arch" => "arm64"}. `nil` for a job with no matrix. Legs of
41 + # one matrix share a `job_id` — `needs:` names the group, and this
42 + # is what tells them apart.
43 + field :matrix, :map
39 44
40 45 belongs_to :workflow_run, WorkflowRun
41 46 belongs_to :runner, Runner
@@ -46,7 +51,7 @@ defmodule GitGud.Workflows.WorkflowJob do
46 51
47 52 def create_changeset(job, attrs) do
48 53 job
49 − |> cast(attrs, [:job_id, :name, :runs_on, :needs, :definition])
54 + |> cast(attrs, [:job_id, :name, :runs_on, :needs, :definition, :matrix])
50 55 |> validate_required([:job_id, :definition])
51 56 end
52 57
modified lib/git_gud/workflows/yaml.ex
+9 −4
@@ -8,10 +8,13 @@ defmodule GitGud.Workflows.Yaml do
8 8 * Top-level `name`, `on`, `jobs` keys
9 9 * `on:` as a string ("push"), list (["push", "pull_request"]) or map
10 10 with per-trigger config; we keep the raw map for filtering later
11 − * `jobs.<id>.{name, runs-on, needs, steps, env, if, container}` are
12 − preserved as the job's definition; richer features (services,
13 − strategy/matrix, reusable workflows, etc.) flow through as-is and
14 − will be honored by the runner that consumes them.
11 + * `jobs.<id>.{name, runs-on, needs, strategy, steps, env, if,
12 + container}` are preserved as the job's definition. `strategy` is
13 + lifted out because the forge acts on it — see
14 + `GitGud.Workflows.Matrix`, which expands a matrix into one job
15 + row per combination so the legs can be claimed by different
16 + runners. Richer features (services, reusable workflows, …) flow
17 + through as-is for the runner that consumes them.
15 18 """
16 19
17 20 @type workflow :: %{
@@ -25,6 +28,7 @@ defmodule GitGud.Workflows.Yaml do
25 28 id: String.t(),
26 29 name: String.t() | nil,
27 30 runs_on: any(),
31 + strategy: map() | nil,
28 32 needs: [String.t()],
29 33 steps: [map()],
30 34 raw: map()
@@ -79,6 +83,7 @@ defmodule GitGud.Workflows.Yaml do
79 83 id: id,
80 84 name: Map.get(definition, "name"),
81 85 runs_on: Map.get(definition, "runs-on", "ubuntu-latest"),
86 + strategy: Map.get(definition, "strategy"),
82 87 needs: Map.get(definition, "needs", []) |> List.wrap(),
83 88 steps: Map.get(definition, "steps", []),
84 89 raw: definition
added priv/repo/migrations/20260911030000_add_workflow_job_matrix.exs
+92 −0
@@ -0,0 +1,92 @@
1 +defmodule GitGud.Repo.Migrations.AddWorkflowJobMatrix do
2 + use Ecto.Migration
3 +
4 + @moduledoc """
5 + Matrix builds, and the `runs-on` change they need underneath.
6 +
7 + `runs_on` was stored as `%{"value" => <whatever the YAML said>}` and
8 + matched with `runs_on->>'value' = ANY(runner labels)`. That only ever
9 + worked for the scalar form: `runs-on: [self-hosted, arm64]` stored a
10 + JSON array, `->>` rendered it as the literal text `["self-hosted",
11 + "arm64"]`, and it matched no runner at all — silently, as a job that
12 + simply never got claimed.
13 +
14 + The list form is exactly how you select an architecture, so it has to
15 + work. `runs_on` becomes `%{"labels" => [...]}` — always a list, even
16 + for a scalar — and claiming asks whether the job's labels are
17 + *contained by* the runner's, which is GitHub's rule: a runner must
18 + carry every label the job asks for.
19 +
20 + `matrix` holds the one combination a job row was expanded for
21 + (`%{"arch" => "arm64"}`). Jobs from one matrix keep their YAML
22 + `job_id`, so `needs:` still names the group rather than a synthetic
23 + per-leg id; the combination is what tells the legs apart.
24 + """
25 +
26 + def up do
27 + alter table(:workflow_jobs) do
28 + add :matrix, :map
29 + end
30 +
31 + execute """
32 + UPDATE workflow_jobs
33 + SET runs_on = jsonb_build_object('labels',
34 + CASE jsonb_typeof(runs_on -> 'value')
35 + WHEN 'array' THEN runs_on -> 'value'
36 + WHEN 'string' THEN jsonb_build_array(runs_on -> 'value')
37 + ELSE '[]'::jsonb
38 + END)
39 + WHERE runs_on ? 'value'
40 + """
41 +
42 + # Containment (`<@`) is a GIN operator; without this every claim
43 + # sequentially scans the queued jobs, and claiming happens on every
44 + # runner poll.
45 + create index(:workflow_jobs, ["(runs_on -> 'labels') jsonb_path_ops"],
46 + using: :gin,
47 + name: :workflow_jobs_runs_on_labels_index
48 + )
49 +
50 + # `(run, job_id)` was unique, which is exactly what a matrix breaks:
51 + # its legs deliberately share a `job_id` so `needs:` can name the
52 + # group. The combination is what separates them, and `COALESCE` is
53 + # load-bearing — NULLs compare as distinct in a unique index, so
54 + # without it two matrix-less jobs with the same id would slip
55 + # through the very constraint being replaced.
56 + drop unique_index(:workflow_jobs, [:workflow_run_id, :job_id])
57 +
58 + create unique_index(
59 + :workflow_jobs,
60 + [:workflow_run_id, :job_id, "(COALESCE(matrix, '{}'::jsonb))"],
61 + name: :workflow_jobs_run_job_matrix_index
62 + )
63 + end
64 +
65 + def down do
66 + # `IF EXISTS` / `IF NOT EXISTS` throughout: a teardown that only
67 + # works from the exact state the matching `up` left behind is a
68 + # teardown you can't run when you actually need it.
69 + execute "DROP INDEX IF EXISTS workflow_jobs_run_job_matrix_index"
70 +
71 + execute """
72 + CREATE UNIQUE INDEX IF NOT EXISTS workflow_jobs_workflow_run_id_job_id_index
73 + ON workflow_jobs (workflow_run_id, job_id)
74 + """
75 +
76 + execute "DROP INDEX IF EXISTS workflow_jobs_runs_on_labels_index"
77 +
78 + # Back to the scalar shape. A multi-label job can't be represented,
79 + # so it keeps its first label — which is what the old code would
80 + # have matched on had it been written as a scalar.
81 + execute """
82 + UPDATE workflow_jobs
83 + SET runs_on = jsonb_build_object('value',
84 + COALESCE(runs_on -> 'labels' -> 0, '"ubuntu-latest"'::jsonb))
85 + WHERE runs_on ? 'labels'
86 + """
87 +
88 + alter table(:workflow_jobs) do
89 + remove_if_exists :matrix, :map
90 + end
91 + end
92 +end
added test/git_gud/workflows/matrix_dispatch_test.exs
+509 −0
@@ -0,0 +1,509 @@
1 +defmodule GitGud.Workflows.MatrixDispatchTest do
2 + @moduledoc """
3 + Matrix builds end to end through the orchestration: a workflow with a
4 + matrix becomes several jobs, each routed to a runner that carries the
5 + labels that leg asked for.
6 +
7 + The motivating case is two architectures — an arm64 machine and an
8 + amd64 one, both feeding one workflow — which is also the case that
9 + exercises every moving part: expansion, `runs-on` substitution, and
10 + label-set claiming.
11 + """
12 + use GitGud.DataCase, async: false
13 +
14 + alias GitGud.Workflows
15 + alias GitGud.Workflows.Runners
16 + alias GitGud.Workflows.WorkflowJob
17 + alias GitGud.Workflows.WorkflowRun
18 + alias GitGud.Workflows.Yaml
19 +
20 + import GitGud.ForgeFixtures
21 +
22 + setup do
23 + {_user, repo} = repository_fixture()
24 + {:ok, repo: repo}
25 + end
26 +
27 + # `dispatch/2` wants a real git tree to read workflows out of, and the
28 + # tree isn't what's under test. `rerun/1` seeds jobs from a stored
29 + # `workflow_doc` through exactly the same creation path, so a seed row
30 + # plus a re-run gets the real thing without a repository on disk.
31 + defp run_workflow(repo, yaml) do
32 + {:ok, doc} = Yaml.parse(yaml)
33 +
34 + seed =
35 + %WorkflowRun{repository_id: repo.id}
36 + |> WorkflowRun.create_changeset(%{
37 + "number" => System.unique_integer([:positive]),
38 + "workflow_path" => ".forgejo/workflows/ci.yml",
39 + "workflow_name" => doc.name,
40 + "trigger" => "push",
41 + "head_ref" => "refs/heads/main",
42 + "head_sha" => :crypto.strong_rand_bytes(20),
43 + "workflow_doc" => doc.raw
44 + })
45 + |> Repo.insert!()
46 +
47 + {:ok, run} = Workflows.rerun(seed)
48 + run
49 + end
50 +
51 + defp jobs(run) do
52 + from(j in WorkflowJob, where: j.workflow_run_id == ^run.id, order_by: j.id) |> Repo.all()
53 + end
54 +
55 + defp runner(repo, labels) do
56 + {:ok, runner, _uuid, _token} =
57 + Runners.create_offline(
58 + %{name: "r-#{System.unique_integer([:positive])}", labels: %{"labels" => labels}},
59 + repo
60 + )
61 +
62 + runner
63 + end
64 +
65 + describe "expansion" do
66 + test "a two-architecture matrix becomes two jobs", %{repo: repo} do
67 + run =
68 + run_workflow(repo, """
69 + name: ci
70 + on: push
71 + jobs:
72 + build:
73 + strategy:
74 + matrix:
75 + arch: [amd64, arm64]
76 + runs-on: [self-hosted, "${{ matrix.arch }}"]
77 + steps:
78 + - run: echo building for ${{ matrix.arch }}
79 + """)
80 +
81 + assert [amd, arm] = jobs(run)
82 +
83 + # One YAML job id, two rows — `needs: build` still names the group.
84 + assert amd.job_id == "build"
85 + assert arm.job_id == "build"
86 +
87 + assert amd.name == "build (amd64)"
88 + assert arm.name == "build (arm64)"
89 +
90 + assert amd.matrix == %{"arch" => "amd64"}
91 + assert arm.matrix == %{"arch" => "arm64"}
92 + end
93 +
94 + test "`runs-on` is resolved per leg, because it decides the runner", %{repo: repo} do
95 + run =
96 + run_workflow(repo, """
97 + name: ci
98 + on: push
99 + jobs:
100 + build:
101 + strategy:
102 + matrix:
103 + arch: [amd64, arm64]
104 + runs-on: [self-hosted, "${{ matrix.arch }}"]
105 + steps:
106 + - run: true
107 + """)
108 +
109 + assert [amd, arm] = jobs(run)
110 +
111 + assert amd.runs_on == %{"labels" => ["self-hosted", "amd64"]}
112 + assert arm.runs_on == %{"labels" => ["self-hosted", "arm64"]}
113 + end
114 +
115 + test "each leg's stored definition pins the matrix to one value", %{repo: repo} do
116 + run =
117 + run_workflow(repo, """
118 + name: ci
119 + on: push
120 + jobs:
121 + build:
122 + strategy:
123 + matrix:
124 + arch: [amd64, arm64]
125 + runs-on: self-hosted
126 + steps:
127 + - run: echo ${{ matrix.arch }}
128 + """)
129 +
130 + assert [amd, _arm] = jobs(run)
131 +
132 + # One value, so the runner's own engine expands it to exactly one
133 + # job and `${{ matrix.arch }}` in the step still resolves. Left
134 + # whole, the runner would fan out again inside this one job.
135 + assert amd.definition["strategy"] == %{"matrix" => %{"arch" => ["amd64"]}}
136 + end
137 +
138 + test "a job with no matrix is still one job, and keeps its name", %{repo: repo} do
139 + run =
140 + run_workflow(repo, """
141 + name: ci
142 + on: push
143 + jobs:
144 + test:
145 + name: unit tests
146 + runs-on: ubuntu-latest
147 + steps:
148 + - run: true
149 + """)
150 +
151 + assert [job] = jobs(run)
152 + assert job.name == "unit tests"
153 + assert job.matrix == nil
154 + assert job.runs_on == %{"labels" => ["ubuntu-latest"]}
155 + refute Map.has_key?(job.definition, "strategy")
156 + end
157 +
158 + test "steps are created for every leg", %{repo: repo} do
159 + run =
160 + run_workflow(repo, """
161 + name: ci
162 + on: push
163 + jobs:
164 + build:
165 + strategy:
166 + matrix:
167 + arch: [amd64, arm64]
168 + runs-on: self-hosted
169 + steps:
170 + - run: one
171 + - run: two
172 + """)
173 +
174 + for job <- jobs(run) do
175 + assert length(Repo.preload(job, :steps).steps) == 2
176 + end
177 + end
178 + end
179 +
180 + describe "routing to the right architecture" do
181 + setup %{repo: repo} do
182 + run =
183 + run_workflow(repo, """
184 + name: ci
185 + on: push
186 + jobs:
187 + build:
188 + strategy:
189 + matrix:
190 + arch: [amd64, arm64]
191 + runs-on: [self-hosted, "${{ matrix.arch }}"]
192 + steps:
193 + - run: true
194 + """)
195 +
196 + {:ok, run: run}
197 + end
198 +
199 + test "an arm64 runner claims the arm64 leg and only that one", %{repo: repo, run: _run} do
200 + arm = runner(repo, ["self-hosted", "arm64"])
201 +
202 + assert %WorkflowJob{} = claimed = Workflows.claim_next_job(arm)
203 + assert claimed.matrix == %{"arch" => "arm64"}
204 +
205 + # Nothing else here is for arm64.
206 + assert Workflows.claim_next_job(arm) == nil
207 + end
208 +
209 + test "two runners of different architectures take one leg each", %{repo: repo} do
210 + amd = runner(repo, ["self-hosted", "amd64"])
211 + arm = runner(repo, ["self-hosted", "arm64"])
212 +
213 + assert amd_job = Workflows.claim_next_job(amd)
214 + assert arm_job = Workflows.claim_next_job(arm)
215 +
216 + assert amd_job.matrix == %{"arch" => "amd64"}
217 + assert arm_job.matrix == %{"arch" => "arm64"}
218 + refute amd_job.id == arm_job.id
219 +
220 + # Both legs are spoken for.
221 + assert Workflows.claim_next_job(amd) == nil
222 + assert Workflows.claim_next_job(arm) == nil
223 + end
224 +
225 + test "a runner missing one of the required labels claims nothing", %{repo: repo} do
226 + # Carries `self-hosted` but neither architecture.
227 + idle = runner(repo, ["self-hosted", "riscv64"])
228 +
229 + assert Workflows.claim_next_job(idle) == nil
230 + end
231 +
232 + test "a runner with extra labels still qualifies — containment, not equality", %{repo: repo} do
233 + beefy = runner(repo, ["self-hosted", "arm64", "linux", "gpu"])
234 +
235 + assert claimed = Workflows.claim_next_job(beefy)
236 + assert claimed.matrix == %{"arch" => "arm64"}
237 + end
238 + end
239 +
240 + describe "routing without a matrix" do
241 + # Its own block: the matrix setup above queues work for both
242 + # architectures, which is exactly what a "this runner should find
243 + # nothing" assertion cannot have sitting around.
244 + test "a scalar runs-on still routes, which is every pre-existing workflow", %{repo: repo} do
245 + run =
246 + run_workflow(repo, """
247 + name: ci
248 + on: push
249 + jobs:
250 + plain:
251 + runs-on: ubuntu-latest
252 + steps:
253 + - run: true
254 + """)
255 +
256 + wrong = runner(repo, ["self-hosted", "arm64"])
257 + right = runner(repo, ["ubuntu-latest"])
258 +
259 + assert Workflows.claim_next_job(wrong) == nil
260 +
261 + assert claimed = Workflows.claim_next_job(right)
262 + assert claimed.workflow_run_id == run.id
263 + end
264 +
265 + test "`runs-on: self-hosted` matches any self-registered runner", %{repo: repo} do
266 + run_workflow(repo, """
267 + name: ci
268 + on: push
269 + jobs:
270 + anywhere:
271 + runs-on: self-hosted
272 + steps:
273 + - run: true
274 + """)
275 +
276 + # Doesn't carry the `self-hosted` label itself; it's implied.
277 + any = runner(repo, ["arm64"])
278 +
279 + assert claimed = Workflows.claim_next_job(any)
280 + assert claimed.job_id == "anywhere"
281 + end
282 + end
283 +
284 + describe "dependencies across a matrix" do
285 + setup %{repo: repo} do
286 + run =
287 + run_workflow(repo, """
288 + name: ci
289 + on: push
290 + jobs:
291 + build:
292 + strategy:
293 + matrix:
294 + arch: [amd64, arm64]
295 + runs-on: [self-hosted, "${{ matrix.arch }}"]
296 + steps:
297 + - run: true
298 + release:
299 + needs: build
300 + runs-on: self-hosted
301 + steps:
302 + - run: true
303 + """)
304 +
305 + {:ok, run: run}
306 + end
307 +
308 + test "a downstream job waits for every leg, not the first", %{run: run} do
309 + [amd, arm, release] = jobs(run)
310 +
311 + assert release.status == "blocked"
312 +
313 + {:ok, _} = Workflows.update_job_status(amd, "success")
314 + assert Repo.get!(WorkflowJob, release.id).status == "blocked"
315 +
316 + {:ok, _} = Workflows.update_job_status(arm, "success")
317 + assert Repo.get!(WorkflowJob, release.id).status == "queued"
318 + end
319 +
320 + test "one failing leg cancels the downstream job", %{run: run} do
321 + [amd, arm, release] = jobs(run)
322 +
323 + {:ok, _} = Workflows.update_job_status(amd, "success")
324 + {:ok, _} = Workflows.update_job_status(arm, "failure")
325 +
326 + assert Repo.get!(WorkflowJob, release.id).status == "cancelled"
327 + end
328 +
329 + test "the snapshot collapses the legs into one name", %{run: run} do
330 + [amd, arm, release] = jobs(run)
331 +
332 + {:ok, _} = Workflows.update_job_status(amd, "success")
333 + {:ok, _} = Workflows.update_job_status(arm, "success")
334 +
335 + # Downstream `${{ needs.build }}` is one name however many
336 + # machines produced it.
337 + assert %{"build" => %{"status" => "success"}} =
338 + Repo.get!(WorkflowJob, release.id).needs_snapshot
339 + end
340 + end
341 +
342 + describe "fail-fast" do
343 + test "a failing leg cancels its queued siblings", %{repo: repo} do
344 + run =
345 + run_workflow(repo, """
346 + name: ci
347 + on: push
348 + jobs:
349 + build:
350 + strategy:
351 + matrix:
352 + arch: [amd64, arm64, riscv64]
353 + runs-on: [self-hosted, "${{ matrix.arch }}"]
354 + steps:
355 + - run: true
356 + """)
357 +
358 + [first | rest] = jobs(run)
359 +
360 + {:ok, _} = Workflows.update_job_status(first, "failure")
361 +
362 + for sibling <- rest do
363 + assert Repo.get!(WorkflowJob, sibling.id).status == "cancelled"
364 + end
365 + end
366 +
367 + test "`fail-fast: false` lets the others finish", %{repo: repo} do
368 + run =
369 + run_workflow(repo, """
370 + name: ci
371 + on: push
372 + jobs:
373 + build:
374 + strategy:
375 + fail-fast: false
376 + matrix:
377 + arch: [amd64, arm64, riscv64]
378 + runs-on: [self-hosted, "${{ matrix.arch }}"]
379 + steps:
380 + - run: true
381 + """)
382 +
383 + [first | rest] = jobs(run)
384 +
385 + {:ok, _} = Workflows.update_job_status(first, "failure")
386 +
387 + for sibling <- rest do
388 + assert Repo.get!(WorkflowJob, sibling.id).status == "queued"
389 + end
390 + end
391 +
392 + test "a leg already running is left to finish rather than killed", %{repo: repo} do
393 + run =
394 + run_workflow(repo, """
395 + name: ci
396 + on: push
397 + jobs:
398 + build:
399 + strategy:
400 + matrix:
401 + arch: [amd64, arm64]
402 + runs-on: self-hosted
403 + steps:
404 + - run: true
405 + """)
406 +
407 + [first, second] = jobs(run)
408 +
409 + {:ok, _} = Workflows.update_job_status(second, "running")
410 + {:ok, _} = Workflows.update_job_status(first, "failure")
411 +
412 + # Killing it mid-flight would lose its logs for no gain.
413 + assert Repo.get!(WorkflowJob, second.id).status == "running"
414 + end
415 +
416 + test "a job with no matrix never fail-fasts its unrelated siblings", %{repo: repo} do
417 + run =
418 + run_workflow(repo, """
419 + name: ci
420 + on: push
421 + jobs:
422 + a:
423 + runs-on: self-hosted
424 + steps:
425 + - run: true
426 + b:
427 + runs-on: self-hosted
428 + steps:
429 + - run: true
430 + """)
431 +
432 + [a, b] = jobs(run)
433 +
434 + {:ok, _} = Workflows.update_job_status(a, "failure")
435 +
436 + assert Repo.get!(WorkflowJob, b.id).status == "queued"
437 + end
438 + end
439 +
440 + describe "re-running" do
441 + test "re-running a matrix job re-runs every leg, not the first one found", %{repo: repo} do
442 + run =
443 + run_workflow(repo, """
444 + name: ci
445 + on: push
446 + jobs:
447 + build:
448 + strategy:
449 + fail-fast: false
450 + matrix:
451 + arch: [amd64, arm64]
452 + runs-on: self-hosted
453 + steps:
454 + - run: true
455 + """)
456 +
457 + [amd, arm] = jobs(run)
458 + {:ok, _} = Workflows.update_job_status(amd, "failure")
459 + {:ok, _} = Workflows.update_job_status(arm, "success")
460 +
461 + assert {:ok, _} = Workflows.rerun_job(run, "build")
462 +
463 + assert Repo.get!(WorkflowJob, amd.id).status == "queued"
464 + assert Repo.get!(WorkflowJob, arm.id).status == "queued"
465 + end
466 +
467 + test "an unknown job id is still not found", %{repo: repo} do
468 + run =
469 + run_workflow(repo, """
470 + name: ci
471 + on: push
472 + jobs:
473 + build:
474 + runs-on: self-hosted
475 + steps:
476 + - run: true
477 + """)
478 +
479 + assert {:error, :not_found} = Workflows.rerun_job(run, "nope")
480 + end
481 + end
482 +
483 + describe "the run as a whole" do
484 + test "completes only once every leg is terminal", %{repo: repo} do
485 + run =
486 + run_workflow(repo, """
487 + name: ci
488 + on: push
489 + jobs:
490 + build:
491 + strategy:
492 + fail-fast: false
493 + matrix:
494 + arch: [amd64, arm64]
495 + runs-on: self-hosted
496 + steps:
497 + - run: true
498 + """)
499 +
500 + [amd, arm] = jobs(run)
501 +
502 + {:ok, _} = Workflows.update_job_status(amd, "success")
503 + assert Repo.get!(WorkflowRun, run.id).status in ~w(queued running)
504 +
505 + {:ok, _} = Workflows.update_job_status(arm, "success")
506 + assert Repo.get!(WorkflowRun, run.id).status == "success"
507 + end
508 + end
509 +end
added test/git_gud/workflows/matrix_test.exs
+243 −0
@@ -0,0 +1,243 @@
1 +defmodule GitGud.Workflows.MatrixTest do
2 + @moduledoc """
3 + Matrix expansion, checked against the examples in GitHub's own
4 + workflow-syntax documentation — `include` and `exclude` are the parts
5 + people actually get wrong, and "it seemed reasonable" is not a
6 + specification.
7 + """
8 + use ExUnit.Case, async: true
9 +
10 + alias GitGud.Workflows.Matrix
11 +
12 + defp strategy(matrix), do: %{"matrix" => matrix}
13 +
14 + describe "expand/1" do
15 + test "no strategy is one leg with nothing in it" do
16 + assert Matrix.expand(nil) == [%{}]
17 + assert Matrix.expand(%{}) == [%{}]
18 + assert Matrix.expand(%{"fail-fast" => false}) == [%{}]
19 + end
20 +
21 + test "one axis is one job per value" do
22 + assert Matrix.expand(strategy(%{"arch" => ["amd64", "arm64"]})) == [
23 + %{"arch" => "amd64"},
24 + %{"arch" => "arm64"}
25 + ]
26 + end
27 +
28 + test "two axes are their product" do
29 + combinations =
30 + Matrix.expand(strategy(%{"arch" => ["amd64", "arm64"], "otp" => ["27", "28"]}))
31 +
32 + assert length(combinations) == 4
33 +
34 + assert %{"arch" => "amd64", "otp" => "27"} in combinations
35 + assert %{"arch" => "arm64", "otp" => "28"} in combinations
36 + end
37 +
38 + test "a scalar value is a single-value axis" do
39 + assert Matrix.expand(strategy(%{"arch" => "arm64"})) == [%{"arch" => "arm64"}]
40 + end
41 +
42 + test "expansion is stable — map order must not decide job order" do
43 + matrix = %{"arch" => ["amd64", "arm64"], "otp" => ["27", "28"], "os" => ["a", "b"]}
44 +
45 + assert Matrix.expand(strategy(matrix)) == Matrix.expand(strategy(matrix))
46 + end
47 + end
48 +
49 + describe "exclude" do
50 + test "drops combinations matching every key it names" do
51 + combinations =
52 + Matrix.expand(
53 + strategy(%{
54 + "arch" => ["amd64", "arm64"],
55 + "otp" => ["27", "28"],
56 + "exclude" => [%{"arch" => "arm64", "otp" => "27"}]
57 + })
58 + )
59 +
60 + assert length(combinations) == 3
61 + refute %{"arch" => "arm64", "otp" => "27"} in combinations
62 + end
63 +
64 + test "a partial exclude drops every combination it matches" do
65 + combinations =
66 + Matrix.expand(
67 + strategy(%{
68 + "arch" => ["amd64", "arm64"],
69 + "otp" => ["27", "28"],
70 + "exclude" => [%{"arch" => "arm64"}]
71 + })
72 + )
73 +
74 + assert combinations == [
75 + %{"arch" => "amd64", "otp" => "27"},
76 + %{"arch" => "amd64", "otp" => "28"}
77 + ]
78 + end
79 + end
80 +
81 + describe "include" do
82 + test "adds keys to the combinations it matches rather than appending a job" do
83 + combinations =
84 + Matrix.expand(
85 + strategy(%{
86 + "arch" => ["amd64", "arm64"],
87 + "include" => [%{"arch" => "arm64", "coverage" => true}]
88 + })
89 + )
90 +
91 + # Still two jobs — the arm64 one just gained a key.
92 + assert combinations == [
93 + %{"arch" => "amd64"},
94 + %{"arch" => "arm64", "coverage" => true}
95 + ]
96 + end
97 +
98 + test "an entry matching nothing becomes a combination of its own" do
99 + combinations =
100 + Matrix.expand(
101 + strategy(%{
102 + "arch" => ["amd64", "arm64"],
103 + "include" => [%{"arch" => "riscv64"}]
104 + })
105 + )
106 +
107 + assert combinations == [
108 + %{"arch" => "amd64"},
109 + %{"arch" => "arm64"},
110 + %{"arch" => "riscv64"}
111 + ]
112 + end
113 +
114 + test "an entry with no overlapping keys applies to every combination" do
115 + combinations =
116 + Matrix.expand(
117 + strategy(%{
118 + "arch" => ["amd64", "arm64"],
119 + "include" => [%{"otp" => "28"}]
120 + })
121 + )
122 +
123 + assert combinations == [
124 + %{"arch" => "amd64", "otp" => "28"},
125 + %{"arch" => "arm64", "otp" => "28"}
126 + ]
127 + end
128 +
129 + test "include runs after exclude, so it can add a combination back" do
130 + combinations =
131 + Matrix.expand(
132 + strategy(%{
133 + "arch" => ["amd64", "arm64"],
134 + "exclude" => [%{"arch" => "arm64"}],
135 + "include" => [%{"arch" => "arm64", "experimental" => true}]
136 + })
137 + )
138 +
139 + assert combinations == [
140 + %{"arch" => "amd64"},
141 + %{"arch" => "arm64", "experimental" => true}
142 + ]
143 + end
144 +
145 + test "an include-only matrix is just its entries" do
146 + combinations =
147 + Matrix.expand(
148 + strategy(%{
149 + "include" => [
150 + %{"arch" => "amd64", "runner" => "big"},
151 + %{"arch" => "arm64", "runner" => "small"}
152 + ]
153 + })
154 + )
155 +
156 + assert combinations == [
157 + %{"arch" => "amd64", "runner" => "big"},
158 + %{"arch" => "arm64", "runner" => "small"}
159 + ]
160 + end
161 + end
162 +
163 + describe "substitute/2" do
164 + test "resolves matrix references in a runs-on" do
165 + assert Matrix.substitute("${{ matrix.arch }}", %{"arch" => "arm64"}) == "arm64"
166 + assert Matrix.substitute("${{matrix.arch}}", %{"arch" => "arm64"}) == "arm64"
167 + assert Matrix.substitute("linux-${{ matrix.arch }}", %{"arch" => "arm64"}) == "linux-arm64"
168 + end
169 +
170 + test "resolves inside a list, which is how an architecture is usually selected" do
171 + assert Matrix.substitute(["self-hosted", "${{ matrix.arch }}"], %{"arch" => "arm64"}) ==
172 + ["self-hosted", "arm64"]
173 + end
174 +
175 + test "an unknown key resolves to empty, as it does in Actions" do
176 + assert Matrix.substitute("${{ matrix.nope }}", %{"arch" => "arm64"}) == ""
177 + end
178 +
179 + test "leaves other expressions alone — the runner evaluates those" do
180 + assert Matrix.substitute("${{ github.sha }}", %{"arch" => "arm64"}) == "${{ github.sha }}"
181 + end
182 +
183 + test "non-strings pass through" do
184 + assert Matrix.substitute(42, %{}) == 42
185 + assert Matrix.substitute(nil, %{}) == nil
186 + end
187 + end
188 +
189 + describe "labels/1" do
190 + test "a scalar and a one-element list are the same thing" do
191 + assert Matrix.labels("ubuntu-latest") == ["ubuntu-latest"]
192 + assert Matrix.labels(["ubuntu-latest"]) == ["ubuntu-latest"]
193 + end
194 +
195 + test "keeps every label in a list — all of them must be satisfied" do
196 + assert Matrix.labels(["self-hosted", "arm64"]) == ["self-hosted", "arm64"]
197 + end
198 +
199 + test "drops blanks and duplicates" do
200 + assert Matrix.labels(["self-hosted", "", "self-hosted"]) == ["self-hosted"]
201 + end
202 + end
203 +
204 + describe "pin_definition/2" do
205 + test "pins the matrix to this leg so the runner expands it to exactly one job" do
206 + definition = %{
207 + "runs-on" => "self-hosted",
208 + "strategy" => %{"matrix" => %{"arch" => ["amd64", "arm64"]}, "fail-fast" => false},
209 + "steps" => [%{"run" => "echo ${{ matrix.arch }}"}]
210 + }
211 +
212 + pinned = Matrix.pin_definition(definition, %{"arch" => "arm64"})
213 +
214 + # One value, so act fans out to one job and the `matrix` context
215 + # still resolves for the step.
216 + # `fail-fast` rides along: the forge reads it back off the stored
217 + # definition, and this is the only copy that survives.
218 + assert pinned["strategy"] == %{"matrix" => %{"arch" => ["arm64"]}, "fail-fast" => false}
219 + assert pinned["steps"] == definition["steps"]
220 + end
221 +
222 + test "a job with no matrix loses its strategy entirely" do
223 + definition = %{"strategy" => %{"fail-fast" => false}, "runs-on" => "x"}
224 +
225 + assert Matrix.pin_definition(definition, %{}) == %{"runs-on" => "x"}
226 + end
227 + end
228 +
229 + describe "suffix/1 and fail_fast?/1" do
230 + test "names a leg the way Actions does" do
231 + assert Matrix.suffix(%{}) == nil
232 + assert Matrix.suffix(%{"arch" => "arm64"}) == "arm64"
233 + assert Matrix.suffix(%{"arch" => "arm64", "otp" => "28"}) == "arm64, 28"
234 + end
235 +
236 + test "fail-fast defaults on, and only an explicit false turns it off" do
237 + assert Matrix.fail_fast?(nil)
238 + assert Matrix.fail_fast?(%{"matrix" => %{}})
239 + refute Matrix.fail_fast?(%{"fail-fast" => false})
240 + refute Matrix.fail_fast?(%{"fail-fast" => "false"})
241 + end
242 + end
243 +end

Parents: 279c2d1