neiam /gitgud

Git Gud

public · Issues · Pulls · Labels · Forks · Compare · Actions success · Badges · Packages

Actions: matrices computed by earlier jobs, max-parallel, and an empty matrix fails

6b0b893 · Gabriel Morell · 2026-10-17 17:28

20 files +899 -72

Message

{commit_body(@commit)}

Files changed

modified README.md

+2 −2

@@ -86,7 +86,7 @@ reinvent them.

86	86	- CI
87	87	- Speaks the Forgejo Actions runner protocol (Twirp) — point any modern `forgejo-runner` at it; protocol path autodetected (`ping.v1`, `runner.v1`, with/without `_apis/`)
88	88	- Repo-scoped + org-scoped runner registration with one-shot tokens minted from the settings UI
89		− - Matrix builds, with `runs-on` as a real label set — `strategy.matrix` expands into one job per combination, each claimed independently, so `runs-on: [self-hosted, "${{ matrix.arch }}"]` sends the arm64 leg to an arm64 machine and the amd64 leg to an amd64 one. `include`/`exclude` follow GitHub's semantics, `fail-fast` defaults on, and `needs:` names the group rather than a leg
	89	+ - Matrix builds, with `runs-on` as a real label set — `strategy.matrix` expands into one job per combination, each claimed independently, so `runs-on: [self-hosted, "${{ matrix.arch }}"]` sends the arm64 leg to an arm64 machine and the amd64 leg to an amd64 one. `include`/`exclude` follow GitHub's semantics, `fail-fast` defaults on, `max-parallel` caps how many legs of the job run at once, and `needs:` names the group rather than a leg. The matrix, an axis, or `include`/`exclude` can be computed — `matrix: ${{ fromJSON(needs.setup.outputs.matrix) }}` — with the expression subset below plus `fromJSON()`/`toJSON()`; one that reads `needs` shows as a single job until those needs finish, then becomes its legs. A matrix with no combinations left (everything excluded, an empty axis) or an expression that doesn't evaluate to the right shape fails the job, saying why in its log
90	90	- Five deployment styles offered in the registration UI: Native binary, Docker, Podman, Podman Quadlet, Kubernetes (Deployment + dind sidecar) — copy-pasteable `register` + `daemon` snippets per style
91	91	- Triggers: `push` (with `branches:` / `branches-ignore:` / `tags:` / `tags-ignore:`, GitHub's rule that naming one kind excludes the other kind of ref, and a list beating its `-ignore` if both are set), `pull_request` and `workflow_dispatch`
92	92	- `on: pull_request` for pull requests within one repository: `types:` `opened`, `synchronize` (the head moved) and `reopened` (the default; other activity types never fire). `branches:` / `branches-ignore:` filter the target branch, and `paths:` the PR's diff from its merge-base. Workflows are read from the PR's head commit, so a PR can change its own CI, and the job tests the head itself at `refs/pull/<N>/head` — not a merge with the target, as GitHub's `refs/pull/<N>/merge` does. The job gets `github.event_name`, `head_ref` and `base_ref` (the source and target branches), `ref_name` `<N>/head` and an empty `ref_type`. Fork and federated pull requests don't run yet: their head is code the target's maintainers haven't reviewed, and a job is handed the target repository's secrets and a token that can publish its packages; GitHub runs them without secrets and with a read-only token, and until that exists here they don't run at all. `pull_request_target` isn't supported

@@ -95,7 +95,7 @@ reinvent them.

95	95	- The job's context follows Forgejo's: `github.event_name`, `ref`, `ref_name`, `ref_type`, `sha`, `repository` (`owner/name`), `repository_id`, `repository_owner(_id)`, `actor(_id)`, `run_id`, `run_number`, `run_attempt` (1, bumped by re-running the job; re-running the whole run makes a new run and a new `run_id`), `workflow`, `workflow_ref` (`owner/name/<path>@<ref>`), `job`, `server_url`, `api_url`, `token`. `github.event` is GitHub's webhook payload, trimmed: every event has `repository` and `sender`; `push` has `ref`, `before` (40 zeros for a new ref), `after`, `created`, `deleted`, `forced` (always `false`: not tracked), `compare`, `commits` (at most 20) and `head_commit`, `pusher`; `pull_request` has `action`, `number` and `pull_request` (`title`, `body`, `html_url`, `user`, `head`/`base` with `ref`, `sha`, `label`, `repo`); `workflow_dispatch` has `inputs`, `ref`, `workflow`. `secrets.GITHUB_TOKEN` (and `GITEA_TOKEN`, `FORGEJO_TOKEN`) is the job token. The workflow's top-level `env:`, `defaults:`, `permissions:`, `run-name:` and `on:` reach the runner with the job
96	96	- Jobs that depend on jobs, as on GitHub: `${{ needs.<id>.outputs.* }}` and `needs.<id>.result` (`success`, `failure`, `cancelled`, `skipped`; for a matrix, the worst leg's, with the legs' outputs merged) reach the dependent job. A job waits until all its needs finish; if any didn't succeed it is skipped — and its own dependents with it — unless its `if:` calls `always()`, `failure()` or `cancelled()`, so cleanup and "report the failure" jobs run. A job its own `if:` skipped shows as skipped, not green. Job-level `continue-on-error: true` (or `${{ matrix.* }}` that comes out `true`) keeps a failure from skipping its dependents, failing the run or fail-fasting its matrix siblings, though the job still shows as failed; any other expression there counts as false. Skipped jobs don't fail a run. Unlike GitHub, cancelling a run cancels its waiting jobs too, `always()` or not
97	97	- Re-run / cancel buttons on the workflow show page. Cancelling stops running jobs: the runner hears it in the reply to its next progress report and stops the job, still uploading its log; a cancelled job stays cancelled whatever its runner reports afterwards
98		− - `concurrency:`, workflow and job level, as a group name or `{group:, cancel-in-progress:}`: within a group (per repository) one run or job is in progress and at most one waits (shown as "Waiting for <group>"); a newer arrival cancels the waiting one, or with `cancel-in-progress` the one in progress too, and when the one in progress finishes the waiting one starts. A run (or job) cancelled only by a newer one in its group concludes `cancelled`, not failed. A workflow's group is decided when the run is created, a job's when it would be queued, so it can read `matrix` and `needs`. Group and `cancel-in-progress` may use `${{ }}` with the forge's own expression subset: property paths into `github`, `inputs`, `vars`, `matrix`, `needs` (`a.b`, `a['b']`), string/number/`true`/`false`/`null` literals, `\|\|`, `&&`, `==`, `!=`, `!`, parentheses and `format()`. A group outside that subset (any other function, say) is ignored with a warning, shown on the job — the run goes ahead without concurrency control rather than being held on a guess. Re-running a single job puts it back through its job-level group, not its run's
	98	+ - `concurrency:`, workflow and job level, as a group name or `{group:, cancel-in-progress:}`: within a group (per repository) one run or job is in progress and at most one waits (shown as "Waiting for <group>"); a newer arrival cancels the waiting one, or with `cancel-in-progress` the one in progress too, and when the one in progress finishes the waiting one starts. A run (or job) cancelled only by a newer one in its group concludes `cancelled`, not failed. A workflow's group is decided when the run is created, a job's when it would be queued, so it can read `matrix` and `needs`. Group and `cancel-in-progress` may use `${{ }}` with the forge's own expression subset: property paths into `github`, `inputs`, `vars`, `matrix`, `needs` (`a.b`, `a['b']`), string/number/`true`/`false`/`null` literals, `\|\|`, `&&`, `==`, `!=`, `!`, parentheses, `format()`, `fromJSON()` and `toJSON()`. A group outside that subset (any other function, say) is ignored with a warning, shown on the job — the run goes ahead without concurrency control rather than being held on a guess. Re-running a single job puts it back through its job-level group, not its run's
99	99	- A commit status per job, named as Forgejo names them — `<workflow> / <job> (<event>)`, e.g. `ci / test (push)` or `ci / build (arm64, 28) (pull_request)` for a matrix leg — so a pull request's `push` run and `pull_request` run on the same head (or two jobs of one workflow) don't overwrite each other. Queued, blocked, waiting and running jobs are pending; success and skipped are success; failure and cancelled are failure, except a job cancelled by a newer one in its concurrency group, which shows as cancelled. Each pill links to its job on the run page, and the pull request page updates them live
100	100	- Required status checks in branch protection: with "Require status checks to pass before merging", a pull request merges into the branch only when every required check (one per line; `` matches anything, `/` and spaces included, so `ci / test ` covers both events) matches at least one status on its head and every status it matches is success. An empty list means every status on the head, and there must be one. The merge box lists what's missing or not passing and disables the merge; an admin's override merges past them, recorded like a review-rule override. Direct pushes aren't checked — there's no head to have checks on, as on Forgejo
101	101	- The job token (`${{ github.token }}`, an HS256 JWT minted per job on `FetchTask`) works only while its job runs and for five minutes after (for the runner's post-job cache save and artifact upload), and only for that attempt of the job — a leaked token no longer stays valid for a day. It clones and fetches its own repository, reads and publishes its packages, reads its metadata over the API, dispatches workflows (its own repository's, or one that granted it), and reaches the artifact + cache endpoints of its run. `permissions:` narrows it as on GitHub: workflow level, replaced by a job's own; `read-all`, `write-all`, `{}` or a map in which unnamed scopes are `none`. `contents` (`read` fetch, `write` also push — to its own repository, subject to branch protection like any direct push, attributed to whoever started the run), `packages` (`read` / `write`) and `actions` (`write` to dispatch) mean something; `metadata` is always read; the others (`issues`, `pull-requests`, `statuses`, `checks`, `deployments`, `id-token`, …) are accepted and ignored, since the token can't do those things here anyway. With no `permissions:` the token keeps `contents: read, packages: write, actions: write` — GitHub's permissive default would also let it push. A push made with the job token starts no workflows (no `push` run, no pull request `synchronize`), as on GitHub, so a workflow that pushes can't trigger itself forever; webhooks and pull request heads still update

modified lib/git_gud/workflows.ex

+220 −15

@@ -456,6 +456,9 @@ defmodule GitGud.Workflows do

456	456	# concurrency group cancelled it (`run_concurrency/2`).
457	457	@run_superseded "Cancelled by run #"
458	458
	459	+ # The `hold` of a job created outside `create_run/2`: no run group.
	460	+ @no_hold %{waiting: nil, note: nil}
	461	+
459	462	defp create_run(%Repository{id: rid} = repo, %{jobs: jobs} = attrs) do
460	463	Repo.transaction(fn ->
461	464	number = next_run_number(rid)

@@ -475,7 +478,10 @@ defmodule GitGud.Workflows do

475	478	# Every leg exists before any is queued: queueing one can cancel
476	479	# another in its job-level group, and a run whose only jobs so far
477	480	# are cancelled would be settled as finished half-built.
478		− legs = Enum.flat_map(jobs, &create_job(run, &1, hold))
	481	+ contexts = fn -> Context.for_run(run, repo) end
	482	+ created = Enum.map(jobs, &create_job(run, &1, hold, contexts))
	483	+ legs = Enum.flat_map(created, &elem(&1, 0))
	484	+ bad_matrices = for {_legs, {:error, row, reason}} <- created, do: {row, reason}
479	485
480	486	# A pending commit status per job, so the SCM views show the run is
481	487	# in flight before the first claim lands.

@@ -485,12 +491,14 @@ defmodule GitGud.Workflows do

485	491
486	492	for %WorkflowJob{status: "queued"} = leg <- legs, do: enqueue_job(leg)
487	493
488		− {run, hold}
	494	+ {run, hold, bad_matrices}
489	495	end)
490	496	\|> case do
491		− {:ok, {run, hold}} ->
	497	+ {:ok, {run, hold, bad_matrices}} ->
492	498	broadcast_ci_change(run.id)
493	499
	500	+ for {row, reason} <- bad_matrices, do: fail_matrix(row, reason)
	501	+
494	502	# Cancelled once this run is committed, rather than from inside its
495	503	# transaction: `cancel/2` releases each one's group, and that
496	504	# should see this run.

@@ -630,13 +638,59 @@ defmodule GitGud.Workflows do

630	638	# matrix support on this side. Separate rows are what let two legs be
631	639	# claimed by two runners, which is the point when the axis is the
632	640	# machine's architecture.
633		− defp create_job(%WorkflowRun{} = run, %{} = job, hold) do
634		− for combination <- Matrix.expand(job[:strategy] \|\| job.raw["strategy"]) do
635		− create_job_leg(run, job, combination, hold)
	641	+ #
	642	+ # A matrix that is (or holds) an expression is expanded now, against the
	643	+ # run's contexts, unless the job has `needs` — it may read them, so it
	644	+ # gets one `blocked` placeholder row (`placeholder?/1`) that
	645	+ # `resolve_dependents/1` expands once they finish. A matrix that won't
	646	+ # expand gets one row too, `{:error, row, reason}`, which `create_run/2`
	647	+ # fails once the run is committed.
	648	+ defp create_job(%WorkflowRun{} = run, %{} = job, hold, contexts) do
	649	+ strategy = job[:strategy] \|\| job.raw["strategy"]
	650	+ dynamic? = Matrix.dynamic?(strategy)
	651	+
	652	+ if dynamic? and job.needs != [] do
	653	+ {[create_placeholder(run, job, hold)], :ok}
	654	+ else
	655	+ case Matrix.expand(strategy, if(dynamic?, do: contexts.(), else: %{})) do
	656	+ {:ok, combinations} ->
	657	+ {Enum.map(combinations, &create_job_leg(run, job, &1, hold)), :ok}
	658	+
	659	+ {:error, reason} ->
	660	+ row = create_placeholder(run, job, hold)
	661	+ {[row], {:error, row, reason}}
	662	+ end
636	663	end
637	664	end
638	665
639		− defp create_job_leg(%WorkflowRun{id: rid}, %{} = job, combination, hold) do
	666	+ # The job as one `blocked` row, its definition (and so its matrix
	667	+ # expression) whole.
	668	+ defp create_placeholder(run, job, hold),
	669	+ do: create_job_leg(run, job, %{}, hold, status: "blocked", definition: job.raw)
	670	+
	671	+ # A row standing for a matrix not yet expanded (or that failed to):
	672	+ # no combination of its own, an expression for a matrix.
	673	+ defp placeholder?(%WorkflowJob{matrix: nil, definition: definition}),
	674	+ do: Matrix.dynamic?(definition["strategy"])
	675	+
	676	+ defp placeholder?(_job), do: false
	677	+
	678	+ # The `Yaml` job a stored row was created from, to create its legs.
	679	+ defp job_spec(%WorkflowJob{} = row) do
	680	+ definition = row.definition \|\| %{}
	681	+
	682	+ %{
	683	+ id: row.job_id,
	684	+ name: definition["name"],
	685	+ runs_on: Map.get(definition, "runs-on", "ubuntu-latest"),
	686	+ strategy: definition["strategy"],
	687	+ needs: row.needs,
	688	+ steps: Map.get(definition, "steps", []),
	689	+ raw: definition
	690	+ }
	691	+ end
	692	+
	693	+ defp create_job_leg(%WorkflowRun{id: rid}, %{} = job, combination, hold, opts \\ []) do
640	694	# Jobs with unresolved `needs:` start in `blocked` so claim_next_job
641	695	# can never pick them up before their prerequisites complete.
642	696	# Promotion to `queued` happens in `resolve_dependents/1`. The others

@@ -644,6 +698,7 @@ defmodule GitGud.Workflows do

644	698	# `queued` (and `create_run/2` then decides their own group).
645	699	{initial_status, reason} =
646	700	cond do
	701	+ opts[:status] -> {opts[:status], hold.note}
647	702	job.needs != [] -> {"blocked", hold.note}
648	703	hold.waiting -> {"waiting", hold.waiting}
649	704	true -> {"queued", hold.note}

@@ -664,7 +719,7 @@ defmodule GitGud.Workflows do

664	719	runs_on: %{"labels" => labels},
665	720	needs: job.needs,
666	721	matrix: if(combination == %{}, do: nil, else: combination),
667		− definition: Matrix.pin_definition(job.raw, combination)
	722	+ definition: opts[:definition] \|\| Matrix.pin_definition(job.raw, combination)
668	723	})
669	724	\|> Ecto.Changeset.change(status: initial_status, status_reason: reason)
670	725	\|> Repo.insert!()

@@ -1017,14 +1072,18 @@ defmodule GitGud.Workflows do

1017	1072	[j],
1018	1073	fragment("(? -> 'labels') <@ to_jsonb(?::text[])", j.runs_on, ^match_labels)
1019	1074	)
	1075	+ \|> under_max_parallel()
1020	1076	\|> lock("FOR UPDATE SKIP LOCKED")
1021	1077	\|> Repo.one()
1022	1078
1023		− case job do
	1079	+ case job && max_parallel_slot?(job) do
1024	1080	nil ->
1025	1081	nil
1026	1082
1027		− job ->
	1083	+ false ->
	1084	+ nil
	1085	+
	1086	+ true ->
1028	1087	{:ok, claimed} = job \|> WorkflowJob.claim_changeset(rid) \|> Repo.update()
1029	1088	run = Repo.get!(WorkflowRun, claimed.workflow_run_id)
1030	1089

@@ -1043,7 +1102,9 @@ defmodule GitGud.Workflows do

1043	1102	end
1044	1103	end
1045	1104
1046		− defp do_claim_next_job(%Runner{id: rid} = runner, labels) do
	1105	+ # `passed_over` are jobs found at their `max-parallel` once locked
	1106	+ # (`max_parallel_slot?/1`), which the next attempt looks past.
	1107	+ defp do_claim_next_job(%Runner{id: rid} = runner, labels, passed_over \\ []) do
1047	1108	match_labels = match_labels(labels)
1048	1109
1049	1110	Repo.transaction(fn ->

@@ -1054,16 +1115,21 @@ defmodule GitGud.Workflows do

1054	1115	[j],
1055	1116	fragment("(? -> 'labels') <@ to_jsonb(?::text[])", j.runs_on, ^match_labels)
1056	1117	)
	1118	+ \|> where([j], j.id not in ^passed_over)
	1119	+ \|> under_max_parallel()
1057	1120	\|> order_by([j], asc: j.id)
1058	1121	\|> limit(1)
1059	1122	\|> lock("FOR UPDATE SKIP LOCKED")
1060	1123	\|> Repo.one()
1061	1124
1062		− case job do
	1125	+ case job && max_parallel_slot?(job) do
1063	1126	nil ->
1064	1127	nil
1065	1128
1066		− job ->
	1129	+ false ->
	1130	+ Repo.rollback({:passed_over, job.id})
	1131	+
	1132	+ true ->
1067	1133	{:ok, claimed} = job \|> WorkflowJob.claim_changeset(rid) \|> Repo.update()
1068	1134	# Promote the parent run to "running" on first claim.
1069	1135	run = Repo.get!(WorkflowRun, claimed.workflow_run_id)

@@ -1079,10 +1145,61 @@ defmodule GitGud.Workflows do

1079	1145	end)
1080	1146	\|> case do
1081	1147	{:ok, result} -> result
	1148	+ {:error, {:passed_over, id}} -> do_claim_next_job(runner, labels, [id \| passed_over])
1082	1149	err -> err
1083	1150	end
1084	1151	end
1085	1152
	1153	+ # `strategy.max-parallel` (`Matrix.max_parallel/1`, pinned onto every
	1154	+ # leg): a queued leg isn't claimable while that many legs of its job are
	1155	+ # running in its run.
	1156	+ defp under_max_parallel(query) do
	1157	+ where(
	1158	+ query,
	1159	+ [j],
	1160	+ fragment(
	1161	+ """
	1162	+ CASE WHEN (? #>> '{strategy,max-parallel}') ~ '^[1-9][0-9]{0,8}$'
	1163	+ THEN (SELECT count(*) FROM workflow_jobs AS r
	1164	+ WHERE r.workflow_run_id = ? AND r.job_id = ? AND r.status = 'running')
	1165	+ < (? #>> '{strategy,max-parallel}')::integer
	1166	+ ELSE true END
	1167	+ """,
	1168	+ j.definition,
	1169	+ j.workflow_run_id,
	1170	+ j.job_id,
	1171	+ j.definition
	1172	+ )
	1173	+ )
	1174	+ end
	1175	+
	1176	+ # The query's count is from before the row lock, so two runners can
	1177	+ # both see the last free slot. Legs with a limit are claimed one at a
	1178	+ # time per job (an advisory lock until the transaction ends), and the
	1179	+ # count re-read under it sees any claim committed meanwhile.
	1180	+ defp max_parallel_slot?(%WorkflowJob{} = job) do
	1181	+ case Matrix.max_parallel(strategy_of(job)) do
	1182	+ nil ->
	1183	+ true
	1184	+
	1185	+ limit ->
	1186	+ Repo.query!("SELECT pg_advisory_xact_lock(hashtext($1))", [
	1187	+ "workflow_job_max_parallel:#{job.workflow_run_id}:#{job.job_id}"
	1188	+ ])
	1189	+
	1190	+ running =
	1191	+ from(j in WorkflowJob,
	1192	+ where:
	1193	+ j.workflow_run_id == ^job.workflow_run_id and j.job_id == ^job.job_id and
	1194	+ j.status == "running",
	1195	+ select: count()
	1196	+ )
	1197	+ \|> Repo.one()
	1198	+
	1199	+ running < limit
	1200	+ end
	1201	+ end
	1202	+
1086	1203	# The label set a runner can satisfy, as JSON for the containment
1087	1204	# test. A job is claimable when every label its `runs-on` lists is
1088	1205	# one this runner carries — GitHub's rule, and the reason

@@ -1757,7 +1874,7 @@ defmodule GitGud.Workflows do

1757	1874	{:ok, _} = update_job_status(dep, "skipped")
1758	1875
1759	1876	{:queue, snapshot} ->
1760		− enqueue_job(dep, needs_snapshot: snapshot)
	1877	+ promote(dep, snapshot)
1761	1878
1762	1879	_blocked_or_decided ->
1763	1880	:ok

@@ -1767,6 +1884,74 @@ defmodule GitGud.Workflows do

1767	1884	:ok
1768	1885	end
1769	1886
	1887	+ # Queue a job whose needs let it run — or, for a matrix placeholder
	1888	+ # (`create_job/4`), expand it now that `needs` can be read and queue its
	1889	+ # legs in its place.
	1890	+ defp promote(%WorkflowJob{} = job, snapshot) do
	1891	+ if placeholder?(job),
	1892	+ do: expand_placeholder(job, snapshot),
	1893	+ else: enqueue_job(job, needs_snapshot: snapshot)
	1894	+ end
	1895	+
	1896	+ # The legs replace the placeholder in one transaction: they keep its
	1897	+ # `job_id` (so `needs: build` downstream finds them) and get the same
	1898	+ # `needs_snapshot`. Its pending commit status goes with it, since the
	1899	+ # legs report under their own names. Re-read under a lock, so two needs
	1900	+ # finishing at once expand it once.
	1901	+ defp expand_placeholder(%WorkflowJob{} = placeholder, snapshot) do
	1902	+ run = Repo.get!(WorkflowRun, placeholder.workflow_run_id)
	1903	+ repo = Repositories.get_repository!(run.repository_id)
	1904	+ contexts = Context.for_job(%{placeholder \| needs_snapshot: snapshot}, run, repo)
	1905	+ spec = job_spec(placeholder)
	1906	+
	1907	+ case Matrix.expand(spec.strategy, contexts) do
	1908	+ {:ok, combinations} ->
	1909	+ Repo.transaction(fn ->
	1910	+ current =
	1911	+ from(j in WorkflowJob, where: j.id == ^placeholder.id and j.status == "blocked")
	1912	+ \|> lock("FOR UPDATE")
	1913	+ \|> Repo.one()
	1914	+
	1915	+ if is_nil(current), do: Repo.rollback(:already_expanded)
	1916	+
	1917	+ legs =
	1918	+ for combination <- combinations,
	1919	+ do: create_job_leg(run, spec, combination, @no_hold, status: "blocked")
	1920	+
	1921	+ drop_job_commit_status(placeholder, run)
	1922	+
	1923	+ # Its steps (and their logs) go with it, by foreign key.
	1924	+ Repo.delete_all(from j in WorkflowJob, where: j.id == ^placeholder.id)
	1925	+
	1926	+ Enum.map(legs, &enqueue_job(&1, needs_snapshot: snapshot))
	1927	+ end)
	1928	+ \|> case do
	1929	+ {:ok, legs} ->
	1930	+ # Again now they're committed: the page re-reads the run on
	1931	+ # each, and one sent from inside the transaction can read it
	1932	+ # before the placeholder is gone.
	1933	+ for leg <- legs, do: broadcast({:job_status, run.id, leg})
	1934	+ broadcast_ci_change(run.id)
	1935	+ legs
	1936	+
	1937	+ {:error, :already_expanded} ->
	1938	+ []
	1939	+ end
	1940	+
	1941	+ {:error, reason} ->
	1942	+ fail_matrix(placeholder, reason)
	1943	+ end
	1944	+ end
	1945	+
	1946	+ # A matrix that wouldn't expand fails its job, saying why in its log,
	1947	+ # and so the run and the job's dependents, the usual way.
	1948	+ defp fail_matrix(%WorkflowJob{} = row, reason) do
	1949	+ message = Matrix.describe_error(reason)
	1950	+ Logger.warning("workflow job #{row.id} (#{row.name}): #{message}")
	1951	+ :ok = append_log(ensure_log_step(row), "Error: " <> message)
	1952	+ update_job_status(row, "failure", reason: message)
	1953	+ end
	1954	+
1770	1955	# What `dep`'s needs, as they stand in `by_jid`, make of it: `:blocked`
1771	1956	# while one is in flight, then `{:queue, snapshot}`, `:skip` or
1772	1957	# `:cancel` (see `resolve_dependents/1`).

@@ -2013,6 +2198,22 @@ defmodule GitGud.Workflows do

2013	2198	)
2014	2199	end
2015	2200
	2201	+ # A job that no longer exists — a matrix placeholder its legs replaced.
	2202	+ defp drop_job_commit_status(_job, %WorkflowRun{head_sha: nil}), do: :ok
	2203	+
	2204	+ defp drop_job_commit_status(%WorkflowJob{} = job, %WorkflowRun{} = run) do
	2205	+ context = job_status_context(job, run)
	2206	+
	2207	+ from(c in GitGud.Workflows.CommitStatus,
	2208	+ where:
	2209	+ c.repository_id == ^run.repository_id and c.sha == ^run.head_sha and
	2210	+ c.context == ^context
	2211	+ )
	2212	+ \|> Repo.delete_all()
	2213	+
	2214	+ :ok
	2215	+ end
	2216	+
2016	2217	# The one status per workflow written before statuses were per job, so
2017	2218	# a re-run on such a commit doesn't leave it beside the jobs' (where a
2018	2219	# rule requiring every status to pass would never get past it).

@@ -2199,7 +2400,10 @@ defmodule GitGud.Workflows do

2199	2400	)
2200	2401	\|> Repo.update()
2201	2402
2202		− case verdict.(job) do
	2403	+ # A matrix placeholder is expanded once this is committed,
	2404	+ # below: expanding inside it could fail the job, and settle
	2405	+ # the run, before the run is flipped back to `running`.
	2406	+ case not placeholder?(row) && verdict.(job) do
2203	2407	{:queue, snapshot} -> enqueue_job(row, needs_snapshot: snapshot)
2204	2408	_ -> row
2205	2409	end

@@ -2225,6 +2429,7 @@ defmodule GitGud.Workflows do

2225	2429	case verdict.(row) do
2226	2430	:skip -> {:ok, _} = update_job_status(row, "skipped")
2227	2431	:cancel -> {:ok, _} = update_job_status(row, "cancelled")
	2432	+ {:queue, snapshot} -> promote(row, snapshot)
2228	2433	_ -> :ok
2229	2434	end
2230	2435

modified lib/git_gud/workflows/expr.ex

+35 −2

@@ -7,7 +7,9 @@ defmodule GitGud.Workflows.Expr do

7	7	`concurrency:` groups — and those are nearly always built from a few
8	8	context values: `${{ github.workflow }}-${{ github.ref }}`,
9	9	`${{ github.event.pull_request.number \|\| github.ref }}`,
10		− `${{ matrix.target }}`. So this covers that and no more:
	10	+ `${{ matrix.target }}` — and a `strategy.matrix` computed by an earlier
	11	+ job, `${{ fromJSON(needs.setup.outputs.matrix) }}`. So this covers that
	12	+ and no more:
11	13
12	14	* property paths into `contexts` — `github.ref`, `inputs['env']`,
13	15	`needs.build.outputs.x` — case-insensitively, as GitHub does; a path

@@ -15,7 +17,7 @@ defmodule GitGud.Workflows.Expr do

15	17	* `'string'` (`''` escapes a quote), number, `true`, `false`, `null`
16	18	* `\|\|`, `&&` (returning an operand, not a boolean), `==`, `!=`, `!`,
17	19	parentheses
18		− * `format('{0}-{1}', …)`
	20	+ * `format('{0}-{1}', …)`, `fromJSON(string)`, `toJSON(value)`
19	21
20	22	Anything else — another function, a comparison, `*` filters, a context
21	23	that wasn't supplied — is `:error`, and the caller acts as though the

@@ -59,6 +61,28 @@ defmodule GitGud.Workflows.Expr do

59	61	:error -> :error
60	62	end
61	63
	64	+ @doc """
	65	+ The value of a string that is exactly one `${{ … }}` — not stringified,
	66	+ so `${{ fromJSON('[1, 2]') }}` is a list — else `interpolate/2`'s
	67	+ string. What a workflow field that may hold an expression means.
	68	+ """
	69	+ @spec resolve(String.t(), map()) :: {:ok, value()} \| :error
	70	+ def resolve(string, contexts) when is_binary(string) do
	71	+ with "${{" <> after_open <- String.trim(string),
	72	+ {expression, rest} <- take_whole_expression(after_open),
	73	+ "" <- String.trim(rest) do
	74	+ evaluate(expression, contexts)
	75	+ else
	76	+ _ -> interpolate(string, contexts)
	77	+ end
	78	+ end
	79	+
	80	+ defp take_whole_expression(after_open) do
	81	+ take_expression(after_open, [])
	82	+ catch
	83	+ :error -> :error
	84	+ end
	85	+
62	86	@doc "GitHub's truthiness: `false`, `0`, `\"\"` and `null` are false."
63	87	def truthy?(value), do: value not in [false, nil, 0, +0.0, -0.0, ""]
64	88

@@ -280,6 +304,15 @@ defmodule GitGud.Workflows.Expr do

280	304	defp call("format", [template \| args]) when is_binary(template),
281	305	do: format(template, Enum.map(args, &stringify/1), [])
282	306
	307	+ defp call("fromjson", [json]) when is_binary(json) do
	308	+ case Jason.decode(json) do
	309	+ {:ok, value} -> value
	310	+ {:error, _} -> throw(:error)
	311	+ end
	312	+ end
	313	+
	314	+ defp call("tojson", [value]), do: Jason.encode!(value, pretty: true)
	315	+
283	316	defp call(_function, _args), do: throw(:error)
284	317
285	318	defp format("", _args, acc), do: acc \|> Enum.reverse() \|> IO.iodata_to_binary()

modified lib/git_gud/workflows/matrix.ex

+172 −35

@@ -40,52 +40,106 @@ defmodule GitGud.Workflows.Matrix do

40	40	know beats an arbitrary choice.
41	41	"""
42	42
	43	+ alias GitGud.Workflows.Expr
	44	+
43	45	@reserved ~w(include exclude)
44	46
45	47	@typedoc "One combination: matrix key to the value this leg was expanded for."
46	48	@type combination :: %{String.t() => term()}
47	49
48	50	@doc """
49		− Every combination `strategy` expands to.
	51	+ Every combination `strategy` expands to, as `{:ok, combinations}`.
	52	+
	53	+ Returns `{:ok, [%{}]}` — a single empty combination — when there is no
	54	+ matrix (or one with no axes and no `include`), so callers can treat
	55	+ "no matrix" as "one leg with nothing in it" rather than branching.
50	56
51		− Returns `[%{}]` — a single empty combination — when there is no
52		− matrix, so callers can treat "no matrix" as "one leg with nothing in
53		− it" rather than branching.
	57	+ Expressions are resolved first, against `contexts` (`GitGud.Workflows.Expr`):
	58	+ the whole matrix (`matrix: ${{ fromJSON(needs.setup.outputs.matrix) }}`,
	59	+ which must yield an object), an axis or an `include` / `exclude`
	60	+ (which must yield an array), an entry of those (an object), or a value.
	61	+
	62	+ `{:error, reason}` when an expression can't be evaluated or yields the
	63	+ wrong shape, or — as on GitHub, where it fails the workflow — when
	64	+ `exclude` leaves nothing (`:empty_matrix`). `describe_error/1` words it.
54	65	"""
55		− @spec expand(map() \| nil) :: [combination()]
56		− def expand(strategy)
57		−
58		− def expand(%{"matrix" => %{} = matrix}) do
59		− axes =
60		− matrix
61		− \|> Map.drop(@reserved)
62		− \|> Enum.map(fn {key, values} -> {key, List.wrap(values)} end)
63		− # Map iteration order is undefined; sort so a given workflow always
64		− # expands the same way rather than shuffling between deploys.
65		− \|> Enum.sort_by(&elem(&1, 0))
66		−
67		− axes
68		− \|> product()
69		− \|> exclude(List.wrap(matrix["exclude"]))
70		− \|> include(List.wrap(matrix["include"]))
71		− \|> case do
72		− [] -> [%{}]
73		− combinations -> combinations
	66	+ @spec expand(map() \| nil, map()) :: {:ok, [combination()]} \| {:error, term()}
	67	+ def expand(strategy, contexts \\ %{})
	68	+
	69	+ def expand(%{"matrix" => matrix}, contexts) when is_binary(matrix) or is_map(matrix) do
	70	+ with {:ok, matrix} <- resolve_matrix(matrix, contexts),
	71	+ {:ok, axes} <- resolve_axes(matrix, contexts),
	72	+ {:ok, exclusions} <- resolve_entries(matrix, "exclude", contexts),
	73	+ {:ok, inclusions} <- resolve_entries(matrix, "include", contexts) do
	74	+ if axes == [] and inclusions == [] do
	75	+ {:ok, [%{}]}
	76	+ else
	77	+ axes
	78	+ \|> product()
	79	+ \|> exclude(exclusions)
	80	+ \|> include(inclusions)
	81	+ \|> case do
	82	+ [] -> {:error, :empty_matrix}
	83	+ combinations -> {:ok, combinations}
	84	+ end
	85	+ end
74	86	end
75	87	end
76	88
77		− def expand(_strategy), do: [%{}]
	89	+ def expand(_strategy, _contexts), do: {:ok, [%{}]}
	90	+
	91	+ @doc "The log line for an `expand/2` error."
	92	+ @spec describe_error(term()) :: String.t()
	93	+ def describe_error(:empty_matrix),
	94	+ do: "The matrix has no combinations: exclude removed every one, or an axis is empty."
	95	+
	96	+ def describe_error({:expression, text}),
	97	+ do: "The matrix expression #{text} could not be evaluated."
	98	+
	99	+ def describe_error({:not_object, what}), do: "The matrix #{what} is not an object."
	100	+ def describe_error({:not_list, what}), do: "The matrix #{what} is not an array."
78	101
79	102	@doc """
80	103	Whether `strategy` asks for a matrix at all.
81	104
82	105	A `strategy:` with only `fail-fast:` on it is not a matrix, and its
83		− job should not be renamed as though it were one leg of something.
	106	+ job should not be renamed as though it were one leg of something. A
	107	+ matrix that is an expression is one, whatever it turns out to hold.
84	108	"""
85	109	@spec matrix?(map() \| nil) :: boolean()
86	110	def matrix?(%{"matrix" => %{} = matrix}), do: matrix != %{}
	111	+ def matrix?(%{"matrix" => matrix}) when is_binary(matrix), do: true
87	112	def matrix?(_), do: false
88	113
	114	+ @doc """
	115	+ Whether the matrix holds an expression anywhere — so it can only be
	116	+ expanded once its contexts exist (`needs`, when it reads them).
	117	+ """
	118	+ @spec dynamic?(map() \| nil) :: boolean()
	119	+ def dynamic?(%{"matrix" => matrix}), do: expression?(matrix)
	120	+ def dynamic?(_), do: false
	121	+
	122	+ defp expression?(value) when is_binary(value), do: String.contains?(value, "${{")
	123	+ defp expression?(%{} = map), do: Enum.any?(map, fn {_key, value} -> expression?(value) end)
	124	+ defp expression?(list) when is_list(list), do: Enum.any?(list, &expression?/1)
	125	+ defp expression?(_value), do: false
	126	+
	127	+ @doc """
	128	+ `strategy.max-parallel`: how many legs of the job may run at once, or
	129	+ nil for no limit. A positive integer (or its digits as a string, which
	130	+ is how a quoted YAML value arrives); anything else, an expression
	131	+ included, is no limit. Claiming reads the same rule in SQL
	132	+ (`Workflows.do_claim_next_job/3`).
	133	+ """
	134	+ @spec max_parallel(map() \| nil) :: pos_integer() \| nil
	135	+ def max_parallel(%{"max-parallel" => n}) when is_integer(n) and n > 0, do: n
	136	+
	137	+ def max_parallel(%{"max-parallel" => n}) when is_binary(n) do
	138	+ if Regex.match?(~r/\A[1-9][0-9]{0,8}\z/, n), do: String.to_integer(n)
	139	+ end
	140	+
	141	+ def max_parallel(_strategy), do: nil
	142	+
89	143	@doc """
90	144	Whether a failing leg should cancel its siblings. GitHub's default is
91	145	`true`, and so is ours.

@@ -169,21 +223,106 @@ defmodule GitGud.Workflows.Matrix do

169	223	pinned = Map.new(combination, fn {key, value} -> {key, [value]} end)
170	224	strategy = Map.get(definition, "strategy") \|\| %{}
171	225
172		− # `fail-fast` rides along even though it means nothing to a
173		− # one-value matrix: the forge reads it back off the stored
174		− # definition to decide whether a failing leg cancels its siblings,
175		− # and the pinned strategy is the only copy that survives.
	226	+ # `fail-fast` and `max-parallel` ride along even though they mean
	227	+ # nothing to a one-value matrix: the forge reads them back off the
	228	+ # stored definition — to decide whether a failing leg cancels its
	229	+ # siblings, and how many legs may be claimed at once — and the
	230	+ # pinned strategy is the only copy that survives.
176	231	kept =
177		− case Map.fetch(strategy, "fail-fast") do
178		− {:ok, value} -> %{"matrix" => pinned, "fail-fast" => value}
179		− :error -> %{"matrix" => pinned}
180		− end
	232	+ strategy
	233	+ \|> Map.take(["fail-fast", "max-parallel"])
	234	+ \|> Map.put("matrix", pinned)
181	235
182	236	Map.put(definition, "strategy", kept)
183	237	end
184	238
185	239	# ── expansion internals ─────────────────────────────────────────────
186	240
	241	+ # The matrix itself as a map: an expression must yield one.
	242	+ defp resolve_matrix(matrix, contexts) when is_binary(matrix) do
	243	+ case evaluate(matrix, contexts) do
	244	+ {:ok, %{} = resolved} -> {:ok, resolved}
	245	+ {:ok, _other} -> {:error, {:not_object, matrix}}
	246	+ error -> error
	247	+ end
	248	+ end
	249	+
	250	+ defp resolve_matrix(matrix, _contexts), do: {:ok, matrix}
	251	+
	252	+ # `[{key, values}]`, sorted. An axis that is an expression must yield
	253	+ # an array; a literal scalar is a one-value axis, as it always was.
	254	+ defp resolve_axes(matrix, contexts) do
	255	+ matrix
	256	+ \|> Map.drop(@reserved)
	257	+ # Map iteration order is undefined; sort so a given workflow always
	258	+ # expands the same way rather than shuffling between deploys.
	259	+ \|> Enum.sort_by(&elem(&1, 0))
	260	+ \|> map_ok(fn {key, values} ->
	261	+ case resolve_value(values, contexts) do
	262	+ {:ok, list} when is_list(list) -> {:ok, {key, list}}
	263	+ {:ok, _other} when is_binary(values) -> not_list(values, key)
	264	+ {:ok, other} -> {:ok, {key, List.wrap(other)}}
	265	+ error -> error
	266	+ end
	267	+ end)
	268	+ end
	269	+
	270	+ defp not_list(values, key) do
	271	+ if expression?(values),
	272	+ do: {:error, {:not_list, "axis #{key}"}},
	273	+ else: {:ok, {key, [values]}}
	274	+ end
	275	+
	276	+ # `include` / `exclude`: a list of objects, each of which (or the whole
	277	+ # list) may be an expression.
	278	+ defp resolve_entries(matrix, key, contexts) do
	279	+ with {:ok, entries} <- resolve_value(Map.get(matrix, key), contexts) do
	280	+ entries
	281	+ \|> List.wrap()
	282	+ \|> map_ok(fn
	283	+ %{} = entry -> {:ok, Map.new(entry, fn {k, v} -> {to_string(k), v} end)}
	284	+ _other -> {:error, {:not_object, "#{key} entry"}}
	285	+ end)
	286	+ end
	287	+ end
	288	+
	289	+ defp resolve_value(value, contexts) when is_binary(value) do
	290	+ if String.contains?(value, "${{"), do: evaluate(value, contexts), else: {:ok, value}
	291	+ end
	292	+
	293	+ defp resolve_value(%{} = map, contexts) do
	294	+ with {:ok, pairs} <-
	295	+ map_ok(map, fn {key, value} ->
	296	+ with {:ok, value} <- resolve_value(value, contexts), do: {:ok, {key, value}}
	297	+ end),
	298	+ do: {:ok, Map.new(pairs)}
	299	+ end
	300	+
	301	+ defp resolve_value(list, contexts) when is_list(list),
	302	+ do: map_ok(list, &resolve_value(&1, contexts))
	303	+
	304	+ defp resolve_value(value, _contexts), do: {:ok, value}
	305	+
	306	+ defp evaluate(text, contexts) do
	307	+ case Expr.resolve(text, contexts) do
	308	+ {:ok, value} -> {:ok, value}
	309	+ :error -> {:error, {:expression, text}}
	310	+ end
	311	+ end
	312	+
	313	+ defp map_ok(enumerable, fun) do
	314	+ Enum.reduce_while(enumerable, {:ok, []}, fn item, {:ok, acc} ->
	315	+ case fun.(item) do
	316	+ {:ok, value} -> {:cont, {:ok, [value \| acc]}}
	317	+ error -> {:halt, error}
	318	+ end
	319	+ end)
	320	+ \|> case do
	321	+ {:ok, acc} -> {:ok, Enum.reverse(acc)}
	322	+ error -> error
	323	+ end
	324	+ end
	325	+
187	326	# Cartesian product with the last axis varying fastest, matching the
188	327	# order Actions lists a matrix's jobs in.
189	328	defp product([]), do: []

@@ -206,8 +345,6 @@ defmodule GitGud.Workflows.Matrix do

206	345
207	346	defp include(combinations, inclusions) do
208	347	Enum.reduce(inclusions, combinations, fn entry, acc ->
209		− entry = Map.new(entry, fn {k, v} -> {to_string(k), v} end)
210		−
211	348	merged = Enum.map(acc, &merge_if_compatible(&1, entry))
212	349
213	350	# Nothing took it — it describes a combination of its own.

modified lib/git_gud_web/live/workflow_live/show.ex

+8 −3

@@ -62,10 +62,14 @@ defmodule GitGudWeb.WorkflowLive.Show do

62	62
63	63	defp admin?(_, _), do: false
64	64
	65	+ # Also for jobs that appear later — the legs a matrix expands into once
	66	+ # its `needs` finish — so their logs have a stream to land in.
65	67	defp seed_step_streams(socket, run) do
66	68	Enum.reduce(run.jobs, socket, fn job, acc ->
67	69	Enum.reduce(job.steps, acc, fn step, acc2 ->
68		− stream(acc2, stream_name(step.id), [], dom_id: &"log-#{&1.seq}")
	70	+ if Map.has_key?(Map.get(acc2.assigns, :streams, %{}), stream_name(step.id)),
	71	+ do: acc2,
	72	+ else: stream(acc2, stream_name(step.id), [], dom_id: &"log-#{&1.seq}")
69	73	end)
70	74	end)
71	75	end

@@ -119,6 +123,7 @@ defmodule GitGudWeb.WorkflowLive.Show do

119	123	{:noreply,
120	124	socket
121	125	\|> assign(:run, run)
	126	+ \|> seed_step_streams(run)
122	127	\|> assign(:artifacts, Workflows.list_artifacts(run))
123	128	\|> ensure_ticking()}
124	129	end

@@ -406,8 +411,8 @@ defmodule GitGudWeb.WorkflowLive.Show do

406	411	<span class="font-mono">{a.name}</span>
407	412	<span class="text-xs opacity-60">
408	413	{format_size(a.size)}
409		− <span :if={a.content_type}> · {a.content_type}</span>
410		− <span :if={a.inserted_at}> · {format_dt(a.inserted_at)}</span>
	414	+ <span :if={a.content_type}> · {a.content_type}</span>
	415	+ <span :if={a.inserted_at}> · {format_dt(a.inserted_at)}</span>
411	416	</span>
412	417	</div>
413	418	<a

modified test/git_gud/workflows/expr_test.exs

+30 −1

@@ -94,7 +94,7 @@ defmodule GitGud.Workflows.ExprTest do

94	94
95	95	describe "what isn't supported is an error, not a guess" do
96	96	test "other functions" do
97		− assert i("${{ toJSON(github) }}") == :error
	97	+ assert i("${{ startsWith(github.ref, 'refs/') }}") == :error
98	98	assert i("${{ contains(github.ref, 'main') }}") == :error
99	99	end
100	100

@@ -113,4 +113,33 @@ defmodule GitGud.Workflows.ExprTest do

113	113	assert i("${{ github.event }}") == :error
114	114	end
115	115	end
	116	+
	117	+ describe "a matrix computed by an earlier job" do
	118	+ defp r(string, contexts \\ @contexts), do: Expr.resolve(string, contexts)
	119	+
	120	+ test "fromJSON yields the value, not its text" do
	121	+ contexts = put_in(@contexts, ["needs", "build", "outputs", "m"], ~s({"os":["a","b"]}))
	122	+
	123	+ assert r("${{ fromJSON(needs.build.outputs.m) }}", contexts) ==
	124	+ {:ok, %{"os" => ["a", "b"]}}
	125	+
	126	+ assert r(" ${{ fromJSON('[1, 2]') }} ") == {:ok, [1, 2]}
	127	+ end
	128	+
	129	+ test "bad JSON is an error" do
	130	+ assert r("${{ fromJSON('nope') }}") == :error
	131	+ assert r("${{ fromJSON(needs.build.outputs.missing) }}") == :error
	132	+ end
	133	+
	134	+ test "toJSON is the value's JSON" do
	135	+ assert {:ok, json} = r("${{ toJSON(matrix) }}")
	136	+ assert Jason.decode!(json) == %{"target" => "arm64", "n" => 2}
	137	+ end
	138	+
	139	+ test "anything but one whole expression is a string" do
	140	+ assert r("${{ matrix.target }}-${{ matrix.n }}") == {:ok, "arm64-2"}
	141	+ assert r("plain") == {:ok, "plain"}
	142	+ assert r("${{ format('{{0}}') }}") == {:ok, "{0}"}
	143	+ end
	144	+ end
116	145	end

modified test/git_gud/workflows/matrix_dispatch_test.exs

+220 −0

@@ -507,4 +507,224 @@ defmodule GitGud.Workflows.MatrixDispatchTest do

507	507	assert Repo.get!(WorkflowRun, run.id).status == "success"
508	508	end
509	509	end
	510	+
	511	+ describe "a matrix computed by an earlier job" do
	512	+ # The failures log a warning, as they should.
	513	+ @describetag :capture_log
	514	+
	515	+ setup %{repo: repo} do
	516	+ run =
	517	+ run_workflow(repo, """
	518	+ name: ci
	519	+ on: push
	520	+ jobs:
	521	+ setup:
	522	+ runs-on: self-hosted
	523	+ steps:
	524	+ - run: true
	525	+ build:
	526	+ needs: setup
	527	+ strategy:
	528	+ matrix: ${{ fromJSON(needs.setup.outputs.m) }}
	529	+ runs-on: self-hosted
	530	+ steps:
	531	+ - run: echo ${{ matrix.os }}
	532	+ deploy:
	533	+ needs: build
	534	+ runs-on: self-hosted
	535	+ steps:
	536	+ - run: true
	537	+ """)
	538	+
	539	+ {:ok, run: run}
	540	+ end
	541	+
	542	+ defp finish_setup(run, outputs) do
	543	+ setup = Enum.find(jobs(run), &(&1.job_id == "setup"))
	544	+ {:ok, setup} = Workflows.set_outputs(setup, outputs)
	545	+ {:ok, _} = Workflows.update_job_status(setup, "success")
	546	+ end
	547	+
	548	+ test "waits as one placeholder, then becomes a leg per combination", %{run: run} do
	549	+ assert [placeholder] = Enum.filter(jobs(run), &(&1.job_id == "build"))
	550	+ assert placeholder.name == "build"
	551	+ assert placeholder.status == "blocked"
	552	+ assert placeholder.matrix == nil
	553	+
	554	+ finish_setup(run, %{"m" => ~s({"os":["a","b"]})})
	555	+
	556	+ refute Repo.get(WorkflowJob, placeholder.id)
	557	+
	558	+ legs = Enum.filter(jobs(run), &(&1.job_id == "build"))
	559	+ assert Enum.map(legs, & &1.matrix) == [%{"os" => "a"}, %{"os" => "b"}]
	560	+ assert Enum.map(legs, & &1.name) == ["build (a)", "build (b)"]
	561	+ assert Enum.all?(legs, &(&1.status == "queued"))
	562	+ assert Enum.all?(legs, &(&1.needs_snapshot["setup"]["status"] == "success"))
	563	+
	564	+ assert Enum.all?(
	565	+ legs,
	566	+ &(&1.definition["strategy"]["matrix"] == %{"os" => [&1.matrix["os"]]})
	567	+ )
	568	+
	569	+ assert Enum.all?(legs, &(length(Repo.preload(&1, :steps).steps) == 1))
	570	+
	571	+ # The placeholder's pending status went with it; each leg has its own.
	572	+ contexts =
	573	+ run.repository_id
	574	+ \|> then(
	575	+ &Workflows.list_commit_statuses(%GitGud.Repositories.Repository{id: &1}, run.head_sha)
	576	+ )
	577	+ \|> Enum.map(& &1.context)
	578	+
	579	+ refute "ci / build (push)" in contexts
	580	+ assert "ci / build (a) (push)" in contexts
	581	+ end
	582	+
	583	+ test "a job needing it waits for every leg", %{run: run} do
	584	+ finish_setup(run, %{"m" => ~s({"os":["a","b"]})})
	585	+
	586	+ [a, b] = Enum.filter(jobs(run), &(&1.job_id == "build"))
	587	+ deploy = Enum.find(jobs(run), &(&1.job_id == "deploy"))
	588	+
	589	+ {:ok, _} = Workflows.update_job_status(a, "success")
	590	+ assert Repo.get!(WorkflowJob, deploy.id).status == "blocked"
	591	+
	592	+ {:ok, _} = Workflows.update_job_status(b, "success")
	593	+ assert Repo.get!(WorkflowJob, deploy.id).status == "queued"
	594	+ end
	595	+
	596	+ test "an output that isn't a matrix fails the job, saying why", %{run: run} do
	597	+ finish_setup(run, %{"m" => "nope"})
	598	+
	599	+ [build] = Enum.filter(jobs(run), &(&1.job_id == "build"))
	600	+ assert build.status == "failure"
	601	+ assert build.status_reason =~ "could not be evaluated"
	602	+
	603	+ [step \| _] =
	604	+ build \|> Repo.preload(:steps) \|> Map.fetch!(:steps) \|> Enum.sort_by(& &1.number)
	605	+
	606	+ assert [%{body: "Error: " <> _}] = Workflows.list_step_logs(step)
	607	+
	608	+ assert Enum.find(jobs(run), &(&1.job_id == "deploy")).status == "skipped"
	609	+ assert Repo.get!(WorkflowRun, run.id).status == "failure"
	610	+ end
	611	+
	612	+ test "re-running the failed job expands it again", %{run: run} do
	613	+ finish_setup(run, %{"m" => "nope"})
	614	+
	615	+ setup = Enum.find(jobs(run), &(&1.job_id == "setup"))
	616	+ {:ok, _} = Workflows.set_outputs(setup, %{"m" => ~s({"os":["a"]})})
	617	+ {:ok, _} = Workflows.rerun_job(Repo.get!(WorkflowRun, run.id), "build")
	618	+
	619	+ assert [%{matrix: %{"os" => "a"}, status: "queued"}] =
	620	+ Enum.filter(jobs(run), &(&1.job_id == "build"))
	621	+
	622	+ assert Repo.get!(WorkflowRun, run.id).status == "running"
	623	+ end
	624	+ end
	625	+
	626	+ describe "a matrix computed without needs" do
	627	+ test "is expanded as the run is created", %{repo: repo} do
	628	+ run =
	629	+ run_workflow(repo, """
	630	+ name: ci
	631	+ on: push
	632	+ jobs:
	633	+ build:
	634	+ strategy:
	635	+ matrix:
	636	+ os: ${{ fromJSON('["a", "b"]') }}
	637	+ ref: ["${{ github.ref_name }}"]
	638	+ runs-on: self-hosted
	639	+ steps:
	640	+ - run: true
	641	+ """)
	642	+
	643	+ assert Enum.map(jobs(run), & &1.matrix) == [
	644	+ %{"os" => "a", "ref" => "main"},
	645	+ %{"os" => "b", "ref" => "main"}
	646	+ ]
	647	+ end
	648	+
	649	+ @tag :capture_log
	650	+ test "one that excludes everything fails the run", %{repo: repo} do
	651	+ run =
	652	+ run_workflow(repo, """
	653	+ name: ci
	654	+ on: push
	655	+ jobs:
	656	+ build:
	657	+ strategy:
	658	+ matrix:
	659	+ os: [a]
	660	+ exclude:
	661	+ - os: a
	662	+ runs-on: self-hosted
	663	+ steps:
	664	+ - run: true
	665	+ """)
	666	+
	667	+ assert [%{status: "failure", status_reason: reason}] = jobs(run)
	668	+ assert reason =~ "no combinations"
	669	+ assert Repo.get!(WorkflowRun, run.id).status == "failure"
	670	+ end
	671	+ end
	672	+
	673	+ describe "max-parallel" do
	674	+ test "caps the legs of a job running at once", %{repo: repo} do
	675	+ run =
	676	+ run_workflow(repo, """
	677	+ name: ci
	678	+ on: push
	679	+ jobs:
	680	+ deploy:
	681	+ strategy:
	682	+ max-parallel: 1
	683	+ matrix:
	684	+ env: [a, b, c]
	685	+ runs-on: self-hosted
	686	+ steps:
	687	+ - run: true
	688	+ """)
	689	+
	690	+ [r1, r2, r3] = for _ <- 1..3, do: runner(repo, ["self-hosted"])
	691	+
	692	+ assert %WorkflowJob{} = first = Workflows.claim_next_job(r1)
	693	+ assert Workflows.claim_next_job(r2) == nil
	694	+ assert Workflows.claim_jobs_for(r3, 3) == []
	695	+
	696	+ [_, second, _] = jobs(run)
	697	+ assert Workflows.claim_job_by_handle(r2, Integer.to_string(second.id)) == nil
	698	+
	699	+ {:ok, _} = Workflows.update_job_status(first, "success")
	700	+
	701	+ assert %WorkflowJob{} = Workflows.claim_next_job(r2)
	702	+ assert Workflows.claim_next_job(r3) == nil
	703	+ end
	704	+
	705	+ test "another job's legs are still claimable meanwhile", %{repo: repo} do
	706	+ run_workflow(repo, """
	707	+ name: ci
	708	+ on: push
	709	+ jobs:
	710	+ deploy:
	711	+ strategy:
	712	+ max-parallel: 1
	713	+ matrix:
	714	+ env: [a, b]
	715	+ runs-on: self-hosted
	716	+ steps:
	717	+ - run: true
	718	+ test:
	719	+ runs-on: self-hosted
	720	+ steps:
	721	+ - run: true
	722	+ """)
	723	+
	724	+ r = runner(repo, ["self-hosted"])
	725	+ claimed = Workflows.claim_jobs_for(r, 3)
	726	+
	727	+ assert claimed \|> Enum.map(& &1.job_id) \|> Enum.sort() == ["deploy", "test"]
	728	+ end
	729	+ end
510	730	end

modified test/git_gud/workflows/matrix_test.exs

+132 −14

@@ -11,15 +11,21 @@ defmodule GitGud.Workflows.MatrixTest do

11	11
12	12	defp strategy(matrix), do: %{"matrix" => matrix}
13	13
	14	+ defp expand!(strategy, contexts \\ %{}) do
	15	+ {:ok, combinations} = Matrix.expand(strategy, contexts)
	16	+ combinations
	17	+ end
	18	+
14	19	describe "expand/1" do
15	20	test "no strategy is one leg with nothing in it" do
16		− assert Matrix.expand(nil) == [%{}]
17		− assert Matrix.expand(%{}) == [%{}]
18		− assert Matrix.expand(%{"fail-fast" => false}) == [%{}]
	21	+ assert Matrix.expand(nil) == {:ok, [%{}]}
	22	+ assert Matrix.expand(%{}) == {:ok, [%{}]}
	23	+ assert Matrix.expand(%{"fail-fast" => false}) == {:ok, [%{}]}
	24	+ assert Matrix.expand(strategy(%{})) == {:ok, [%{}]}
19	25	end
20	26
21	27	test "one axis is one job per value" do
22		− assert Matrix.expand(strategy(%{"arch" => ["amd64", "arm64"]})) == [
	28	+ assert expand!(strategy(%{"arch" => ["amd64", "arm64"]})) == [
23	29	%{"arch" => "amd64"},
24	30	%{"arch" => "arm64"}
25	31	]

@@ -27,7 +33,7 @@ defmodule GitGud.Workflows.MatrixTest do

27	33
28	34	test "two axes are their product" do
29	35	combinations =
30		− Matrix.expand(strategy(%{"arch" => ["amd64", "arm64"], "otp" => ["27", "28"]}))
	36	+ expand!(strategy(%{"arch" => ["amd64", "arm64"], "otp" => ["27", "28"]}))
31	37
32	38	assert length(combinations) == 4
33	39

@@ -36,20 +42,20 @@ defmodule GitGud.Workflows.MatrixTest do

36	42	end
37	43
38	44	test "a scalar value is a single-value axis" do
39		− assert Matrix.expand(strategy(%{"arch" => "arm64"})) == [%{"arch" => "arm64"}]
	45	+ assert expand!(strategy(%{"arch" => "arm64"})) == [%{"arch" => "arm64"}]
40	46	end
41	47
42	48	test "expansion is stable — map order must not decide job order" do
43	49	matrix = %{"arch" => ["amd64", "arm64"], "otp" => ["27", "28"], "os" => ["a", "b"]}
44	50
45		− assert Matrix.expand(strategy(matrix)) == Matrix.expand(strategy(matrix))
	51	+ assert expand!(strategy(matrix)) == expand!(strategy(matrix))
46	52	end
47	53	end
48	54
49	55	describe "exclude" do
50	56	test "drops combinations matching every key it names" do
51	57	combinations =
52		− Matrix.expand(
	58	+ expand!(
53	59	strategy(%{
54	60	"arch" => ["amd64", "arm64"],
55	61	"otp" => ["27", "28"],

@@ -63,7 +69,7 @@ defmodule GitGud.Workflows.MatrixTest do

63	69
64	70	test "a partial exclude drops every combination it matches" do
65	71	combinations =
66		− Matrix.expand(
	72	+ expand!(
67	73	strategy(%{
68	74	"arch" => ["amd64", "arm64"],
69	75	"otp" => ["27", "28"],

@@ -81,7 +87,7 @@ defmodule GitGud.Workflows.MatrixTest do

81	87	describe "include" do
82	88	test "adds keys to the combinations it matches rather than appending a job" do
83	89	combinations =
84		− Matrix.expand(
	90	+ expand!(
85	91	strategy(%{
86	92	"arch" => ["amd64", "arm64"],
87	93	"include" => [%{"arch" => "arm64", "coverage" => true}]

@@ -97,7 +103,7 @@ defmodule GitGud.Workflows.MatrixTest do

97	103
98	104	test "an entry matching nothing becomes a combination of its own" do
99	105	combinations =
100		− Matrix.expand(
	106	+ expand!(
101	107	strategy(%{
102	108	"arch" => ["amd64", "arm64"],
103	109	"include" => [%{"arch" => "riscv64"}]

@@ -113,7 +119,7 @@ defmodule GitGud.Workflows.MatrixTest do

113	119
114	120	test "an entry with no overlapping keys applies to every combination" do
115	121	combinations =
116		− Matrix.expand(
	122	+ expand!(
117	123	strategy(%{
118	124	"arch" => ["amd64", "arm64"],
119	125	"include" => [%{"otp" => "28"}]

@@ -128,7 +134,7 @@ defmodule GitGud.Workflows.MatrixTest do

128	134
129	135	test "include runs after exclude, so it can add a combination back" do
130	136	combinations =
131		− Matrix.expand(
	137	+ expand!(
132	138	strategy(%{
133	139	"arch" => ["amd64", "arm64"],
134	140	"exclude" => [%{"arch" => "arm64"}],

@@ -144,7 +150,7 @@ defmodule GitGud.Workflows.MatrixTest do

144	150
145	151	test "an include-only matrix is just its entries" do
146	152	combinations =
147		− Matrix.expand(
	153	+ expand!(
148	154	strategy(%{
149	155	"include" => [
150	156	%{"arch" => "amd64", "runner" => "big"},

@@ -219,6 +225,15 @@ defmodule GitGud.Workflows.MatrixTest do

219	225	assert pinned["steps"] == definition["steps"]
220	226	end
221	227
	228	+ test "keeps `max-parallel`, which claiming reads back off the leg" do
	229	+ definition = %{
	230	+ "strategy" => %{"matrix" => "${{ fromJSON(needs.setup.outputs.m) }}", "max-parallel" => 1}
	231	+ }
	232	+
	233	+ assert Matrix.pin_definition(definition, %{"os" => "a"})["strategy"] ==
	234	+ %{"matrix" => %{"os" => ["a"]}, "max-parallel" => 1}
	235	+ end
	236	+
222	237	test "a job with no matrix loses its strategy entirely" do
223	238	definition = %{"strategy" => %{"fail-fast" => false}, "runs-on" => "x"}
224	239

@@ -240,4 +255,107 @@ defmodule GitGud.Workflows.MatrixTest do

240	255	refute Matrix.fail_fast?(%{"fail-fast" => "false"})
241	256	end
242	257	end
	258	+
	259	+ describe "expressions" do
	260	+ @needs %{"needs" => %{"setup" => %{"result" => "success", "outputs" => %{}}}}
	261	+
	262	+ defp with_output(name, value), do: put_in(@needs, ["needs", "setup", "outputs", name], value)
	263	+
	264	+ test "a whole matrix from an earlier job's output" do
	265	+ contexts = with_output("m", ~s({"os": ["a", "b"], "include": [{"os": "b", "big": true}]}))
	266	+
	267	+ assert expand!(strategy("${{ fromJSON(needs.setup.outputs.m) }}"), contexts) == [
	268	+ %{"os" => "a"},
	269	+ %{"os" => "b", "big" => true}
	270	+ ]
	271	+ end
	272	+
	273	+ test "one axis from an expression, beside a literal one" do
	274	+ contexts = with_output("targets", ~s(["x86", "arm"]))
	275	+
	276	+ assert expand!(
	277	+ strategy(%{
	278	+ "target" => "${{ fromJSON(needs.setup.outputs.targets) }}",
	279	+ "otp" => ["28"]
	280	+ }),
	281	+ contexts
	282	+ ) == [%{"otp" => "28", "target" => "x86"}, %{"otp" => "28", "target" => "arm"}]
	283	+ end
	284	+
	285	+ test "include from an expression, and an entry of one" do
	286	+ contexts = with_output("extra", ~s([{"os": "c"}]))
	287	+
	288	+ assert expand!(
	289	+ strategy(%{
	290	+ "os" => ["a"],
	291	+ "include" => "${{ fromJSON(needs.setup.outputs.extra) }}"
	292	+ }),
	293	+ contexts
	294	+ ) == [%{"os" => "a"}, %{"os" => "c"}]
	295	+
	296	+ contexts = with_output("one", ~s({"os": "d"}))
	297	+
	298	+ assert expand!(
	299	+ strategy(%{
	300	+ "os" => ["a"],
	301	+ "include" => ["${{ fromJSON(needs.setup.outputs.one) }}"]
	302	+ }),
	303	+ contexts
	304	+ ) == [%{"os" => "a"}, %{"os" => "d"}]
	305	+ end
	306	+
	307	+ test "bad JSON, or the wrong shape, is an error" do
	308	+ assert {:error, {:expression, _}} =
	309	+ Matrix.expand(
	310	+ strategy("${{ fromJSON(needs.setup.outputs.m) }}"),
	311	+ with_output("m", "nope")
	312	+ )
	313	+
	314	+ assert {:error, {:not_object, _}} =
	315	+ Matrix.expand(
	316	+ strategy("${{ fromJSON(needs.setup.outputs.m) }}"),
	317	+ with_output("m", "[1]")
	318	+ )
	319	+
	320	+ assert {:error, {:not_list, "axis os"}} =
	321	+ Matrix.expand(
	322	+ strategy(%{"os" => "${{ fromJSON(needs.setup.outputs.m) }}"}),
	323	+ with_output("m", ~s({"a": 1}))
	324	+ )
	325	+
	326	+ # A context the expression reads that wasn't supplied.
	327	+ assert {:error, {:expression, _}} =
	328	+ Matrix.expand(strategy("${{ fromJSON(needs.setup.outputs.m) }}"))
	329	+ end
	330	+
	331	+ test "a matrix exclude empties is an error, as on GitHub" do
	332	+ assert Matrix.expand(strategy(%{"os" => ["a"], "exclude" => [%{"os" => "a"}]})) ==
	333	+ {:error, :empty_matrix}
	334	+
	335	+ assert Matrix.expand(
	336	+ strategy("${{ fromJSON(needs.setup.outputs.m) }}"),
	337	+ with_output("m", ~s({"os": []}))
	338	+ ) == {:error, :empty_matrix}
	339	+
	340	+ assert Matrix.describe_error(:empty_matrix) =~ "no combinations"
	341	+ end
	342	+
	343	+ test "dynamic?/1 and matrix?/1" do
	344	+ assert Matrix.dynamic?(strategy("${{ fromJSON(needs.setup.outputs.m) }}"))
	345	+ assert Matrix.dynamic?(strategy(%{"os" => ["a", "${{ vars.OS }}"]}))
	346	+ refute Matrix.dynamic?(strategy(%{"os" => ["a", "b"]}))
	347	+ refute Matrix.dynamic?(nil)
	348	+
	349	+ assert Matrix.matrix?(strategy("${{ fromJSON(needs.setup.outputs.m) }}"))
	350	+ refute Matrix.matrix?(strategy(%{}))
	351	+ end
	352	+
	353	+ test "max_parallel/1" do
	354	+ assert Matrix.max_parallel(%{"max-parallel" => 2}) == 2
	355	+ assert Matrix.max_parallel(%{"max-parallel" => "3"}) == 3
	356	+ assert Matrix.max_parallel(%{"max-parallel" => 0}) == nil
	357	+ assert Matrix.max_parallel(%{"max-parallel" => "${{ vars.N }}"}) == nil
	358	+ assert Matrix.max_parallel(nil) == nil
	359	+ end
	360	+ end
243	361	end

added test/git_gud_web/live/workflow_run_matrix_test.exs

+80 −0

@@ -0,0 +1,80 @@

	1	+defmodule GitGudWeb.WorkflowRunMatrixTest do
	2	+ @moduledoc """
	3	+ The run page while a matrix computed by an earlier job expands: the
	4	+ placeholder row goes, its legs arrive, and their logs (whose streams
	5	+ didn't exist when the page mounted) still render.
	6	+ """
	7	+ use GitGudWeb.ConnCase, async: false
	8	+
	9	+ import Phoenix.LiveViewTest
	10	+ import GitGud.ForgeFixtures
	11	+ import Ecto.Query
	12	+
	13	+ alias GitGud.Repo
	14	+ alias GitGud.Repositories.Storage
	15	+ alias GitGud.Workflows
	16	+ alias GitGud.Workflows.WorkflowJob
	17	+ alias GitGud.Workflows.WorkflowRun
	18	+ alias GitGud.Workflows.Yaml
	19	+
	20	+ @moduletag :capture_log
	21	+
	22	+ test "legs replace the placeholder live, and stream their logs", %{conn: conn} do
	23	+ {_user, repo} = repository_fixture(%{visibility: "public"})
	24	+ repo = Repo.preload(repo, [:owner, :organization])
	25	+
	26	+ {:ok, doc} =
	27	+ Yaml.parse("""
	28	+ name: ci
	29	+ on: push
	30	+ jobs:
	31	+ setup:
	32	+ runs-on: self-hosted
	33	+ steps:
	34	+ - run: true
	35	+ build:
	36	+ needs: setup
	37	+ strategy:
	38	+ matrix: ${{ fromJSON(needs.setup.outputs.m) }}
	39	+ runs-on: self-hosted
	40	+ steps:
	41	+ - run: true
	42	+ """)
	43	+
	44	+ {:ok, run} =
	45	+ %WorkflowRun{repository_id: repo.id}
	46	+ \|> WorkflowRun.create_changeset(%{
	47	+ "number" => 1,
	48	+ "workflow_path" => ".forgejo/workflows/ci.yml",
	49	+ "workflow_name" => "ci",
	50	+ "trigger" => "push",
	51	+ "head_ref" => "refs/heads/main",
	52	+ "head_sha" => :crypto.strong_rand_bytes(20),
	53	+ "workflow_doc" => doc.raw
	54	+ })
	55	+ \|> Repo.insert!()
	56	+ \|> Workflows.rerun()
	57	+
	58	+ jobs = fn -> Repo.all(from j in WorkflowJob, where: j.workflow_run_id == ^run.id) end
	59	+ [placeholder] = Enum.filter(jobs.(), &(&1.job_id == "build"))
	60	+ setup = Enum.find(jobs.(), &(&1.job_id == "setup"))
	61	+
	62	+ {:ok, lv, _html} =
	63	+ live(conn, "/r/#{Storage.repo_handle(repo)}/#{repo.name}/actions/#{run.number}")
	64	+
	65	+ assert has_element?(lv, "#job-#{placeholder.id}")
	66	+
	67	+ {:ok, setup} = Workflows.set_outputs(setup, %{"m" => ~s({"os":["a","b"]})})
	68	+ {:ok, _} = Workflows.update_job_status(setup, "success")
	69	+
	70	+ [leg \| _] = Enum.filter(jobs.(), &(&1.job_id == "build"))
	71	+ [step] = leg \|> Repo.preload(:steps) \|> Map.fetch!(:steps)
	72	+ :ok = Workflows.append_log(step, "hello from a leg")
	73	+
	74	+ refute has_element?(lv, "#job-#{placeholder.id}")
	75	+ assert has_element?(lv, "#job-#{leg.id}")
	76	+
	77	+ lv \|> element("button[phx-value-id='#{step.id}']") \|> render_click()
	78	+ assert has_element?(lv, "#step-log-#{step.id}", "hello from a leg")
	79	+ end
	80	+end

Parents: 91138c0