Plan: dynamic matrices, max-parallel, and an empty matrix
Status: ready to execute. Written to be carried out start to finish by
one agent without further questions. Needs the expression evaluator
GitGud.Workflows.Expr described in
workflow-cancellation-and-concurrency.md
step 3: if that plan has been executed, reuse it; if not, build it here
exactly as specified there (same module, same tests), and that plan will
reuse yours. Execute after workflow-job-graph.md
if possible (it changes resolve_dependents/1, which step 2 also edits).
Why
strategy.matrix is expanded on the server (GitGud.Workflows.Matrix)
into one job row per combination, so legs can go to different runners.
Three shapes a workflow commonly uses are handled wrongly, without error:
- A matrix computed by an earlier job.
matrix: ${{ fromJSON(needs.setup.outputs.matrix) }}(or an axis liketarget: ${{ fromJSON(needs.setup.outputs.targets) }}) is the standard way to fan out over a computed list.Matrix.expand/1only understands a literal map: a string matrix hitsexpand(_strategy) -> [%{}], one leg with no combination, andMatrix.pin_definition/2then dropsstrategyentirely — so the runner gets one job whose${{ matrix.* }}are all empty. A string axis value becomes a single leg whose value is the literal${{ … }}text. max-parallelis ignored.pin_definition/2keeps onlymatrixandfail-fast, andWorkflows.do_claim_next_job/2doesn’t limit how many legs of one job run at once.max-parallel: 1(common for deploys to several environments, or a shared test database) runs every leg simultaneously.- A matrix that
excludes everything expands to[], whichexpand/1turns into[%{}]: one leg, empty matrix. GitHub fails the workflow (“Matrix vector … does not contain any values”).
Semantics to match (GitHub, “jobs.<job_id>.strategy”)
matrixmay be an expression yielding an object (fromJSON(...)), and any axis orinclude/excludemay be an expression yielding an array. Expressions may useneeds,inputs,vars,github— so a matrix that readsneedsis expanded only once those needs have finished.max-parallel: the maximum number of legs of this job running at the same time; default unlimited.- A matrix with no combinations after
exclude/includeis an error; the job fails with that message.
Where things are
| Expansion | lib/git_gud/workflows/matrix.ex expand/1, matrix?/1, pin_definition/2, fail_fast?/1 |
| Job creation | lib/git_gud/workflows.ex create_job/2, create_job_leg/3, create_run/2 |
| Promotion | Workflows.resolve_dependents/1 (snapshots needs_snapshot, groups legs by job_id) |
| Claiming | Workflows.do_claim_next_job/2, claim_jobs_for/3, do_claim_specific_job/2 |
| Expressions | GitGud.Workflows.Expr (see the header) — needs fromJSON and toJSON added if not present |
| Logs | Workflows.append_log/2 / ensure_log_step/1 to put an error message on a job |
| Tests | test/git_gud/workflows/matrix_test.exs (pure), matrix_dispatch_test.exs (on-disk dispatch, legs, fail-fast) |
Steps
1. Expressions in a matrix
Expr: addfromJSON(string)(Jason.decode; error →:error) andtoJSON(value); and anevaluate(expr_string, contexts)returning the value (not stringified) for a string that is exactly one${{ … }}.Matrix.expand(strategy, contexts \\ %{}): before expanding, resolve expressions: ifmatrixis a string, evaluate it (must yield a map); for each axis /include/excludethat is a string containing${{, evaluate it (must yield a list, or a map for include/exclude entries). Return{:ok, combinations}or{:error, reason}; change the callers.matrix?/1must say true for a string matrix.Matrix.dynamic?(strategy): whether the matrix contains${{anywhere.
2. Expand late when the matrix needs needs
- In
create_job/2: ifMatrix.dynamic?/1and the job hasneeds, create one placeholder row (statusblocked,matrix: nil, name = the job name, definition unchanged) instead of legs. If dynamic withoutneeds, evaluate now withgithub/inputs/varscontexts (build them as the concurrency plan’s step 3 says; if that context helper doesn’t exist yet,github=%{}plusref,sha,event_namefrom the run,varsfromGitGud.Secrets.resolve_for/1,inputsfrom the event payload if present). - In
resolve_dependents/1, where a dependent is promoted: if it is a placeholder (dynamic strategy,matrix: nil), evaluate the matrix withneeds= the snapshot just computed (shape:%{"setup" => %{"result" => …, "outputs" => %{…}}}— note the snapshot stores"status"; map it toresult). Then, in one transaction, insert the legs viacreate_job_leg/3(each with the sameneeds_snapshot, statusqueued) and delete the placeholder (its steps too). Legs keep the placeholder’sjob_id, so laterneeds:on this job still find them. Make suremaybe_complete_run/1and the run page cope with the placeholder disappearing (they query by run; check the LiveView’s stream of jobs gets the new rows — broadcast each leg). - On
{:error, reason}(bad JSON, not a list, empty): mark the placeholder (or, for an eager matrix, a single job row created for the purpose)failure, with a log line explaining it (viaensure_log_step/1+append_log/2), which fails the run and skips its dependents through the normal path.
3. max-parallel
pin_definition/2: keepmax-parallelin the pinned strategy as it keepsfail-fast.- Claiming: in
do_claim_next_job/2(and the other two claim paths), exclude a queued job when its definition’sstrategy["max-parallel"]is a positive integer N and N legs with the sameworkflow_run_idandjob_idare alreadyrunning. Do it in the query (aNOT EXISTS/count subquery overworkflow_jobs) so it holds underFOR UPDATE SKIP LOCKED; two runners claiming at the same instant can still both pass — accept that, or serialize per job withpg_advisory_xact_lock(hashtext(run_id || job_id)); choose the lock if it’s simple to add in the transaction, and say which in the commit.
4. Empty matrix
expand/1 returns {:error, :empty_matrix} when exclusion/inclusion
leaves nothing (a matrix that had no axes at all is still “not a matrix”
→ [%{}]). Handled by step 2’s failure path.
5. Docs
README CI list (the matrix bullet): dynamic matrices via fromJSON,
expanded when their needs finish; max-parallel; empty matrix fails.
Tests
matrix_test.exs: string matrix with contexts; string axis; string include; bad JSON → error; empty after exclude → error;dynamic?/1;pin_definition/2keepsmax-parallel.matrix_dispatch_test.exs(on disk):setupjob (no matrix) thenbuildwithneeds: setupandmatrix: ${{ fromJSON(needs.setup.outputs.m) }}: dispatch → onebuildplaceholder; completesetupwith outputsm = {"os":["a","b"]}(viaWorkflows.set_outputs/2+update_job_status/3) → two queuedbuildlegs with matrices%{"os" => "a"}/"b", placeholder gone; adeployjob needingbuildwaits for both legs.- Bad output (
m = "nope") →buildfailed with a log line, runfailure. max-parallel: 1with three legs and three matching runners: only one is claimed until it finishes.
Done when
mix testpasses; no new compile warnings (comparemix compile --forcewarning lists before/after);mix formatonly touched files.- One commit in the repository’s message style.
- Not pushed.