7.7 KiB · text History b868907

Plan: dynamic matrices, max-parallel, and an empty matrix

Status: ready to execute. Written to be carried out start to finish by one agent without further questions. Needs the expression evaluator GitGud.Workflows.Expr described in workflow-cancellation-and-concurrency.md step 3: if that plan has been executed, reuse it; if not, build it here exactly as specified there (same module, same tests), and that plan will reuse yours. Execute after workflow-job-graph.md if possible (it changes resolve_dependents/1, which step 2 also edits).

Why

strategy.matrix is expanded on the server (GitGud.Workflows.Matrix) into one job row per combination, so legs can go to different runners. Three shapes a workflow commonly uses are handled wrongly, without error:

  1. A matrix computed by an earlier job. matrix: ${{ fromJSON(needs.setup.outputs.matrix) }} (or an axis like target: ${{ fromJSON(needs.setup.outputs.targets) }}) is the standard way to fan out over a computed list. Matrix.expand/1 only understands a literal map: a string matrix hits expand(_strategy) -> [%{}], one leg with no combination, and Matrix.pin_definition/2 then drops strategy entirely — so the runner gets one job whose ${{ matrix.* }} are all empty. A string axis value becomes a single leg whose value is the literal ${{ … }} text.
  2. max-parallel is ignored. pin_definition/2 keeps only matrix and fail-fast, and Workflows.do_claim_next_job/2 doesn’t limit how many legs of one job run at once. max-parallel: 1 (common for deploys to several environments, or a shared test database) runs every leg simultaneously.
  3. A matrix that excludes everything expands to [], which expand/1 turns into [%{}]: one leg, empty matrix. GitHub fails the workflow (“Matrix vector … does not contain any values”).

Semantics to match (GitHub, “jobs.<job_id>.strategy”)

  • matrix may be an expression yielding an object (fromJSON(...)), and any axis or include/exclude may be an expression yielding an array. Expressions may use needs, inputs, vars, github — so a matrix that reads needs is expanded only once those needs have finished.
  • max-parallel: the maximum number of legs of this job running at the same time; default unlimited.
  • A matrix with no combinations after exclude/include is an error; the job fails with that message.

Where things are

Expansion lib/git_gud/workflows/matrix.ex expand/1, matrix?/1, pin_definition/2, fail_fast?/1
Job creation lib/git_gud/workflows.ex create_job/2, create_job_leg/3, create_run/2
Promotion Workflows.resolve_dependents/1 (snapshots needs_snapshot, groups legs by job_id)
Claiming Workflows.do_claim_next_job/2, claim_jobs_for/3, do_claim_specific_job/2
Expressions GitGud.Workflows.Expr (see the header) — needs fromJSON and toJSON added if not present
Logs Workflows.append_log/2 / ensure_log_step/1 to put an error message on a job
Tests test/git_gud/workflows/matrix_test.exs (pure), matrix_dispatch_test.exs (on-disk dispatch, legs, fail-fast)

Steps

1. Expressions in a matrix

  • Expr: add fromJSON(string) (Jason.decode; error → :error) and toJSON(value); and an evaluate(expr_string, contexts) returning the value (not stringified) for a string that is exactly one ${{ … }}.
  • Matrix.expand(strategy, contexts \\ %{}): before expanding, resolve expressions: if matrix is a string, evaluate it (must yield a map); for each axis / include / exclude that is a string containing ${{, evaluate it (must yield a list, or a map for include/exclude entries). Return {:ok, combinations} or {:error, reason}; change the callers. matrix?/1 must say true for a string matrix.
  • Matrix.dynamic?(strategy): whether the matrix contains ${{ anywhere.

2. Expand late when the matrix needs needs

  • In create_job/2: if Matrix.dynamic?/1 and the job has needs, create one placeholder row (status blocked, matrix: nil, name = the job name, definition unchanged) instead of legs. If dynamic without needs, evaluate now with github/inputs/vars contexts (build them as the concurrency plan’s step 3 says; if that context helper doesn’t exist yet, github = %{} plus ref, sha, event_name from the run, vars from GitGud.Secrets.resolve_for/1, inputs from the event payload if present).
  • In resolve_dependents/1, where a dependent is promoted: if it is a placeholder (dynamic strategy, matrix: nil), evaluate the matrix with needs = the snapshot just computed (shape: %{"setup" => %{"result" => …, "outputs" => %{…}}} — note the snapshot stores "status"; map it to result). Then, in one transaction, insert the legs via create_job_leg/3 (each with the same needs_snapshot, status queued) and delete the placeholder (its steps too). Legs keep the placeholder’s job_id, so later needs: on this job still find them. Make sure maybe_complete_run/1 and the run page cope with the placeholder disappearing (they query by run; check the LiveView’s stream of jobs gets the new rows — broadcast each leg).
  • On {:error, reason} (bad JSON, not a list, empty): mark the placeholder (or, for an eager matrix, a single job row created for the purpose) failure, with a log line explaining it (via ensure_log_step/1 + append_log/2), which fails the run and skips its dependents through the normal path.

3. max-parallel

  • pin_definition/2: keep max-parallel in the pinned strategy as it keeps fail-fast.
  • Claiming: in do_claim_next_job/2 (and the other two claim paths), exclude a queued job when its definition’s strategy["max-parallel"] is a positive integer N and N legs with the same workflow_run_id and job_id are already running. Do it in the query (a NOT EXISTS/count subquery over workflow_jobs) so it holds under FOR UPDATE SKIP LOCKED; two runners claiming at the same instant can still both pass — accept that, or serialize per job with pg_advisory_xact_lock(hashtext(run_id || job_id)); choose the lock if it’s simple to add in the transaction, and say which in the commit.

4. Empty matrix

expand/1 returns {:error, :empty_matrix} when exclusion/inclusion leaves nothing (a matrix that had no axes at all is still “not a matrix” → [%{}]). Handled by step 2’s failure path.

5. Docs

README CI list (the matrix bullet): dynamic matrices via fromJSON, expanded when their needs finish; max-parallel; empty matrix fails.

Tests

  • matrix_test.exs: string matrix with contexts; string axis; string include; bad JSON → error; empty after exclude → error; dynamic?/1; pin_definition/2 keeps max-parallel.
  • matrix_dispatch_test.exs (on disk): setup job (no matrix) then build with needs: setup and matrix: ${{ fromJSON(needs.setup.outputs.m) }}: dispatch → one build placeholder; complete setup with outputs m = {"os":["a","b"]} (via Workflows.set_outputs/2 + update_job_status/3) → two queued build legs with matrices %{"os" => "a"} / "b", placeholder gone; a deploy job needing build waits for both legs.
  • Bad output (m = "nope") → build failed with a log line, run failure.
  • max-parallel: 1 with three legs and three matching runners: only one is claimed until it finishes.

Done when

  • mix test passes; no new compile warnings (compare mix compile --force warning lists before/after); mix format only touched files.
  • One commit in the repository’s message style.
  • Not pushed.