neiam /gitgud

Git Gud

public · Issues · Labels · Forks · Compare · Actions success · Packages

gix

d4728bb · gmorell · 2026-05-15 00:56

11 files +559 -53

Files changed

modified .gitignore

+7 −0

@@ -45,3 +45,10 @@ npm-debug.log

45	45	# with `GitGud.Storage.Disk`.
46	46	/priv/storage/
47	47
	48	+# Populated copies of the k8s secret templates — never check the real
	49	+# values into git. `app-secrets.example.yml` (committed) carries the
	50	+# placeholder copy.
	51	+/app-secrets.yml
	52	+/secret.yml
	53	+*.secret.yml
	54	+

modified PROGRESS.md

+59 −1

@@ -12,6 +12,62 @@ intermediate "I was here when I stopped" state) lives in the

12	12
13	13	---
14	14
	15	+## Slice 57 — libcluster + L1/L2 distributed Nebulex
	16	+
	17	+Lifts the cache from per-node ETS to a two-tier cluster cache so the syntax-highlight and rendered-markdown payloads survive request landings on any pod.
	18	+
	19	+### Dependencies + topology
	20	+
	21	+- Added `{:libcluster, "~> 3.4"}` to `mix.exs`. Its `Cluster.Strategy.Kubernetes` strategy enumerates peer pod IPs through the K8s API.
	22	+- `runtime.exs` wires the topology only when `LIBCLUSTER_KUBERNETES=1` is set, so single-node and dev runs skip clustering entirely. Other env knobs: `LIBCLUSTER_NAMESPACE`, `LIBCLUSTER_SERVICE`, `LIBCLUSTER_SELECTOR`, `LIBCLUSTER_BASENAME`.
	23	+- `GitGud.Application.start/2` spawns `Cluster.Supervisor` only when a topology exists. No topology → no supervisor, `pg` group has a single member, replicated writes are local — exactly right on a single node.
	24	+
	25	+### Nebulex cache, L1 + L2
	26	+
	27	+- `GitGud.Cache.L1` — `Nebulex.Adapters.Local` (ETS). Per-node, sub-microsecond reads.
	28	+- `GitGud.Cache.L2` — `Nebulex.Adapters.Replicated`. Each clustered node holds a full copy; writes broadcast via `pg` once libcluster connects the nodes.
	29	+- `GitGud.Cache` — `Nebulex.Adapters.Multilevel` with `model: :inclusive` and `levels: [L1, L2]`. Reads cascade L1 → L2 with L2 hits promoted up; writes fan out to both.
	30	+- Multilevel supervises the level caches itself, so the supervision tree only lists `GitGud.Cache`. Listing L1/L2 directly causes a `:already_started`.
	31	+- Configuration split into three blocks in `config/config.exs`: L1 (tighter caps, faster turnover), L2 (looser caps as the "real" working set), and the aggregator's level wiring.
	32	+
	33	+### Manifest (`app.yml`)
	34	+
	35	+- ServiceAccount + Role + RoleBinding — read `pods`/`endpoints` in the namespace, scoped to the `git-gud` ServiceAccount.
	36	+- Headless Service `git-gud-headless` (`clusterIP: None`, `publishNotReadyAddresses: true`) exposes `epmd:4369` + `dist:9100` so libcluster (`mode: :ip`) can resolve peers and rolled pods can join the cluster before readiness probes pass.
	37	+- New env vars: `LIBCLUSTER_*`, `POD_IP` (from `status.podIP`), `RELEASE_NODE=git_gud@$(POD_IP)`, `RELEASE_DISTRIBUTION=name`, `RELEASE_COOKIE` (from `elixir-args/erlang_cookie`). Pod container exposes `4369` and `9100` ports.
	38	+- `app-secrets.example.yml` gains an `erlang_cookie` placeholder with the `openssl rand -hex 32` recipe.
	39	+
	40	+### Tests
	41	+
	42	+- The Multilevel adapter starts cleanly under test (no libcluster topology configured → L2 runs single-member), suite is 501 / 501 on the new cache shape.
	43	+
	44	+## Slice 56 — gix NIF fill-in: commit / log / tree
	45	+
	46	+Wires the three request-path hot operations through gitoxide. Previously every commit fetch, every directory listing, and every history page forked a `git` subprocess via the Cli backend. Now those land in-process; the Cli backend stays the floor for the remaining ops via per-call fallback.
	47	+
	48	+### What's wired
	49	+
	50	+- `commit(path, sha)` — `repo.find_object(id).try_into_commit().decode()` → atom-keyed map matching the Cli shape exactly: `sha`, `tree_sha`, `parent_shas: [sha]`, `author_name`, `author_email`, `authored_at`, `committer_name`, `committer_email`, `committed_at`, `message`.
	51	+- `log(path, start_sha, opts)` — `repo.rev_walk([id]).all()` newest-first, honors `:limit` (default 50) and `:skip`. Per-commit serialization reuses the same `commit_to_map/3` helper as `commit/2`.
	52	+ - When `opts[:paths]` is set, returns `{:error, :unsupported}` so the facade falls back to Cli (`git log -- <paths>`). The single existing call site (`CommitBackfill`) doesn't use path filtering.
	53	+- `tree(path, sha)` — `find_object → try_into_tree → decode` → entries with `name`, `mode` (octal string), `type` (`tree`/`blob`/`commit`), `sha` (20-byte binary), `size` (blob size via `find_header`, nil for non-blobs to match the Cli's `ls-tree --long` `-` cells).
	54	+
	55	+### Dates across the boundary
	56	+
	57	+- gix returns commit times as unix seconds + offset. The NIF emits `%Elixir.DateTime{}` structs (normalized to UTC; the offset is dropped) built field-by-field via a `map_of` helper with `:__struct__ => Elixir.DateTime`, `:calendar => Elixir.Calendar.ISO`, year/month/day/hour/minute/second from `chrono`, `:microsecond {0, 0}`, `:utc_offset 0`, `:std_offset 0`, `:zone_abbr "UTC"`, `:time_zone "Etc/UTC"` — the last two are strings (Ecto's `check_utc_timezone!` does a string equality check).
	58	+
	59	+### Deferred (still on Cli)
	60	+
	61	+- `diff_stats`, `merge_base`, `merge_tree`, `commit_tree`, `update_ref` — return `{:error, :unsupported}`, which the facade's `dispatch/2` already handles by re-applying the same args to the Cli backend. gix 0.66's surface for these is churning across point releases (gix-diff types diverged between our explicit gix-diff dep and gix's transitive copy; `repo.merge_base` was renamed/relocated). Cli works correctly today; lift them when the gix API for our pinned version settles.
	62	+
	63	+### Backend selection
	64	+
	65	+- `GitGud.Git.pick_backend/0` already preferred Nif when `available?/0` is true. With the NIF now exporting the new functions, `current_backend/0` reports `GitGud.Git.Nif` in default config; nothing extra needed.
	66	+
	67	+### Tests
	68	+
	69	+- The full suite is the spec — every existing test exercising the API now runs through the NIF where supported, and through Cli for the deferred ones. The pre-existing JWT-tampering flake aside, the suite is 501 / 501 on clean seeds.
	70	+
15	71	## Slice 55 — Org members management
16	72
17	73	New admin page to add / remove / promote / demote org members. Last-admin guard: the org can never end up without an admin.

@@ -1067,7 +1123,9 @@ Closed slices 14, 15, 16, 17 in this session. Final task graph:

1067	1123	\| 118 \| done \| Registration modes + invite system (slice 53) \|
1068	1124	\| 119 \| done \| Org visibility + public-org anonymous access + description in settings (slice 54) \|
1069	1125	\| 120 \| done \| Org members management (slice 55) \|
	1126	+\| 121 \| done \| gix NIF: commit + log + tree (slice 56) \|
	1127	+\| 122 \| done \| libcluster + L1/L2 distributed Nebulex (slice 57) \|
1070	1128
1071		−End-of-session: 501 tests pass. Next pickups: gix NIF fill-in, in-browser PR conflict resolution, or whichever next feature lands.
	1129	+End-of-session: 501 tests pass on the NIF backend with the new Multilevel cache. Next pickups: in-browser PR conflict resolution, or filling in the remaining gix NIF ops (diff_stats / merge_base / merge_tree / commit_tree / update_ref) once the gix API shakes out.
1072	1130	Embedded SSH daemon verified end-to-end against `git push --set-upstream origin master` from a real client (after fixing two bugs: spawning git with the dashed subcommand name failed; `:stderr_to_stdout` on the Port corrupted the pkt-line stream).
1073	1131

modified config/config.exs

+23 −3

@@ -134,13 +134,33 @@ config :git_gud, GitGud.Accounts.TwoFactor,

134	134	required: false,
135	135	grace_period_days: 14
136	136
137		−# In-memory cache. ETS-backed; partitioned for write-heavy workloads.
138		−# Override `gc_interval` to tune memory pressure.
139		−config :git_gud, GitGud.Cache,
	137	+# In-memory cache, L1 + L2.
	138	+#
	139	+# L1 = per-node ETS. Tighter caps; this tier turns over quickly under
	140	+# typical browsing patterns.
	141	+config :git_gud, GitGud.Cache.L1,
140	142	gc_interval: :timer.hours(12),
141	143	max_size: 50_000,
142	144	allocated_memory: 200_000_000
143	145
	146	+# L2 = Replicated across cluster nodes. Larger caps; L2 is what
	147	+# survives across node hops, so it's the "real" working set.
	148	+config :git_gud, GitGud.Cache.L2,
	149	+ primary: [
	150	+ gc_interval: :timer.hours(24),
	151	+ max_size: 100_000,
	152	+ allocated_memory: 400_000_000
	153	+ ]
	154	+
	155	+# Multilevel aggregator — reads cascade L1 → L2, writes fan out to both.
	156	+# Inclusive model promotes L2 hits back into L1 on the way back.
	157	+config :git_gud, GitGud.Cache,
	158	+ model: :inclusive,
	159	+ levels: [
	160	+ {GitGud.Cache.L1, []},
	161	+ {GitGud.Cache.L2, []}
	162	+ ]
	163	+
144	164	# Garage (S3-compatible). Endpoint, region, and keys are all set in
145	165	# runtime.exs from env vars.
146	166	config :ex_aws,

modified config/runtime.exs

+85 −2

@@ -24,11 +24,30 @@ config :git_gud, GitGudWeb.Endpoint,

24	24	http: [port: String.to_integer(System.get_env("PORT", "4000"))]
25	25
26	26	if config_env() == :prod do
	27	+ # Two ways to point at Postgres:
	28	+ #
	29	+ # 1. `DATABASE_URL` — single connection-string env, classic Phoenix.
	30	+ # 2. Split `POSTGRES_HOST` / `POSTGRES_PORT` / `POSTGRES_USER` /
	31	+ # `POSTGRES_PASSWORD` / `POSTGRES_DB` — friendlier under k8s when
	32	+ # passwords come from a Secret and contain URL-hostile chars.
	33	+ #
	34	+ # `DATABASE_URL` wins when both are set.
27	35	database_url =
28	36	System.get_env("DATABASE_URL") \|\|
	37	+ (case {System.get_env("POSTGRES_HOST"), System.get_env("POSTGRES_USER"),
	38	+ System.get_env("POSTGRES_PASSWORD"), System.get_env("POSTGRES_DB")} do
	39	+ {host, user, pass, db} when is_binary(host) and is_binary(user) and
	40	+ is_binary(pass) and is_binary(db) ->
	41	+ port = System.get_env("POSTGRES_PORT") \|\| "5432"
	42	+ "ecto://#{URI.encode_www_form(user)}:#{URI.encode_www_form(pass)}@#{host}:#{port}/#{db}"
	43	+
	44	+ _ ->
	45	+ nil
	46	+ end) \|\|
29	47	raise """
30		− environment variable DATABASE_URL is missing.
31		− For example: ecto://USER:PASS@HOST/DATABASE
	48	+ No database configured. Set DATABASE_URL, or supply the
	49	+ POSTGRES_HOST / POSTGRES_USER / POSTGRES_PASSWORD / POSTGRES_DB
	50	+ bundle (POSTGRES_PORT defaults to 5432).
32	51	"""
33	52
34	53	maybe_ipv6 = if System.get_env("ECTO_IPV6") in ~w(true 1), do: [:inet6], else: []

@@ -41,6 +60,41 @@ if config_env() == :prod do

41	60	# pool_count: 4,
42	61	socket_options: maybe_ipv6
43	62
	63	+ # Persistent paths — point these at mounted PVCs in k8s, or anywhere
	64	+ # writable on a single-node host. Leave unset to keep the in-release
	65	+ # `priv/` defaults (fine for dev, not durable in a release).
	66	+ if path = System.get_env("GIT_REPOS_PATH") do
	67	+ config :git_gud, GitGud.Repositories.Storage, base_path: path
	68	+ end
	69	+
	70	+ if root = System.get_env("STORAGE_DISK_ROOT") do
	71	+ config :git_gud, GitGud.Storage.Disk, root: root
	72	+ end
	73	+
	74	+ # Embedded SSH daemon. Off by default in prod (you may front it with
	75	+ # an external sshd / dedicated TCP service); flip the env to opt in.
	76	+ # `SSH_HOST_KEY_DIR` should point at shared storage when running
	77	+ # multi-replica so every pod presents the same host fingerprint —
	78	+ # otherwise SSH clients get a "remote host identification changed"
	79	+ # warning when their load-balancer connection lands on a new pod.
	80	+ ssh_overrides =
	81	+ [enabled: System.get_env("SSH_SERVER_ENABLED") in ~w(true 1)]
	82	+ \|> Keyword.merge(
	83	+ case System.get_env("SSH_HOST_KEY_DIR") do
	84	+ nil -> []
	85	+ dir -> [host_key_dir: dir]
	86	+ end
	87	+ )
	88	+
	89	+ config :git_gud, GitGud.Ssh.Server, ssh_overrides
	90	+
	91	+ # Registration mode — :invite_only in code defaults; commonly flipped
	92	+ # at deploy time. Accepts the literal strings "open", "invite_only",
	93	+ # or "closed"; anything else falls back to :invite_only.
	94	+ if mode = System.get_env("REGISTRATION_MODE") do
	95	+ config :git_gud, registration_mode: mode
	96	+ end
	97	+
44	98	# The secret key base is used to sign/encrypt cookies and other secrets.
45	99	# A default value is used in config/dev.exs and config/test.exs but you
46	100	# want to use a different value for prod and you most likely don't want

@@ -57,6 +111,35 @@ if config_env() == :prod do

57	111
58	112	config :git_gud, :dns_cluster_query, System.get_env("DNS_CLUSTER_QUERY")
59	113
	114	+ # libcluster — Kubernetes endpoints lookup against the headless
	115	+ # service deployed alongside the app. RBAC (pods + endpoints) is
	116	+ # granted to the `git-gud` ServiceAccount in app.yml.
	117	+ #
	118	+ # Gated on `LIBCLUSTER_KUBERNETES=1`. Off → `GitGud.Application` skips
	119	+ # Cluster.Supervisor entirely and `GitGud.Cache.L2` runs as a single-
	120	+ # member `pg` group, which is exactly right on a single node.
	121	+ if System.get_env("LIBCLUSTER_KUBERNETES") in ~w(true 1) do
	122	+ namespace = System.get_env("LIBCLUSTER_NAMESPACE") \|\| "git-gud"
	123	+ service = System.get_env("LIBCLUSTER_SERVICE") \|\| "git-gud-headless"
	124	+ selector = System.get_env("LIBCLUSTER_SELECTOR") \|\| "name=git-gud"
	125	+ basename = System.get_env("LIBCLUSTER_BASENAME") \|\| "git_gud"
	126	+
	127	+ config :libcluster,
	128	+ topologies: [
	129	+ git_gud: [
	130	+ strategy: Cluster.Strategy.Kubernetes,
	131	+ config: [
	132	+ mode: :ip,
	133	+ kubernetes_selector: selector,
	134	+ kubernetes_service_name: service,
	135	+ kubernetes_node_basename: basename,
	136	+ kubernetes_namespace: namespace,
	137	+ polling_interval: 10_000
	138	+ ]
	139	+ ]
	140	+ ]
	141	+ end
	142	+
60	143	config :git_gud, GitGudWeb.Endpoint,
61	144	url: [host: host, port: 443, scheme: "https"],
62	145	http: [

modified lib/git_gud/application.ex

+31 −11

@@ -7,17 +7,25 @@ defmodule GitGud.Application do

7	7
8	8	@impl true
9	9	def start(_type, _args) do
10		− children = [
11		− GitGudWeb.Telemetry,
12		− GitGud.Repo,
13		− {DNSCluster, query: Application.get_env(:git_gud, :dns_cluster_query) \|\| :ignore},
14		− {Phoenix.PubSub, name: GitGud.PubSub},
15		− GitGud.Cache,
16		− {Oban, Application.fetch_env!(:git_gud, Oban)},
17		− GitGud.Federation.RateLimiter,
18		− GitGud.Ssh.Server,
19		− GitGudWeb.Endpoint
20		− ]
	10	+ topologies = Application.get_env(:libcluster, :topologies, [])
	11	+
	12	+ children =
	13	+ [
	14	+ GitGudWeb.Telemetry,
	15	+ GitGud.Repo,
	16	+ {DNSCluster, query: Application.get_env(:git_gud, :dns_cluster_query) \|\| :ignore}
	17	+ ] ++
	18	+ cluster_children(topologies) ++
	19	+ [
	20	+ {Phoenix.PubSub, name: GitGud.PubSub},
	21	+ # Multilevel supervises its L1 + L2 children itself per the
	22	+ # `levels:` config — listing them here would double-start.
	23	+ GitGud.Cache,
	24	+ {Oban, Application.fetch_env!(:git_gud, Oban)},
	25	+ GitGud.Federation.RateLimiter,
	26	+ GitGud.Ssh.Server,
	27	+ GitGudWeb.Endpoint
	28	+ ]
21	29
22	30	# See https://hexdocs.pm/elixir/Supervisor.html
23	31	# for other strategies and supported options

@@ -25,6 +33,18 @@ defmodule GitGud.Application do

25	33	Supervisor.start_link(children, opts)
26	34	end
27	35
	36	+ # When `:libcluster` has no topology configured (dev / test / single-
	37	+ # node prod), skip Cluster.Supervisor entirely. In a multi-replica
	38	+ # prod with the Kubernetes topology defined in runtime.exs, libcluster
	39	+ # discovers peer pods and connects them; once the cluster forms,
	40	+ # `GitGud.Cache.L2`'s Replicated adapter joins its pg group across
	41	+ # nodes.
	42	+ defp cluster_children([]), do: []
	43	+
	44	+ defp cluster_children(topologies) do
	45	+ [{Cluster.Supervisor, [topologies, [name: GitGud.ClusterSupervisor]]}]
	46	+ end
	47	+
28	48	# Tell Phoenix to update the endpoint configuration
29	49	# whenever the application is updated.
30	50	@impl true

modified lib/git_gud/cache.ex

+40 −9

Click to load diff…

modified mix.exs

+4 −0

Click to load diff…

modified mix.lock

+1 −0

Click to load diff…

modified native/gitgud_gix/Cargo.lock

+25 −0

Click to load diff…

modified native/gitgud_gix/Cargo.toml

+3 −0

Click to load diff…

modified native/gitgud_gix/src/lib.rs

+281 −27

Click to load diff…

Parents: 8da161a