GITGUD

neiam neiam /gitgud

Git Gud

public · Issues · Pulls · Labels · Forks · Compare · Actions success · Badges · Packages

⭐ 1

Log in to mark this repository.

README: the package registries

8e8d69e · Gabriel Morell · 2026-09-11 15:19

1 files +167 -14
Message 
{commit_body(@commit)}

Files changed

modified README.md
+167 −14
@@ -2,8 +2,10 @@
2 2
3 3 A self-hosted code forge built on Phoenix 1.8 — git over HTTPS + SSH,
4 4 issues, pull requests, wikis, ForgeFed federation, CI via the Forgejo
5 −Actions runner protocol, OCI/LFS storage. Designed to embed the right
6 −external tools (Forgejo runner, Garage, zot) rather than reinvent them.
5 +Actions runner protocol, LFS storage, and private package registries
6 +for containers, Python, Elixir, Rust and Debian. Designed to embed the
7 +right external tools (Forgejo runner, Garage, zot) rather than
8 +reinvent them.
7 9
8 10 > Status: development. The forge is feature-complete enough to host its
9 11 > own development and federate with other ForgeFed-speaking instances,
@@ -89,12 +91,16 @@ external tools (Forgejo runner, Garage, zot) rather than reinvent them.
89 91 - `GitGud.Storage.Disk` (default) — single-node dev, single-node prod, NFS-shared multi-node
90 92 - `GitGud.Storage.Garage` — S3-flavored ([Garage](https://garagehq.deuxfleurs.fr/))
91 93 - Add another with one `@behaviour` impl
92 −- **Packages**
93 − - OCI registry hosted in [zot](https://zotregistry.dev/) — deployment manifest ships in `zot.yml` (PVC-backed by default, S3/Garage driver kept in a comment block)
94 − - The forge owns `/v2/token`: `docker login` → Basic-auth handshake → RS256-signed JWT scoped to `repository:<owner>/<name>:pull,push` (`Packages.Token`); zot validates with the matching public PEM
95 − - Keypair minted by `mix gitgud.registry.keygen` (PKCS#1 private, SubjectPublicKeyInfo public — verified against `openssl rsa -pubout`); prints kubectl one-liners for splitting it across the `elixir-args` and `zot` Secrets
96 − - Local mirror of `(package, version)` rows via the webhook at `/_webhooks/zot` (Bearer-token guarded by `ZOT_WEBHOOK_SECRET`)
97 − - Browseable at `/r/<owner>/<name>/packages` with per-package version history
94 +- **Package registries** — five formats: four served by the forge on
95 + one substrate, plus OCI in zot
96 + - **PyPI** ([PEP 503](https://peps.python.org/pep-0503/) + [PEP 691](https://peps.python.org/pep-0691/)) — `pip install --index-url …/simple/`, `twine upload`. Both serialisations off one data structure, negotiated on `Accept` with real q-values; PEP 592 yank state; name normalisation applied before every lookup
97 + - **Hex** ([registry v2](https://github.com/hexpm/specifications/blob/main/registry-v2.md)) — `mix hex.repo add` + `{:dep, "~> 1.0", repo: "…"}`. Protobuf encoded per request out of Postgres rather than rebuilt by `mix hex.registry build`, so publishing is an insert instead of O(everything)
98 + - **Cargo** ([sparse index](https://doc.rust-lang.org/cargo/reference/registry-index.html)) — `cargo publish`, `add`, `yank`. Prefix-sharded index with real `ETag`/`304` revalidation, `features2`/`v: 2` splitting, and the `explicit_name_in_toml` → `package` rename translation
99 + - **apt** ([DebianRepository/Format](https://wiki.debian.org/DebianRepository/Format)) — signed `InRelease`, `Packages.xz`, pool. **Namespace-wide repositories at `_all`**: one `deb` line and one keyring file for everything an org (or a user) publishes, rather than one of each per repository
100 + - **OCI** hosted in [zot](https://zotregistry.dev/) — the forge owns `/v2/token` (`docker login` → Basic-auth handshake → RS256 JWT scoped to `repository:<owner>/<name>:pull,push`); zot owns the manifests and blobs. Deployment manifest in `zot.yml`; keypair from `mix gitgud.registry.keygen`; `(package, version)` mirrored via `/_webhooks/zot`
101 + - **Publish from CI with no stored secret** — `${{ github.token }}` is accepted by every registry and authorized to write its own repository only, exactly as at the OCI token endpoint
102 + - **Signing keys are per namespace**, not per repository — one `signed-by=` entry or one `--public-key` covers every repository an org publishes. Served at `/pkg/_keys/:format/:owner`; private halves encrypted at rest under the `GitGud.Secrets` master key
103 + - Browseable at `/r/<owner>/<name>/packages`, grouped by format, with paste-ready install *and* publish instructions for both a terminal and a workflow
98 104
99 105 ## Architecture
100 106
@@ -113,16 +119,22 @@ external tools (Forgejo runner, Garage, zot) rather than reinvent them.
113 119 │ cache │ └─────────┘ └─────────────────┘
114 120 └──────────────┘
115 121 │
116 − ┌───────▼──────┐ ┌────────────────┐
117 − │ gix (NIF) │ ┌──────► │ GitGud.Storage │
118 − │ + git CLI │ │ │ Disk / Garage │
119 − │ fallback │ │ │ (LFS+CI blobs) │
120 − └──────────────┘ │ └────────────────┘
122 + ┌───────▼──────┐ ┌──────────────────┐
123 + │ gix (NIF) │ ┌──────► │ GitGud.Storage │
124 + │ + git CLI │ │ │ Disk / Garage │
125 + │ fallback │ │ │ LFS + CI blobs │
126 + └──────────────┘ │ │ + package bytes │
127 + │ │ └──────────────────┘
121 128 │ │
122 129 ┌───────▼─────────────┴──────────────────────────┐
123 130 │ priv/repos/<owner>/<name>.git │
124 131 │ priv/storage/<bucket>/<key> │
125 132 └────────────────────────────────────────────────┘
133 +
134 + ┌────────────────────────────────────────────────┐
135 + │ gitgud_apt (NIF) — OpenPGP signatures for the │
136 + │ apt Release, plus the xz/zstd OTP can't do │
137 + └────────────────────────────────────────────────┘
126 138 ```
127 139
128 140 **Embed, don't reinvent.** The stance is to delegate to mature
@@ -135,6 +147,32 @@ external tools where they exist:
135 147 metadata, zot owns the manifests + blobs
136 148 - **gix** (Rust) for git operations via a Rustler NIF, with a `git`
137 149 CLI fallback for anything not yet wired through the NIF
150 +- **sequoia-openpgp** (Rust) for the OpenPGP signatures apt requires —
151 + OTP has no OpenPGP at all, and this is the one place in the
152 + registries that needed a NIF
153 +
154 +The same stance is why the *other four* registries are **not**
155 +embedded. The OCI distribution spec is a genuine protocol — manifests,
156 +blob dedup, cross-mounting, GC, conformance — so zot earns its place.
157 +An apt repository is static files over plain HTTP; a PyPI simple index
158 +is two documents; a Cargo sparse index is a tree of newline-delimited
159 +JSON. There is no daemon to embed, so they're served directly:
160 +
161 +```
162 +GitGud.Packages.Format # index/2, store/3, content_type/1
163 + ├── Formats.Pypi # + normalize/1 — PEP 503 folds -_.
164 + ├── Formats.Hex
165 + ├── Formats.Cargo
166 + └── Formats.Deb # + namespace_index/3 — the _all repository
167 +```
168 +
169 +Everything the four share already existed — blob storage
170 +(`GitGud.Storage`), namespacing (`Package` / `PackageVersion` /
171 +`PackageFile`), auth tokens, a UI. What differs per format is only the
172 +*index shape* and the *upload framing*, which is what the behaviour
173 +captures. One dispatcher at `/pkg/:format/:owner/:name/*rest` resolves
174 +the repository, authenticates, authorizes, and hands the rest to the
175 +adapter.
138 176
139 177 ## Quick start (development)
140 178
@@ -176,6 +214,9 @@ These all live in `config/*.exs`. Key ones:
176 214 | `GitGudWeb.PackagesWebhookController` `shared_secret` | Bearer-token gate on the zot webhook (env: `ZOT_WEBHOOK_SECRET`) | `nil` (open in dev) |
177 215 | `GitGud.Packages.Token` `private_key_pem` | RS256 PEM that signs OCI registry Bearer tokens (env: `GITGUD_REGISTRY_PRIVATE_KEY`) | `nil` (registry auth disabled) |
178 216 | `GitGud.Packages.Token` `issuer` / `audience` / `key_id` / `ttl_seconds` | JWT claim defaults zot must agree on | `gitgud-registry` / `container_registry` / `gitgud-registry-1` / `300` |
217 +| `:package_formats` | Which registry formats this instance serves. A format left out 404s | all of `pypi` / `hex` / `cargo` / `deb` |
218 +| `:package_registry_url` | Absolute base URL the registries put in index documents (behind a proxy that rewrites the host) | endpoint URL |
219 +| `GitGud.Secrets` `master_key` | Also wraps package signing keys' private halves | dev placeholder |
179 220
180 221 To swap to S3-flavored storage in prod:
181 222
@@ -193,7 +234,10 @@ config :git_gud, GitGud.Federation, federation_mode: :open
193 234 ## Storage backends
194 235
195 236 `GitGud.Storage` is a four-method behaviour
196 −(`put / get / delete / presign_url`). Two implementations ship:
237 +(`put / get / delete / presign_url`). It backs LFS objects, CI
238 +artifacts and package bytes — the latter under content-addressed keys,
239 +so re-publishing identical bytes costs nothing. Two implementations
240 +ship:
197 241
198 242 - **Disk** — filesystem-backed. Presigned URLs route through a local
199 243 `StorageController` that verifies a `Phoenix.Token`. Works for
@@ -287,6 +331,106 @@ Add `GITGUD_REGISTRY_PRIVATE_KEY` to the app pod (already wired in
287 331 ingress. Workflows then push to `<registry-host>/<owner>/<repo>:<tag>`
288 332 and the auth handshake flows through the forge.
289 333
334 +## Package registries
335 +
336 +Unlike [the OCI registry](#oci-registry), these need no deployment step
337 +at all — no sidecar, no keygen, no DNS. They are served by the app,
338 +keys mint themselves on first use, and the packages page prints the
339 +exact lines to paste.
340 +
341 +```
342 +/pkg/pypi/<owner>/<repo>/simple/ pip
343 +/pkg/hex/<owner>/<repo> mix
344 +/pkg/cargo/<owner>/<repo>/ cargo (sparse+…)
345 +/pkg/deb/<owner>/<repo> apt
346 +/pkg/deb/<owner>/_all apt, whole namespace
347 +/pkg/_keys/<format>/<owner> the namespace's public key
348 +```
349 +
350 +### Publishing
351 +
352 +From a terminal, with a registry token from
353 +`/r/<owner>/<name>/settings/registry-tokens`:
354 +
355 +```sh
356 +# Python
357 +twine upload --repository-url https://<host>/pkg/pypi/<owner>/<repo>/ \
358 + --username __token__ --password "$TOKEN" dist/*
359 +
360 +# Rust — after adding [registries.<repo>] index = "sparse+…" to .cargo/config.toml
361 +cargo login --registry <repo> "$TOKEN"
362 +cargo publish --registry <repo>
363 +
364 +# Elixir
365 +mix hex.build && curl -H "Authorization: $TOKEN" \
366 + --data-binary "@$(ls *.tar)" https://<host>/pkg/hex/<owner>/<repo>/publish
367 +
368 +# Debian
369 +curl -H "Authorization: $TOKEN" --data-binary @pkg.deb \
370 + "https://<host>/pkg/deb/<owner>/<repo>/upload?dist=stable&component=main"
371 +```
372 +
373 +From CI, there is no token to create. `${{ github.token }}` is minted
374 +per job and authorized to write that job's own repository — the same
375 +deal the container registry already offers:
376 +
377 +```yaml
378 +- name: Publish to the forge
379 + run: twine upload --repository-url https://<host>/pkg/pypi/${{ github.repository }}/ dist/*
380 + env:
381 + TWINE_USERNAME: __token__
382 + TWINE_PASSWORD: ${{ github.token }}
383 +```
384 +
385 +### Consuming, and the signing keys
386 +
387 +Hex and apt both require signatures, and both want the *public* half
388 +pinned by the consumer. Those keys belong to the **namespace** — an
389 +organization, or a user for personal repositories — not to individual
390 +repositories, so one pin covers everything that namespace publishes:
391 +
392 +```sh
393 +curl -fsSL https://<host>/pkg/_keys/deb/acme \
394 + | sudo tee /etc/apt/keyrings/acme.asc > /dev/null
395 +
396 +# one line for every repository acme publishes…
397 +echo "deb [signed-by=/etc/apt/keyrings/acme.asc] https://<host>/pkg/deb/acme/_all stable main" \
398 + | sudo tee /etc/apt/sources.list.d/acme.list
399 +
400 +# …or one repository on its own, verified by the same key
401 +echo "deb [signed-by=/etc/apt/keyrings/acme.asc] https://<host>/pkg/deb/acme/tools stable main" \
402 + | sudo tee /etc/apt/sources.list.d/acme-tools.list
403 +```
404 +
405 +`_all` can never collide with a repository, because a repository name
406 +must start with an alphanumeric. It aggregates every repository in the
407 +namespace **the caller may read** — anonymous sees the public ones, a
408 +user sees what they can read — and the `Release` is signed per request,
409 +so a filtered view is still a correctly signed one.
410 +
411 +Private registries authenticate the way each ecosystem expects: twine
412 +sends Basic with `__token__` in the username, `mix` and `cargo` send
413 +the bare token, apt reads `/etc/apt/auth.conf.d/`. A 401 carries a
414 +Basic challenge, which `pip` and `apt` retry against.
415 +
416 +### The apt NIF
417 +
418 +apt is the one format that needed native code: it requires an OpenPGP
419 +signature on `Release`, and OTP has no OpenPGP — `:public_key` speaks
420 +PKCS and X.509. `native/gitgud_apt` binds `sequoia-openpgp` through
421 +Rustler, and also carries xz and zstd, since a `.deb`'s control member
422 +is `control.tar.xz` on anything since squeeze and `:erl_tar` knows only
423 +gzip.
424 +
425 +Its crypto backend is chosen so the NIF adds **no package to either
426 +container image**: sequoia's default Nettle backend would pull
427 +`libnettle` and `libgmp` into the runtime image, and the pure-Rust
428 +backend is gated behind `allow-variable-time-crypto`, which is the
429 +wrong trade for code holding a private key. The OpenSSL backend links
430 +`libssl`/`libcrypto`, which the builder already installs for gix and
431 +the runner already installs for the Erlang runtime. The compression
432 +crates are pure Rust for the same reason.
433 +
290 434 ## What's not done
291 435
292 436 See [ROADMAP.md](./ROADMAP.md). Highlights:
@@ -298,6 +442,15 @@ See [ROADMAP.md](./ROADMAP.md). Highlights:
298 442 - Code search (Postgres FTS handles issues/PRs; code search is its own beast)
299 443 - RFC 9421 (HTTP Messages) signatures when peers upgrade from draft-cavage-12
300 444 - Reputation aging for new federated hosts
445 +- Namespace-wide indexes (`_all`) exist only for apt so far — the
446 + callback is optional on the Format behaviour, so PyPI and Cargo can
447 + opt in without new routing
448 +- Package retention/GC — nothing prunes old versions yet, and yanking
449 + deliberately keeps the bytes so existing pins keep resolving
450 +- Signing-key rotation for Hex and apt. Keys mint themselves on first
451 + use and are re-wrapped by the `GitGud.Secrets` master-key rotation,
452 + but there is no "roll the keypair" flow, because rolling it breaks
453 + every consumer that pinned the old one
301 454
302 455 See [PROGRESS.md](./PROGRESS.md) for the slice-by-slice history of how
303 456 the forge was built, and what each shipped — currently through slice 58.

Parents: 202c083