neiam
/gitgud
Git Gud
public · Issues · Pulls · Labels · Forks · Compare · Actions success · Badges · Packages
⭐
1
Log in to mark this repository.
Plan: private PyPI, Hex, Cargo and apt registries
bd60adf · Gabriel Morell · 2026-09-11 05:04
Message
{commit_body(@commit)}
Files changed
added
docs/package-registries.md
+244
−0
@@ -0,0 +1,244 @@
| 1 | +# Private package registries — plan | |
| 2 | + | |
| 3 | +Hosting PyPI, Hex, Cargo and apt repositories inside the forge, the way | |
| 4 | +the OCI registry is hosted alongside it today. | |
| 5 | + | |
| 6 | +Spec details are quoted inline deliberately: they were the expensive | |
| 7 | +part to establish, and an implementer shouldn't have to re-derive them. | |
| 8 | + | |
| 9 | +## The framing | |
| 10 | + | |
| 11 | +These are **not four features**. They are one substrate with four | |
| 12 | +format adapters. What they share already exists: | |
| 13 | + | |
| 14 | +| Need | Already there | | |
| 15 | +|---|---| | |
| 16 | +| Blob storage | `GitGud.Storage` — `put/get/delete/presign_url` behaviour, backs LFS + CI artifacts | | |
| 17 | +| Namespacing | `Package` / `PackageVersion`, per repo | | |
| 18 | +| Auth tokens | `RegistryToken` (`ggrt_…`), hashed, `pull`/`push` scopes | | |
| 19 | +| UI | `/r/:owner/:name/packages` | | |
| 20 | + | |
| 21 | +What differs per format is only the **index shape** and the **upload | |
| 22 | +framing**. That is a `@behaviour` with four impls, the same move | |
| 23 | +`GitGud.Storage` already makes for backends. | |
| 24 | + | |
| 25 | +Why this differs from the OCI decision: zot is embedded because the | |
| 26 | +distribution spec is a genuine protocol — manifests, blob dedup, | |
| 27 | +cross-mounting, GC, conformance. An apt repository is static files over | |
| 28 | +plain HTTP. There is no daemon to embed. The only hard part is OpenPGP | |
| 29 | +signing, which OTP cannot do. | |
| 30 | + | |
| 31 | +## UI surface | |
| 32 | + | |
| 33 | +One page per repo listing what has been published and is installable — | |
| 34 | +extending `/r/:owner/:name/packages` from "container images" to | |
| 35 | +"artifacts, by format", each with its versions and a copy-pasteable | |
| 36 | +install line. | |
| 37 | + | |
| 38 | +--- | |
| 39 | + | |
| 40 | +## Phase 0 — substrate | |
| 41 | + | |
| 42 | +- `GitGud.Packages.Format` behaviour: `index/2`, `store/3`, | |
| 43 | + `content_type/1`. | |
| 44 | +- `format` column on `Package` (`oci` | `pypi` | `hex` | `cargo` | | |
| 45 | + `deb`); backfill existing rows to `oci`. | |
| 46 | +- Generalise `RegistryToken` past OCI. The token itself is already | |
| 47 | + format-agnostic — only the auth *header* differs per ecosystem. | |
| 48 | +- Route `/pkg/:format/:owner/:name/*rest`. | |
| 49 | +- Nothing user-visible ships. Everything after is an adapter. | |
| 50 | + | |
| 51 | +## Phase 1 — PyPI | |
| 52 | + | |
| 53 | +Smallest, no cryptography, proves the substrate. | |
| 54 | + | |
| 55 | +**PEP 503 (Simple Repository API)** | |
| 56 | + | |
| 57 | +- Normalisation, exactly: `re.sub(r"[-_.]+", "-", name).lower()`. | |
| 58 | + Apply before *any* lookup. | |
| 59 | +- `/simple/` lists projects; `/simple/<normalized>/` lists files. | |
| 60 | +- Anchor text is the filename; `href` may carry `#sha256=<hex>`. | |
| 61 | +- **Trailing slashes are mandatory.** Unslashed URLs must redirect. | |
| 62 | +- Clients request normalised URLs directly; servers *may* redirect | |
| 63 | + unnormalised ones. | |
| 64 | + | |
| 65 | +**PEP 691 (JSON Simple API)** | |
| 66 | + | |
| 67 | +- Content negotiation on `Accept`; respond with `Content-Type`: | |
| 68 | + - `application/vnd.pypi.simple.v1+json` | |
| 69 | + - `application/vnd.pypi.simple.v1+html` | |
| 70 | + - `text/html` is a legacy alias for `v1+html` | |
| 71 | + - `406` if nothing matches | |
| 72 | +- Root: `{"meta": {"api-version": "1.0"}, "projects": [{"name": …}]}` | |
| 73 | +- Project: `{"meta": …, "name": …, "files": [...]}` | |
| 74 | +- File entry required fields: `filename`, `url`, `hashes` — `hashes` | |
| 75 | + must be present even when empty. `sha256` preferred. | |
| 76 | +- Optional: `requires-python`, `yanked`, `dist-info-metadata`, | |
| 77 | + `gpg-sig`. | |
| 78 | + | |
| 79 | +Serve both serialisations off one data structure; let `Accept` pick. | |
| 80 | + | |
| 81 | +**Upload:** `twine` multipart POST, Basic auth against a registry | |
| 82 | +token. | |
| 83 | + | |
| 84 | +**Consume:** `pip install --index-url https://<host>/pkg/pypi/<owner>/<repo>/simple/` | |
| 85 | + | |
| 86 | +## Phase 2 — Hex | |
| 87 | + | |
| 88 | +Officially supported, and dogfooding, since this is an Elixir project. | |
| 89 | + | |
| 90 | +**Reference behaviour** (`mix hex.registry build`): | |
| 91 | + | |
| 92 | +```sh | |
| 93 | +openssl genrsa -out private_key.pem | |
| 94 | +mkdir public | |
| 95 | +mix hex.registry build public --name=acme --private-key=private_key.pem | |
| 96 | +# public/ → public_key, names, versions, tarballs/ | |
| 97 | +``` | |
| 98 | + | |
| 99 | +Consumer: | |
| 100 | + | |
| 101 | +```sh | |
| 102 | +mix hex.repo add acme <url> --public-key=public/public_key | |
| 103 | +# then in deps: {:dep, "~> 1.0", repo: "acme"} | |
| 104 | +``` | |
| 105 | + | |
| 106 | +**Format (registry v2)** | |
| 107 | + | |
| 108 | +- Resources: `/names`, `/versions`, `/packages/<NAME>`, plus | |
| 109 | + `/tarballs/<NAME>-<VERSION>.tar` and `/public_key`. | |
| 110 | +- Protobuf-encoded, gzip-compressed. | |
| 111 | +- Every resource is wrapped in `Signed{payload, signature}`. | |
| 112 | +- **The signature is an RSA signature over the SHA-512 digest of the | |
| 113 | + payload** (both unencoded). | |
| 114 | +- Repositories are required to sign all registry resources. | |
| 115 | + | |
| 116 | +Reuses the RSA keypair machinery already in | |
| 117 | +`mix gitgud.registry.keygen`. | |
| 118 | + | |
| 119 | +**Two ways in** — shell out to `mix hex.registry build` per repo, or | |
| 120 | +generate the protobuf directly. Prefer the second: it makes publishing | |
| 121 | +a request rather than a rebuild. The first is only a faster start. | |
| 122 | + | |
| 123 | +## Phase 3 — Cargo | |
| 124 | + | |
| 125 | +**Sparse HTTP index** (`sparse+https://`, default since Rust 1.68). | |
| 126 | +Skip the git-index form — tempting because this is a git server, but | |
| 127 | +sparse is simpler and is the modern path. | |
| 128 | + | |
| 129 | +Directory prefix by lowercased name length: | |
| 130 | + | |
| 131 | +| Length | Path | | |
| 132 | +|---|---| | |
| 133 | +| 1 | `1/{name}` | | |
| 134 | +| 2 | `2/{name}` | | |
| 135 | +| 3 | `3/{first}/{name}` | | |
| 136 | +| 4+ | `{first2}/{next2}/{name}` — e.g. `cargo` → `ca/rg/cargo` | | |
| 137 | + | |
| 138 | +`config.json` at the index root: | |
| 139 | + | |
| 140 | +```json | |
| 141 | +{"dl": "…/api/v1/crates", "api": "…", "auth-required": true} | |
| 142 | +``` | |
| 143 | + | |
| 144 | +`dl` supports markers `{crate}`, `{version}`, `{prefix}`, | |
| 145 | +`{lowerprefix}`, `{sha256-checksum}`; with no markers, | |
| 146 | +`/{crate}/{version}/download` is appended. | |
| 147 | + | |
| 148 | +Each index file is newline-delimited JSON, one object per version: | |
| 149 | +`name`, `vers`, `deps[]` (`name`, `req`, `features`, `optional`, | |
| 150 | +`default_features`, `target`, `kind`, `registry`, `package`), `cksum` | |
| 151 | +(sha256), `features`, `features2`, `yanked`, `links`, `v`, | |
| 152 | +`rust_version`. | |
| 153 | + | |
| 154 | +- Only `yanked` may change after creation. | |
| 155 | +- `features2` **and `v: 2`** are required if any feature uses `dep:` or | |
| 156 | + `pkg?/feat` syntax. | |
| 157 | +- `rust_version` carries no operator (`1.60`, not `=1.60`). | |
| 158 | + | |
| 159 | +**Caching:** honour `ETag`/`If-None-Match` (preferred) or | |
| 160 | +`Last-Modified`/`If-Modified-Since`; return `304`. The index is read | |
| 161 | +constantly, so this matters. Missing crates → `404`/`410`/`451`. | |
| 162 | + | |
| 163 | +**Publish:** `PUT /api/v1/crates/new`, | |
| 164 | +`Content-Type: application/octet-stream`, token in `Authorization`. | |
| 165 | +Body framing: | |
| 166 | + | |
| 167 | +1. u32 little-endian — JSON metadata length | |
| 168 | +2. JSON metadata | |
| 169 | +3. u32 little-endian — `.crate` length | |
| 170 | +4. `.crate` bytes | |
| 171 | + | |
| 172 | +Success `{"warnings": {"invalid_categories": [], "invalid_badges": [], "other": []}}`; | |
| 173 | +errors `{"errors": [{"detail": "…"}]}`. | |
| 174 | + | |
| 175 | +Others: yank `DELETE /api/v1/crates/{n}/{v}/yank`, unyank `PUT …/unyank`, | |
| 176 | +owners `GET|PUT|DELETE /api/v1/crates/{n}/owners`, search | |
| 177 | +`GET /api/v1/crates?q=&per_page=`. These return `{"ok": true}`. | |
| 178 | + | |
| 179 | +## Phase 4 — apt, with a Sequoia NIF | |
| 180 | + | |
| 181 | +Last, because signing is the only real unknown. | |
| 182 | + | |
| 183 | +**Tooling correction:** `sequoia-sq` is the CLI. The library crate is | |
| 184 | +**`sequoia-openpgp`**, and that is what Rustler binds. Follow the | |
| 185 | +existing pattern — `native/gitgud_gix`, rustler 0.34, | |
| 186 | +`crate-type = ["cdylib"]` — with `native/gitgud_pgp` beside it exposing | |
| 187 | +roughly `sign_clearsign/2`, `sign_detached/2`, `generate_key/1`. | |
| 188 | + | |
| 189 | +**Decide the crypto backend first.** `sequoia-openpgp` defaults to | |
| 190 | +Nettle, which pulls `nettle`/`gmp` system libraries into the runtime | |
| 191 | +image — and this project ships k8s manifests. Evaluate `crypto-openssl` | |
| 192 | +or the pure-Rust `crypto-rust` backend before committing, so the NIF | |
| 193 | +does not drag native deps into the image. | |
| 194 | + | |
| 195 | +**Format** | |
| 196 | + | |
| 197 | +``` | |
| 198 | +dists/$DIST/InRelease ← clearsigned, preferred | |
| 199 | +dists/$DIST/Release + Release.gpg ← legacy, older clients | |
| 200 | +dists/$DIST/$COMP/binary-$ARCH/Packages{,.xz} | |
| 201 | +pool/$COMP/$p/$pkg/<name>_<ver>_<arch>.deb | |
| 202 | +``` | |
| 203 | + | |
| 204 | +- `Release` mandatory fields: Suite and/or Codename, Architectures, | |
| 205 | + Components, Date, SHA256. | |
| 206 | +- **Clients must not use MD5Sum/SHA1 for security** and must require | |
| 207 | + SHA256 or SHA512. SHA256 is the floor. | |
| 208 | +- `Packages` stanza needs `Package`, `Filename`, `Size`, `sha256` | |
| 209 | + (lowercase field names here, unlike `Release`'s `SHA256`). | |
| 210 | +- **`Description-md5` is mandatory as of APT 2.7.7.** | |
| 211 | +- Servers should offer **xz only**; clients are required to support it. | |
| 212 | +- Checksums and sizes must be provided for uncompressed variants too. | |
| 213 | + | |
| 214 | +Plus `.deb` control parsing: ar archive → `control.tar.{gz,xz,zst}` → | |
| 215 | +control stanza. | |
| 216 | + | |
| 217 | +**Fallback:** if the crypto backend question turns ugly, `aptly` | |
| 218 | +becomes the zot of apt and Phase 4 becomes an integration instead — | |
| 219 | +without disturbing Phases 1–3. | |
| 220 | + | |
| 221 | +--- | |
| 222 | + | |
| 223 | +## Order and reasoning | |
| 224 | + | |
| 225 | +**PyPI → Hex → Cargo → apt.** | |
| 226 | + | |
| 227 | +PyPI proves the substrate with no cryptography. Hex is near-free | |
| 228 | +afterwards and gets used immediately. Cargo is mechanical. apt is last | |
| 229 | +because the Sequoia NIF is the only genuine unknown, and it is the one | |
| 230 | +phase with a clean fallback. | |
| 231 | + | |
| 232 | +**Scope:** Phase 0+1 together are comparable to the PR review flow. All | |
| 233 | +four is considerably more. Land PyPI end-to-end and actually | |
| 234 | +`pip install` from it before committing to the rest. | |
| 235 | + | |
| 236 | +## Sources | |
| 237 | + | |
| 238 | +- [PEP 503](https://peps.python.org/pep-0503/) | |
| 239 | +- [PEP 691](https://peps.python.org/pep-0691/) | |
| 240 | +- [Cargo registry index](https://doc.rust-lang.org/cargo/reference/registry-index.html) | |
| 241 | +- [Cargo registry web API](https://doc.rust-lang.org/cargo/reference/registry-web-api.html) | |
| 242 | +- [Hex self-hosting](https://hex.pm/docs/self-hosting) | |
| 243 | +- [Hex registry v2](https://github.com/hexpm/specifications/blob/main/registry-v2.md) | |
| 244 | +- [DebianRepository/Format](https://wiki.debian.org/DebianRepository/Format) |
Parents: 7a1d662