Private package registries — plan

Hosting PyPI, Hex, Cargo and apt repositories inside the forge, the way the OCI registry is hosted alongside it today.

Spec details are quoted inline deliberately: they were the expensive part to establish, and an implementer shouldn’t have to re-derive them.

The framing

These are not four features. They are one substrate with four format adapters. What they share already exists:

Need	Already there
Blob storage	`GitGud.Storage` — `put/get/delete/presign_url` behaviour, backs LFS + CI artifacts
Namespacing	`Package` / `PackageVersion`, per repo
Auth tokens	`RegistryToken` (`ggrt_…`), hashed, `pull`/`push` scopes
UI	`/r/:owner/:name/packages`

What differs per format is only the index shape and the upload framing. That is a @behaviour with four impls, the same move GitGud.Storage already makes for backends.

Why this differs from the OCI decision: zot is embedded because the distribution spec is a genuine protocol — manifests, blob dedup, cross-mounting, GC, conformance. An apt repository is static files over plain HTTP. There is no daemon to embed. The only hard part is OpenPGP signing, which OTP cannot do.

Visibility, and publishing out of a repository that has none

Packages follow the repository. A private or internal repository serves nothing to an anonymous client and is absent from the namespace-wide _all index, which is what GitGud.Packages.Auth.authorize/3 says and what the registry controller and the packages dashboard both filter through.

That is right for a Python package whose source is private. It is wrong for an Android client: the APK is meant to be on a phone, an F-Droid client is anonymous, and the repository holding the code is not going public to make that happen.

So a repository can name formats it publishes anyway — public_package_formats, set per repository in its settings, empty by default. A format in that list is readable by anyone; everything else is unchanged:

Reads only. Publishing still needs a token or the job’s own credential. authorize/4 widens :read and delegates every other action to authorize/3.
Only the formats named. Ticking fdroid does not open the raw bucket next to it. This is why it is a list and not a boolean: the answer genuinely differs by format.
Only packages. Nothing outside the package registry consults it. The repository stays as private as it was, including its source, its issues, and its CI.

The namespace _all index and the dashboard both filter per format rather than per repository, so a repository that publishes fdroid appears in that index and in no other.

UI surface

One page per repo listing what has been published and is installable — extending /r/:owner/:name/packages from “container images” to “artifacts, by format”, each with its versions and a copy-pasteable install line.

Each package can carry its own summary and description (GitGud.Packages.About). An admin sets them on that page; formats whose artifacts carry nothing (F-Droid, raw) also take them at upload as X-Package-Summary / X-Package-Description. Where nothing explicit is set, a format’s own words fill in: a wheel’s Summary, a crate’s description, a .deb’s synopsis. The org’s packages page (/orgs/:handle/packages) shows the result as cards with icons, beside tabs for the namespace-wide registry URLs and for storage.

Phase 0 — substrate

GitGud.Packages.Format behaviour: index/2, store/3, content_type/1.
format column on Package (oci | pypi | hex | cargo | deb); backfill existing rows to oci.
Generalise RegistryToken past OCI. The token itself is already format-agnostic — only the auth header differs per ecosystem.
Route /pkg/:format/:owner/:name/*rest.
Nothing user-visible ships. Everything after is an adapter.

Phase 1 — PyPI

Smallest, no cryptography, proves the substrate.

PEP 503 (Simple Repository API)

Normalisation, exactly: re.sub(r"[-_.]+", "-", name).lower(). Apply before any lookup.
/simple/ lists projects; /simple/<normalized>/ lists files.
Anchor text is the filename; href may carry #sha256=<hex>.
Trailing slashes are mandatory. Unslashed URLs must redirect.
Clients request normalised URLs directly; servers may redirect unnormalised ones.

PEP 691 (JSON Simple API)

Content negotiation on Accept; respond with Content-Type:
- application/vnd.pypi.simple.v1+json
- application/vnd.pypi.simple.v1+html
- text/html is a legacy alias for v1+html
- 406 if nothing matches
Root: {"meta": {"api-version": "1.0"}, "projects": [{"name": …}]}
Project: {"meta": …, "name": …, "files": [...]}
File entry required fields: filename, url, hashes — hashes must be present even when empty. sha256 preferred.
Optional: requires-python, yanked, dist-info-metadata, gpg-sig.

Serve both serialisations off one data structure; let Accept pick.

Upload: twine multipart POST, Basic auth against a registry token.

Consume: pip install --index-url https://<host>/pkg/pypi/<owner>/<repo>/simple/

Phase 2 — Hex

Officially supported, and dogfooding, since this is an Elixir project.

Reference behaviour (mix hex.registry build):

openssl genrsa -out private_key.pem
mkdir public
mix hex.registry build public --name=acme --private-key=private_key.pem
# public/ → public_key, names, versions, tarballs/

Consumer:

mix hex.repo add acme <url> --public-key=public/public_key
# then in deps: {:dep, "~> 1.0", repo: "acme"}

Format (registry v2)

Resources: /names, /versions, /packages/<NAME>, plus /tarballs/<NAME>-<VERSION>.tar and /public_key.
Protobuf-encoded, gzip-compressed.
Every resource is wrapped in Signed{payload, signature}.
The signature is an RSA signature over the SHA-512 digest of the payload (both unencoded).
Repositories are required to sign all registry resources.

Reuses the RSA keypair machinery already in mix gitgud.registry.keygen.

Two ways in — shell out to mix hex.registry build per repo, or generate the protobuf directly. Prefer the second: it makes publishing a request rather than a rebuild. The first is only a faster start.

Phase 3 — Cargo

Sparse HTTP index (sparse+https://, default since Rust 1.68). Skip the git-index form — tempting because this is a git server, but sparse is simpler and is the modern path.

Directory prefix by lowercased name length:

Length	Path
1	`1/{name}`
2	`2/{name}`
3	`3/{first}/{name}`
4+	`{first2}/{next2}/{name}` — e.g. `cargo` → `ca/rg/cargo`

config.json at the index root:

{"dl": "…/api/v1/crates", "api": "…", "auth-required": true}

dl supports markers {crate}, {version}, {prefix}, {lowerprefix}, {sha256-checksum}; with no markers, /{crate}/{version}/download is appended.

Each index file is newline-delimited JSON, one object per version: name, vers, deps[] (name, req, features, optional, default_features, target, kind, registry, package), cksum (sha256), features, features2, yanked, links, v, rust_version.

Only yanked may change after creation.
features2 and v: 2 are required if any feature uses dep: or pkg?/feat syntax.
rust_version carries no operator (1.60, not =1.60).

Caching: honour ETag/If-None-Match (preferred) or Last-Modified/If-Modified-Since; return 304. The index is read constantly, so this matters. Missing crates → 404/410/451.

Publish: PUT /api/v1/crates/new, Content-Type: application/octet-stream, token in Authorization. Body framing:

u32 little-endian — JSON metadata length
JSON metadata
u32 little-endian — .crate length
.crate bytes

Success {"warnings": {"invalid_categories": [], "invalid_badges": [], "other": []}}; errors {"errors": [{"detail": "…"}]}.

Others: yank DELETE /api/v1/crates/{n}/{v}/yank, unyank PUT …/unyank, owners GET|PUT|DELETE /api/v1/crates/{n}/owners, search GET /api/v1/crates?q=&per_page=. These return {"ok": true}.

Phase 4 — apt, with a Sequoia NIF

Last, because signing is the only real unknown.

Tooling correction: sequoia-sq is the CLI. The library crate is sequoia-openpgp, and that is what Rustler binds. Follow the existing pattern — native/gitgud_gix, rustler 0.34, crate-type = ["cdylib"] — with native/gitgud_pgp beside it exposing roughly sign_clearsign/2, sign_detached/2, generate_key/1.

Decide the crypto backend first. sequoia-openpgp defaults to Nettle, which pulls nettle/gmp system libraries into the runtime image — and this project ships k8s manifests. Evaluate crypto-openssl or the pure-Rust crypto-rust backend before committing, so the NIF does not drag native deps into the image.

Format

dists/$DIST/InRelease                            ← clearsigned, preferred
dists/$DIST/Release + Release.gpg                ← legacy, older clients
dists/$DIST/$COMP/binary-$ARCH/Packages{,.xz}
pool/$COMP/$p/$pkg/<name>_<ver>_<arch>.deb

Release mandatory fields: Suite and/or Codename, Architectures, Components, Date, SHA256.
Clients must not use MD5Sum/SHA1 for security and must require SHA256 or SHA512. SHA256 is the floor.
Packages stanza needs Package, Filename, Size, sha256 (lowercase field names here, unlike Release’s SHA256).
Description-md5 is mandatory as of APT 2.7.7.
Servers should offer xz only; clients are required to support it.
Checksums and sizes must be provided for uncompressed variants too.

Plus .deb control parsing: ar archive → control.tar.{gz,xz,zst} → control stanza.

Fallback: if the crypto backend question turns ugly, aptly becomes the zot of apt and Phase 4 becomes an integration instead — without disturbing Phases 1–3.

Order and reasoning

PyPI → Hex → Cargo → apt.

PyPI proves the substrate with no cryptography. Hex is near-free afterwards and gets used immediately. Cargo is mechanical. apt is last because the Sequoia NIF is the only genuine unknown, and it is the one phase with a clean fallback.

Scope: Phase 0+1 together are comparable to the PR review flow. All four is considerably more. Land PyPI end-to-end and actually pip install from it before committing to the rest.

neiam /gitgud