neiam
/gitgud
Git Gud
public · Issues · Pulls · Labels · Forks · Compare · Actions success · Badges · Packages
⭐
1
Log in to mark this repository.
Document slice 75: package retention and disk usage
88a7933 · Gabriel Morell · 2026-09-12 14:22
Message
{commit_body(@commit)}
Files changed
modified
PROGRESS.md
+43
−0
@@ -12,6 +12,49 @@ intermediate "I was here when I stopped" state) lives in the
| 12 | 12 | |
| 13 | 13 | --- |
| 14 | 14 | |
| 15 | +## Slice 75 — Package retention, and where the disk went | |
| 16 | + | |
| 17 | +Storage only grew. Yanking deliberately keeps the bytes so existing | |
| 18 | +pins keep resolving, publishing is cheap, and CI publishes on every | |
| 19 | +push — so a registry left alone is a disk that fills. This is the other | |
| 20 | +half of slice 70. | |
| 21 | + | |
| 22 | +### The policy | |
| 23 | + | |
| 24 | +- **One policy per package, two modes**: `keep_last` (the newest N) or `quota` (what fits in a byte budget). Defaults are `keep_last`, N=24. | |
| 25 | +- **A package may only be stricter than the instance.** The instance policy is both a default and a ceiling: a team that wants only the last 3 of their nightlies can say so, and a package can't opt itself up to 5,000 because an operator set 24. Looser package settings aren't rejected on read, they simply stop applying — so lowering the ceiling tightens everything under it immediately rather than leaving rows quietly above the new limit. | |
| 26 | +- **Pins are the escape hatch, and they're capped.** A pinned version is never pruned, whatever the policy says — for the release that something you can't change still references. Capped per package (`packages.max_pins`, default 12), because an uncapped escape hatch is an off switch with extra steps. In quota mode pins still consume the budget: a package pinned up to its budget prunes everything else and then stops, which is the honest answer rather than pruning pins. | |
| 27 | +- **A policy may never empty a package.** In quota mode a single version larger than the whole budget would otherwise take everything with it, leaving a package row whose `latest_version` resolves to nothing. The newest version always stays. | |
| 28 | + | |
| 29 | +### Reporting before deleting | |
| 30 | + | |
| 31 | +- **`packages.retention_enforced` is off by default**, and the nightly worker runs anyway — computing and logging what it *would* free. An instance upgrading into this slice has years of versions the default policy would prune on its first pass, and "this would delete 412 versions" is the only honest way to ask an admin to turn it on. Given pushes to `gg` deploy straight to production, this also means the slice itself deletes nothing. | |
| 32 | +- The admin view shows that number before the toggle that acts on it, and a **Prune now** button (with the count and size in its confirmation) for an admin who has read it and wants it applied immediately. | |
| 33 | + | |
| 34 | +### Deleting bytes safely | |
| 35 | + | |
| 36 | +- **Storage keys are content-addressed** per `(repository, format, sha256)`, so two versions that published identical bytes — a reproducible wheel, the same `.deb` in two suites — share one object. Every object is refcounted against `package_files` before deletion, or pruning an old version would pull the bytes out from under a current one. | |
| 37 | +- **OCI bytes are zot's**, so `GitGud.Packages.Zot` deletes the manifest there first, using the same RS256 token zot already trusts, minted with a `delete` scope for that one repository. If zot refuses or is unreachable the mirror row **stays** — a row with no manifest is a package page advertising something nobody can pull. zot reclaims blobs on its own GC schedule, so success means "unreferenced", not "smaller already". | |
| 38 | +- One package's failure never stops the walk: a zot that is down should delay OCI pruning, not stop a PyPI registry from being tidied. | |
| 39 | + | |
| 40 | +### Where the disk went | |
| 41 | + | |
| 42 | +- **`GitGud.Packages.DiskUsage`** answers by format, by repository, by package and in total, each one aggregate query rather than a walk, scopeable to a repo or an org. `SUM` over a bigint is `numeric` in Postgres — which Ecto hands back as a `Decimal`, and `Decimal < 1024` is quietly false — so every sum is cast in the query rather than leaving each caller to find that out by arithmetic error. | |
| 43 | +- **Three views**: the repository's packages page grew per-format sizes, a total, pin buttons and a per-package policy form; `/orgs/:handle/packages` totals an org's registries (scoped to what the viewer can read, so the number is of things they could have opened themselves); `/admin/packages` has instance totals, the biggest packages and repositories, and every lever. | |
| 44 | +- Org-level policy was deliberately **not** added. Retention is set per package or instance-wide; a third place for the same decision to live is a third place for them to disagree. | |
| 45 | + | |
| 46 | +### A latent ordering bug, surfaced | |
| 47 | + | |
| 48 | +Adding an index on `(package_id, pushed_at)` broke a cargo test. Not a | |
| 49 | +flake: `pushed_at` has second granularity, two versions published in | |
| 50 | +the same second tie, and `list_versions/1` had no tiebreaker — so the | |
| 51 | +order came from whatever plan Postgres chose, and adding an index | |
| 52 | +changed it. Cargo's index document is order-sensitive and cached by | |
| 53 | +clients. Fixed where it was wrong (an `id` tiebreaker in the query and | |
| 54 | +in cargo's own sort), not in the test. | |
| 55 | + | |
| 56 | +--- | |
| 57 | + | |
| 15 | 58 | ## Slice 74 — Reconcile divergence in the browser |
| 16 | 59 | |
| 17 | 60 | The 3-way merge editor the roadmap had been carrying, plus the two |
modified
README.md
+25
−3
@@ -55,6 +55,10 @@ reinvent them.
| 55 | 55 | - **Theme-aware identicon avatars** — deterministic 5×5 mirrored grid, fills reference daisyUI CSS variables so swapping themes recolors every avatar live. No upload pipeline yet; the identicon is the avatar. |
| 56 | 56 | - **Pinned repos** — drag-to-reorder grid, up to 7 per user / per org, includes repos from orgs you're a member of. Removed-from-org? Your personal pins of that org's repos clean themselves up. |
| 57 | 57 | - **Auto-created same-name profile README** — registering `alice` provisions `alice/alice` with a seeded "Hi, I'm @alice 👋 …" `README.md` on the default branch. Edit it like any other repo; the rendered output shows on the profile. |
| 58 | +- **Mobile** | |
| 59 | + - **Android client** in [`android/`](android/) — repo list with per-repo CI status, CI runs/jobs/logs with re-run and cancel, issue and PR viewing + commenting + reviewing + merging, wiki reading. Same Compose shell and nine OKLCh palettes as the web | |
| 60 | + - **Pairing by QR**: mint a token with the "Mobile app" scope bundle at `/users/settings/tokens` and scan the `gitgud://pair?…` code. The device *is* its token, so revoking the token signs the phone out | |
| 61 | + - **Per-surface API scopes** — `repo:read`, `ci:read`/`ci:write`, `issues:read`/`issues:write`, `pulls:read`/`pulls:write`, `wiki:read`/`wiki:write`. The mobile checkbox ticks the bundle; tokens carry the scopes individually, so one surface can be revoked without re-pairing. See [docs/mobile-api.md](docs/mobile-api.md) | |
| 58 | 62 | - **Account + org governance** |
| 59 | 63 | - **Registration modes**: `:open` / `:invite_only` (default) / `:closed`, configurable per-instance |
| 60 | 64 | - **Invites** — single-use tokens with create / list / revoke per user; UI at `/users/settings/invites` |
@@ -473,6 +477,26 @@ sends Basic with `__token__` in the username, `mix` and `cargo` send
| 473 | 477 | the bare token, apt reads `/etc/apt/auth.conf.d/`. A 401 carries a |
| 474 | 478 | Basic challenge, which `pip` and `apt` retry against. |
| 475 | 479 | |
| 480 | +### Retention, and what the disk is doing | |
| 481 | + | |
| 482 | +Registries only grow on their own, so each package has a retention | |
| 483 | +policy: keep the newest N versions (24 by default), or keep what fits | |
| 484 | +in a byte budget. A package may set its own, but only a **stricter** | |
| 485 | +one — the instance policy is both the default and the ceiling. | |
| 486 | + | |
| 487 | +Pinned versions are never pruned, capped per package (12 by default) so | |
| 488 | +the escape hatch can't quietly become an off switch. Storage objects | |
| 489 | +are refcounted before deletion, since identical bytes published twice | |
| 490 | +share one content-addressed object; OCI versions delete their manifest | |
| 491 | +in zot first, and keep the mirror row if zot refuses. | |
| 492 | + | |
| 493 | +**Nothing is deleted until an admin says so.** The nightly pass runs in | |
| 494 | +reporting mode out of the box and logs what it would free; | |
| 495 | +`/admin/packages` shows that number, the levers, and a prune-now button | |
| 496 | +next to it. Usage is visible at three levels — per repository on its | |
| 497 | +packages page, per organization at `/orgs/:handle/packages`, and | |
| 498 | +instance-wide in the admin view. | |
| 499 | + | |
| 476 | 500 | ### GitGud ships itself as a .deb |
| 477 | 501 | |
| 478 | 502 | The most honest test of a registry is using it, so CI builds an |
@@ -602,15 +626,13 @@ See [ROADMAP.md](./ROADMAP.md). Highlights:
| 602 | 626 | - Namespace-wide indexes (`_all`) exist only for apt so far — the |
| 603 | 627 | callback is optional on the Format behaviour, so PyPI and Cargo can |
| 604 | 628 | opt in without new routing |
| 605 | −- Package retention/GC — nothing prunes old versions yet, and yanking | |
| 606 | − deliberately keeps the bytes so existing pins keep resolving | |
| 607 | 629 | - Signing-key rotation for Hex and apt. Keys mint themselves on first |
| 608 | 630 | use and are re-wrapped by the `GitGud.Secrets` master-key rotation, |
| 609 | 631 | but there is no "roll the keypair" flow, because rolling it breaks |
| 610 | 632 | every consumer that pinned the old one |
| 611 | 633 | |
| 612 | 634 | See [PROGRESS.md](./PROGRESS.md) for the slice-by-slice history of how |
| 613 | −the forge was built, and what each shipped — currently through slice 74. | |
| 635 | +the forge was built, and what each shipped — currently through slice 75. | |
| 614 | 636 | |
| 615 | 637 | ## Project conventions |
| 616 | 638 |
modified
ROADMAP.md
+5
−5
@@ -9,10 +9,10 @@ See [PROGRESS.md](./PROGRESS.md) for what's done.
| 9 | 9 | |
| 10 | 10 | ## Now — in flight |
| 11 | 11 | |
| 12 | −Nothing in flight. Slice 74 (the in-browser merge editor) closed most | |
| 13 | −recently, on top of slice 73 (import adoption, reconcile-mode upstream | |
| 14 | −sync, the remaining gix NIF ops); slice 70 (private package registries) | |
| 15 | −is the substantial one before them. | |
| 12 | +Nothing in flight. Slice 75 (package retention + disk usage views) | |
| 13 | +closed most recently, on top of slice 74 (the in-browser merge editor) | |
| 14 | +and slice 73 (import adoption, reconcile-mode upstream sync, the | |
| 15 | +remaining gix NIF ops). | |
| 16 | 16 | |
| 17 | 17 | Federation feature set is closed through slice 42 (reviews over AP |
| 18 | 18 | shipped in slice 38, suggested-blocks in slice 38, actionable reports |
@@ -24,7 +24,6 @@ queue in slice 42). Remaining federation items are discretionary
| 24 | 24 | | Item | Why | Size | |
| 25 | 25 | |---|---|---| |
| 26 | 26 | | `${{ gitgud.token }}` in CI + docs | From **G Notes** below. The per-job token is ours, not GitHub's, and every registry and the OCI endpoint now accept it — the name should say so. Keep `${{ github.token }}` working: it is what `forgejo-runner` injects as `GITHUB_TOKEN`, and every existing workflow uses it | small | |
| 27 | −| Package retention / GC | Nothing prunes old package versions. Yanking deliberately keeps the bytes so existing pins keep resolving, which means storage only grows | medium | | |
| 28 | 27 | | Credentials for scheduled upstream sync | Slice 73 ships the scheduler, anonymous only: a private upstream reports "needs credentials" and waits for a human. Storing a per-repo credential is the open question (scope, rotation, who may read it) — `UpstreamSync.credentials_for/1` is the single seam it plugs into | small-medium | |
| 29 | 28 | | Namespace-wide indexes beyond apt | `namespace_index/3` is optional on `GitGud.Packages.Format` and only `Deb` implements it. A namespace-wide PyPI index is the same shape and needs no new routing | small-medium | |
| 30 | 29 |
@@ -59,6 +58,7 @@ These are documented in the auto-memory under `project-git-gud-moderation-deferr
| 59 | 58 | | Item | Notes | |
| 60 | 59 | |---|---| |
| 61 | 60 | | `root_commits/1`, `diff/4` + `merge_analysis/4` in the NIF | The ops still falling through to the Cli backend. Everything else is native as of slice 73; `merge_analysis/4` is Cli-only because `git merge-tree -z` already answers it exactly, where gix reports conflicts in a shape that needs mapping across the boundary | |
| 61 | +| Org-level package quotas | Slice 75 puts retention on the package, with the instance as ceiling — deliberately, since a third place for the policy is a third place for them to disagree. An org that wants one budget across its repositories needs a victim-selection rule across repos owned by different people, which is the actual hard part | medium | | |
| 62 | 62 | | Rename detection in the merge editor | `merge_analysis/4` reports what git reports, and git's rename detection is on by default — but the editor renders a rename conflict as the paths it names rather than as "this file moved". Worth a pass once someone hits one | |
| 63 | 63 | | Test fixture cleanup | `tmp/test/repos/` wiped on suite start; consider per-test isolation if parallelism grows | |
| 64 | 64 | | Documentation as code grows | This file + PROGRESS.md; append on each slice close | |
Parents: aa9e861