neiam /gitgud
Git Gud
public · Issues · Pulls · Labels · Forks · Compare · Actions success · Packages
⭐
Log in to mark this repository.
Import repositories from GitHub, GitLab, or any git URL
69753e1 · Gabriel Morell · 2026-09-09 20:14
Message
{commit_body(@commit)}
Files changed
modified
PROGRESS.md
+47
−0
@@ -12,6 +12,53 @@ intermediate "I was here when I stopped" state) lives in the
| 12 | 12 | |
| 13 | 13 | --- |
| 14 | 14 | |
| 15 | +## Slice 59 — Repository import (GitHub / GitLab / any git URL) | |
| 16 | + | |
| 17 | +Bulk-imports repositories from another forge, plus one-off imports of an arbitrary git URL. Git data only: branches, tags and notes come across via `git clone --mirror`; issues and pull requests do not. | |
| 18 | + | |
| 19 | +### Discovery | |
| 20 | + | |
| 21 | +- **`GitGud.Imports.Provider` behaviour** normalizes "list the repos under this owner" across forges. Three adapters, registered in an ordered list that also drives the UI picker: | |
| 22 | + - `Providers.GitHub` — `/orgs/:owner/repos` with a 404 fallback to `/users/:owner/repos`, so orgs and users both just work. `page`/`per_page=100` pagination to exhaustion. Bearer auth. `base_url` is the API root, so GitHub Enterprise is a URL change (`https://host/api/v3`). | |
| 23 | + - `Providers.GitLab` — `/api/v4/groups/:id/projects?include_subgroups=true` falling back to `/api/v4/users/:id/projects`. Nested group paths (`acme/platform`) are encoded as one path segment. Pagination follows `x-next-page`, falling back to a full-page heuristic for keyset-paginated instances. `PRIVATE-TOKEN` auth. | |
| 24 | + - `Providers.GitUrl` — no API at all; parses a single http(s) URL into a one-element list so a one-off import gets the same batch record, progress, retries and re-sync as a bulk one. Assumes `private` (the closed default when nobody can tell us). | |
| 25 | +- **Discovery runs inline** in the LiveView, not in a job — it's a couple of API calls, and the operator sees the real list (names, visibility, fork/archived flags) before committing to anything. | |
| 26 | +- HTTP goes through `Req`, with `config :git_gud, :imports_req_options` merged in so tests stub the transport with a plug instead of hitting the network. | |
| 27 | + | |
| 28 | +### Batches | |
| 29 | + | |
| 30 | +- **`repository_imports` + `repository_import_items`** — one row per batch, one per remote repo. Free-text columns are `:text`, not `:string`; git's failure output and nested GitLab paths blow past varchar(255). | |
| 31 | +- **One Oban job per item** on a new narrow `imports` queue (concurrency 2, so a 200-repo import doesn't saturate the box or trip a rate limit). An unreachable repo fails alone instead of taking the batch with it. A name collision resolves to `skipped` on the first attempt rather than burning retries. | |
| 32 | +- **`settle/1`** recomputes batch status once nothing is pending or cloning; `cancel/1` skips pending items but lets in-flight clones finish rather than leaving a half-written bare repo on disk. | |
| 33 | +- **Name coercion** — GitLab project paths accept things GitGud's name format doesn't (leading dots, spaces). `Imports.target_name/1` coerces rather than rejects so one odd project can't stall a large import; the preview table lets you rename any row before starting. | |
| 34 | +- **Visibility** is either mirrored from the source or forced to one value, and is then clamped so an imported repo can never be more open than the org it lands in. | |
| 35 | + | |
| 36 | +### Credentials | |
| 37 | + | |
| 38 | +- **The provider token is stored AES-256-GCM encrypted** (`GitGud.Secrets` master key) only while the batch has work left; `settle`/`cancel` wipe it and stamp `token_cleared_at`. | |
| 39 | +- **The token never reaches a clone URL.** `git` gets it via a new `priv/scripts/gitgud-askpass` helper reading `GITGUD_GIT_USERNAME`/`GITGUD_GIT_PASSWORD` from the child environment, with `GIT_TERMINAL_PROMPT=0` so a worker with no tty fails fast instead of hanging on a prompt. Embedding it in the URL would have leaked it into `ps` output and into the bare repo's `remote.origin.url`. | |
| 40 | +- **URLs with embedded credentials are refused** at the import boundary, and `Repositories.sanitize_upstream_url/1` strips userinfo from whatever gets persisted as `upstream_url` — belt and braces for the older `clone_addr` API path. | |
| 41 | +- Clone failure output is scrubbed of the token before it's stored or shown. | |
| 42 | + | |
| 43 | +### Upstream re-sync | |
| 44 | + | |
| 45 | +- **`repositories.upstream_url` / `upstream_synced_at`** record where an imported repo came from. | |
| 46 | +- **`Repositories.sync_upstream/2`** force-fetches heads and tags and prunes refs deleted upstream — mirror semantics, so local pushes to a branch that also exists upstream are discarded. Documented in the UI, not just here. | |
| 47 | +- Ref changes are diffed before/after and **fed through `HookReceiver` as if they were a push**, so the ref cache, `pushed_at`, webhooks and CI all see the new commits without duplicating that logic. | |
| 48 | +- Exposed as a **"Fetch from upstream"** section in repo settings (only for imported repos), run via `start_async` so a large fetch doesn't block the LiveView. The import token isn't kept, so a private upstream asks for one again; it's used for that fetch only. | |
| 49 | + | |
| 50 | +### UI | |
| 51 | + | |
| 52 | +- `/repositories/import` (`ImportLive.New`) — two-step configure → select, reached from a link on the new-repo page. | |
| 53 | +- `/imports/:id` (`ImportLive.Show`) — live per-item progress over PubSub, with a cancel button. | |
| 54 | +- `/imports` (`ImportLive.Index`) — batch history. | |
| 55 | + | |
| 56 | +### Incidental fix | |
| 57 | + | |
| 58 | +- **The metrics listener no longer hardcodes port 9568.** It's `config :git_gud, :metrics_port` (`:disabled` under test). As shipped in the previous slice it made the test suite unrunnable whenever a dev server was up, since both wanted the same port. | |
| 59 | + | |
| 60 | +--- | |
| 61 | + | |
| 15 | 62 | ## Slice 58 — Org webhooks + CI dashboards |
| 16 | 63 | |
| 17 | 64 | Extends webhooks to org scope and adds two CI "where do my builds stand" views: an org-wide single pane of glass and a per-user feed. |
modified
README.md
+1
−0
@@ -31,6 +31,7 @@ external tools (Forgejo runner, Garage, zot) rather than reinvent them.
| 31 | 31 | - **Quote-reply** button on every comment — appends `> `-prefixed source with attribution |
| 32 | 32 | - Pull requests with mergeability probe, 3-way merge, cross-repo (fork) PRs, line-level diff inline |
| 33 | 33 | - Forks + fork-sync (fast-forward CAS) + compare-any-two-refs view |
| 34 | + - **Repository import** from a GitHub org/user, a GitLab group (subgroups included) or user, or any single git URL — `git clone --mirror` per repo on its own background job, with a live progress page and an on-demand "fetch from upstream" re-sync. Git data only; issues and PRs don't come across. Provider tokens are encrypted at rest, handed to `git` out-of-band (never in the clone URL), and wiped when the import finishes | |
| 34 | 35 | - Branch protection, deploy keys, webhooks, wikis |
| 35 | 36 | - **Labels** with HTML color picker + 16-swatch palette + live preview; **org-level labels** inherited by every repo unless the repo opts out (and re-opts in via a hidden-labels expander) |
| 36 | 37 | - **Profiles** |
modified
ROADMAP.md
+7
−4
@@ -9,10 +9,11 @@ See [PROGRESS.md](./PROGRESS.md) for what's done.
| 9 | 9 | |
| 10 | 10 | ## Now — in flight |
| 11 | 11 | |
| 12 | −Nothing in flight. Federation feature set is closed through slice 42 | |
| 13 | −(reviews over AP shipped in slice 38, suggested-blocks in slice 38, | |
| 14 | −actionable reports queue in slice 42). Remaining federation items are | |
| 15 | −discretionary (RFC 9421, reputation aging). | |
| 12 | +Nothing in flight. Slice 59 (repository import) closed most recently. | |
| 13 | +Federation feature set is closed through slice 42 (reviews over AP | |
| 14 | +shipped in slice 38, suggested-blocks in slice 38, actionable reports | |
| 15 | +queue in slice 42). Remaining federation items are discretionary | |
| 16 | +(RFC 9421, reputation aging). | |
| 16 | 17 | |
| 17 | 18 | ## Next — small, high impact |
| 18 | 19 |
@@ -24,6 +25,8 @@ item from "Soon" or "Maintenance" below.
| 24 | 25 | |
| 25 | 26 | | Item | Why | Size | |
| 26 | 27 | |---|---|---| |
| 28 | +| Import issues / PRs / wikis alongside git data | Slice 59 imports git data only. Issues need author mapping for users who don't exist locally; PRs need ref rewriting | large | | |
| 29 | +| Scheduled mirror sync | Re-sync is manual today. A cron would need retained credentials and a divergence policy | medium | | |
| 27 | 30 | | Resolve PR conflicts in-browser | Today a conflict means "fix it locally and push" — a 3-way merge editor would close the loop | medium-large | |
| 28 | 31 | | gix NIF: fill in log, commit, tree, diff_stats, merge_base, merge_tree | 10-100× perf for repo-page hot paths; Cli fallback works today | medium | |
| 29 | 32 |
modified
config/config.exs
+5
−1
@@ -37,7 +37,11 @@ config :git_gud, Oban,
| 37 | 37 | default: 10, |
| 38 | 38 | git_cache: 4, |
| 39 | 39 | webhooks: 8, |
| 40 | − ci: 4 | |
| 40 | + ci: 4, | |
| 41 | + # Imports shell out to `git clone` against a remote forge; keep the | |
| 42 | + # lane narrow so a big org import doesn't saturate the box or trip | |
| 43 | + # the provider's rate limit. | |
| 44 | + imports: 2 | |
| 41 | 45 | ], |
| 42 | 46 | plugins: [ |
| 43 | 47 | {Oban.Plugins.Cron, |
modified
config/test.exs
+4
−0
@@ -64,3 +64,7 @@ config :git_gud, GitGud.Storage.Disk,
| 64 | 64 | # Tests assert against an open federation graph; production defaults |
| 65 | 65 | # to allowlist (see GitGud.Federation moduledoc). |
| 66 | 66 | config :git_gud, GitGud.Federation, federation_mode: :open |
| 67 | + | |
| 68 | +# No metrics listener under test — it would collide with a dev server (or a | |
| 69 | +# second test run) already holding the port. | |
| 70 | +config :git_gud, metrics_port: :disabled |
modified
lib/git_gud/application.ex
+14
−1
Click to load diff…
added
lib/git_gud/imports.ex
+484
−0
Click to load diff…
added
lib/git_gud/imports/import.ex
+120
−0
Click to load diff…
added
lib/git_gud/imports/import_item.ex
+82
−0
Click to load diff…
added
lib/git_gud/imports/provider.ex
+94
−0
Click to load diff…
added
lib/git_gud/imports/providers/git_url.ex
+77
−0
Click to load diff…
added
lib/git_gud/imports/providers/github.ex
+92
−0
Click to load diff…
added
lib/git_gud/imports/providers/gitlab.ex
+105
−0
Click to load diff…
added
lib/git_gud/imports/workers/clone_repo.ex
+108
−0
Click to load diff…
modified
lib/git_gud/repositories.ex
+205
−9
Click to load diff…
modified
lib/git_gud/repositories/repository.ex
+5
−0
Click to load diff…
added
lib/git_gud_web/live/import_live/index.ex
+82
−0
Click to load diff…
added
lib/git_gud_web/live/import_live/new.ex
+525
−0
Click to load diff…
added
lib/git_gud_web/live/import_live/show.ex
+157
−0
Click to load diff…
modified
lib/git_gud_web/live/repo_live/index.ex
+17
−8
Click to load diff…
modified
lib/git_gud_web/live/repo_live/new.ex
+16
−0
Click to load diff…
modified
lib/git_gud_web/live/repo_live/settings.ex
+113
−0
Click to load diff…
modified
lib/git_gud_web/router.ex
+3
−0
Click to load diff…
added
priv/repo/migrations/20260909000000_create_repository_imports.exs
+69
−0
Click to load diff…
added
priv/scripts/gitgud-askpass
+12
−0
Click to load diff…
added
test/git_gud/imports_sync_test.exs
+279
−0
Click to load diff…
added
test/git_gud/imports_test.exs
+562
−0
Click to load diff…
added
test/git_gud_web/live/import_live/new_test.exs
+228
−0
Click to load diff…
added
test/git_gud_web/live/repo_live/new_import_link_test.exs
+47
−0
Click to load diff…
Parents: d2c66e6