defmodule GitGud.Packages.Formats.Fdroid do
@moduledoc """
An F-Droid app repository, per git repository *and* per namespace.
# everything neiam publishes — one repo entry on the phone
https://<host>/pkg/fdroid/neiam/_all/repo
# or one repository on its own
https://<host>/pkg/fdroid/neiam/gitgud/repo
The trailing `/repo` is required, not stylistic — see `repo_url/1`.
Like the apt adapter, this is static documents over HTTP with one
signature on top. `_all` is the shape that matters: a phone should
get one repository for a publisher, not one per Android project, and
`Format.namespace_marker/0` exists for exactly this.
Paths below either root:
/entry.jar signed; what a v2 client fetches first
/entry.json the same content, unsigned — for debugging
/index-v2.json the index proper, verified by hash from entry
/index-v1.jar signed; the whole index, for older clients
/index-v1.json the same, unsigned
/icons/icon.png the namespace avatar
/icons/<pkg>.<code>.png an app's launcher icon
/<package>_<code>.apk
## Both index versions, because the failure is invisible
A client that can't speak v2 falls back to `index-v1.jar` *silently*,
after fetching a perfectly good `entry.jar` — and reports the 404 it
gets. So a repository with a flawless v2 index reads as "invalid
repository / NotFoundException", which points at nothing in
particular. Serving both is the only way that failure doesn't happen.
## Where the fields come from
An APK carries no metadata file, so everything is read out of the
binary at publish time by `Fdroid.Apk` — package name, version code
and name, SDK levels, permissions, ABIs, and the signing
certificate's SHA-256. See `docs/fdroid-registry.md` for the formats.
Per-app `name` and the launcher icon come from `resources.arsc`, via
`Fdroid.Arsc`: `android:label` and `android:icon` in the manifest are
resource ids, and the table is what maps them to a string and a file.
A release build makes that the only route — R8 rewrites every
resource path, so this forge's own app ships `res/o-.png` and no
`mipmap` directory to guess from. Where the table can't be read the
index falls back to the package name and no icon.
`summary` and `description` are still not read: they are F-Droid
metadata an APK doesn't carry. The repository description comes from
the git repository's own.
The *repository* icon is the namespace's avatar, via `GitGud.Avatars`
— an uploaded one if the org has set one, and the identicon the
profile page draws otherwise. Seeded on the namespace rather than the
repository: a repo under `neiam` shows neiam, which is what `_all`
shows anyway.
## signer, and why it is the load-bearing field
F-Droid records the SHA-256 of each APK's signing certificate and
refuses an update whose signer differs from what is installed. It is
the thing standing between a repository and someone replacing an app
on a phone. It is read from the APK rather than asserted.
## Timestamps
Milliseconds since the epoch, which is what F-Droid's index uses
throughout — seconds would be silently interpreted as 1970 and every
client would treat the repository as permanently stale.
"""
@behaviour GitGud.Packages.Format
alias GitGud.Packages
alias GitGud.Packages.Formats.Fdroid.Apk
alias GitGud.Avatars
alias GitGud.Packages.Formats.Fdroid.Jar
alias GitGud.Packages.PackageFile
alias GitGud.Repositories.Repository
alias GitGud.Repositories.Storage
@format "fdroid"
# The index-v2 format version F-Droid's own tooling emits.
@index_version 20_002
# index-v1's own version field. Unrelated to the above beyond looking
# similar — this one is the format the older client understands.
@index_v1_version 20_001
@impl true
def content_type(_filename), do: "application/vnd.android.package-archive"
# ── reads ────────────────────────────────────────────────────────────
@impl true
def index(repo, ctx), do: serve([repo], repo_label(repo), repo, strip_repo_prefix(ctx))
@impl true
def namespace_index(namespace, repos, ctx),
do: serve(repos, Storage.owner_handle(namespace), namespace, strip_repo_prefix(ctx))
@doc """
Drop the `repo/` or `fdroid/repo/` the client adds for itself.
F-Droid rewrites the URL you give it. `RepoUriGetter` appends
`/fdroid/repo` to any address that doesn't already end in `/repo` or
`/archive` — so `…/pkg/fdroid/neiam/_all` is fetched as
`…/pkg/fdroid/neiam/_all/fdroid/repo/entry.jar`, which 404s, which
the client reports as "invalid repository".
So `repo_url/1` advertises an address ending in `/repo`, which the
client leaves alone. This accepts both anyway: anyone who added the
bare URL before has the rewritten form stored as their repository
address and would otherwise never recover.
"""
def strip_repo_prefix(%{path: ["fdroid", "repo" | rest]} = ctx), do: %{ctx | path: rest}
def strip_repo_prefix(%{path: ["repo" | rest]} = ctx), do: %{ctx | path: rest}
def strip_repo_prefix(ctx), do: ctx
@doc """
The address to hand a client, which is `base_url` plus `/repo`.
The suffix is not decoration. Without it F-Droid rewrites the address
and fetches a path that doesn't exist; with it, `pathSegments.last()
== "repo"` and the client uses what it was given.
"""
@spec repo_url(String.t()) :: String.t()
def repo_url(base_url), do: String.trim_trailing(base_url, "/") <> "/repo"
defp serve(repos, label, scope, ctx)
defp serve(repos, label, scope, %{path: ["index-v2.json"]} = ctx) do
{:ok, "application/json", index_v2(repos, label, ctx, scope)}
end
# The unsigned entry, which is what `curl` wants and what no client
# will accept. Kept because "is the index right" and "is the
# signature right" are different questions and debugging them
# together is miserable.
defp serve(repos, label, scope, %{path: ["entry.json"]} = ctx) do
{:ok, "application/json", entry(repos, label, ctx, scope)}
end
defp serve(repos, label, scope, %{path: ["entry.jar"]} = ctx) do
signed_entry(repos, label, scope, ctx)
end
# index-v1, for clients that don't speak v2.
#
# Not optional in practice. A client that can't find this falls back
# to it *silently* after fetching a perfectly good entry.jar, and
# reports the 404 — so a repository with a flawless v2 index reads as
# "invalid repository / NotFoundException", which points at nothing.
defp serve(repos, label, scope, %{path: ["index-v1.jar"]} = ctx) do
document = index_v1(repos, label, ctx, scope) |> IO.iodata_to_binary()
# SHA-256, not the SHA-1 that fdroidserver historically used here.
#
# A deliberate divergence, and measured: a SHA-1-signed JAR handed
# to a current verifier comes back "treated as unsigned, because it
# is signed with a weak algorithm that is now disabled". On a device
# that is a far worse failure than the 404 this path exists to fix —
# an index that looks present and unsigned rather than absent.
#
# SHA-1 was never a weakness for an index (second-preimage
# resistance is the property that matters), but the platforms have
# stopped caring about that distinction, so this follows them.
case Jar.build(scope, [{"index-v1.json", document}], digest: :sha256) do
{:ok, jar, _fingerprint} -> {:ok, "application/java-archive", jar}
{:error, reason} -> {:error, {500, "could not sign the index: #{inspect(reason)}"}}
end
end
# The unsigned form, for the same reason entry.json is served.
defp serve(repos, label, scope, %{path: ["index-v1.json"]} = ctx),
do: {:ok, "application/json", index_v1(repos, label, ctx, scope)}
# The repository icon: the namespace's identicon, the same grid the
# profile page draws, baked into a PNG because a client has no
# stylesheet to resolve the web avatar's CSS variables against.
defp serve(repos, _label, scope, %{path: ["icons", filename]}) do
{content_type, body} = icon(scope)
if filename == icon_filename(content_type) do
{:ok, content_type, body}
else
case find_app_icon(repos, filename) do
{app_type, app_icon} -> {:ok, app_type, app_icon}
nil -> {:error, {404, "no such icon"}}
end
end
end
# An APK, by the filename the index points at.
defp serve(repos, _label, _scope, %{path: [filename]}) do
case Enum.find(apks(repos), &(&1.file.filename == filename)) do
%{file: %PackageFile{} = file} -> {:blob, file}
nil -> {:error, {404, "no such file in this repository"}}
end
end
defp serve(_repos, _label, _scope, %{path: []}),
do: {:error, {404, "add this URL as a repository in F-Droid; there is no index page"}}
defp serve(_repos, _label, _scope, _ctx),
do: {:error, {404, "not found in this repository"}}
# ── index documents ──────────────────────────────────────────────────
@doc false
# index-v2: a repo block, then every package keyed by package name,
# each with its versions keyed by the APK's sha256.
def index_v2(repos, label, ctx), do: index_v2(repos, label, ctx, nil)
@doc false
def index_v2(repos, label, ctx, scope) do
apks = apks(repos)
packages = Enum.group_by(apks, & &1.info["package"])
Jason.encode_to_iodata!(%{
"repo" => %{
"name" => %{"en-US" => label},
"description" => %{"en-US" => description(repos, label)},
"address" => repo_url(ctx.base_url),
"timestamp" => repo_timestamp(apks),
"icon" => %{"en-US" => icon_file(scope)},
"categories" => %{},
"mirrors" => []
},
"packages" => Map.new(packages, fn {name, group} -> {name, package_entry(name, group)} end)
})
end
# index-v2 records every file by hash and size, the icon included, so
# a client can tell a changed icon from an unchanged one without
# fetching it. Deterministic rendering is what makes that work.
defp icon_file(scope) do
{content_type, body} = icon(scope)
%{
"name" => "/icons/" <> icon_filename(content_type),
"sha256" => :sha256 |> :crypto.hash(body) |> Base.encode16(case: :lower),
"size" => byte_size(body)
}
end
@doc """
Everything the index knows about an APK: what the manifest and
resource table say, plus the launcher icon, stored.
Public because `mix gitgud.backfill_fdroid_icons` re-reads APKs that
were published before the adapter could resolve an icon, and a
backfill that computed metadata its own way would be a second
implementation to keep in step.
"""
@spec apk_metadata(map(), binary()) :: map()
def apk_metadata(info, body) when is_map(info) and is_binary(body) do
info |> string_keys() |> Map.merge(store_icon(body))
end
# The launcher icon pulled out of the APK at publish time, keyed by
# content so re-publishing the same image stores it once.
defp store_icon(body) do
case Apk.icon(body) do
{:ok, bytes, ext} ->
sha256 = :sha256 |> :crypto.hash(bytes) |> Base.encode16(case: :lower)
key = "fdroid-icons/#{sha256}"
type = icon_content_type(ext)
case GitGud.Storage.put(Packages.artifact_bucket(), key, bytes, content_type: type) do
:ok ->
%{
"icon_key" => key,
"icon_sha256" => sha256,
"icon_size" => byte_size(bytes),
"icon_ext" => ext
}
# An icon is decoration. Failing the publish over one would
# trade a working release for a missing picture.
{:error, _} ->
%{}
end
:none ->
%{}
end
end
defp icon_content_type("webp"), do: "image/webp"
defp icon_content_type("jpg"), do: "image/jpeg"
defp icon_content_type(_), do: "image/png"
# Versions published before icons carried their format are PNG: that
# is what the old extractor could produce.
defp icon_ext(info), do: info["icon_ext"] || "png"
# `<package>.<versionCode>.png`, which is what fdroidserver emits and
# what a v1 client expects to find under /icons/ — with whatever
# extension the APK actually carried, since AGP has emitted WebP by
# default for years and a client renders what the name claims.
defp app_icon_name(%{info: info}) do
if info["icon_key"], do: "#{info["package"]}.#{info["version_code"]}.#{icon_ext(info)}"
end
# `android:label` off the newest APK, resolved through its resource
# table at publish time. Falls back to the package name, which is what
# every version published before labels were read still carries.
defp app_label(name, group) do
newest = Enum.max_by(group, & &1.info["version_code"])
case newest.info["label"] do
label when is_binary(label) -> if String.trim(label) == "", do: name, else: label
_ -> name
end
end
defp app_icon_file(group) do
newest = Enum.max_by(group, & &1.info["version_code"])
info = newest.info
if info["icon_key"] do
%{
"en-US" => %{
"name" => "/icons/" <> app_icon_name(newest),
"sha256" => info["icon_sha256"],
"size" => info["icon_size"]
}
}
end
end
defp find_app_icon(repos, filename) do
Enum.find_value(apks(repos), fn apk ->
if apk.info["icon_key"] && app_icon_name(apk) == filename do
case GitGud.Storage.get(Packages.artifact_bucket(), apk.info["icon_key"]) do
{:ok, bytes} -> {icon_content_type(icon_ext(apk.info)), bytes}
{:error, _} -> nil
end
end
end)
end
# Whatever the namespace's avatar is — an upload if there is one,
# the identicon otherwise. `Avatars.render/1` makes that one call
# rather than a branch here.
defp icon(scope),
do: scope |> icon_owner() |> Avatars.render() |> then(fn {:ok, t, b} -> {t, b} end)
defp icon_owner(nil), do: %GitGud.Accounts.User{handle: "gitgud"}
defp icon_owner(scope), do: GitGud.Packages.Signing.namespace(scope)
# The index names the icon by filename, so the extension has to match
# what is actually served — a JPEG at `icon.png` is a file every
# client will fetch and some will refuse to draw.
defp icon_filename("image/jpeg"), do: "icon.jpg"
defp icon_filename("image/gif"), do: "icon.gif"
defp icon_filename("image/webp"), do: "icon.webp"
defp icon_filename(_png), do: "icon.png"
defp package_entry(name, group) do
added = group |> Enum.map(& &1.added) |> Enum.min()
updated = group |> Enum.map(& &1.added) |> Enum.max()
%{
"metadata" =>
%{
"added" => added,
"lastUpdated" => updated,
"name" => %{"en-US" => app_label(name, group)},
"icon" => app_icon_file(group)
}
|> drop_nils(),
"versions" => Map.new(group, fn apk -> {apk.file.sha256, version_entry(apk)} end)
}
end
defp version_entry(%{file: file, info: info, added: added}) do
%{
"added" => added,
"file" => %{
# Leading slash: F-Droid resolves it against the repo address.
"name" => "/" <> file.filename,
"sha256" => file.sha256,
"size" => file.size
},
"manifest" =>
%{
"versionCode" => info["version_code"],
"versionName" => info["version_name"],
"usesSdk" => %{
"minSdkVersion" => info["min_sdk"],
"targetSdkVersion" => info["target_sdk"]
},
# A list because an APK signed by a rotated key chain has
# several; we read the current one, so it is a list of one.
"signer" => %{"sha256" => [info["signer"]]},
"usesPermission" =>
Enum.map(info["permissions"] || [], &%{"name" => &1, "maxSdkVersion" => nil}),
"nativecode" => info["nativecode"] || []
}
|> drop_nils()
}
end
@doc false
# index-v1: one flat document rather than v2's entry-plus-index. The
# shapes differ enough that translating between them would be more
# code than emitting both from the same rows.
def index_v1(repos, label, ctx), do: index_v1(repos, label, ctx, nil)
@doc false
def index_v1(repos, label, ctx, scope) do
apks = apks(repos)
by_package = Enum.group_by(apks, & &1.info["package"])
Jason.encode_to_iodata!(%{
"repo" => %{
"timestamp" => repo_timestamp(apks),
"version" => @index_v1_version,
"name" => label,
"icon" => icon_filename(elem(icon(scope), 0)),
"address" => repo_url(ctx.base_url),
"description" => description(repos, label)
},
# Present and empty. The client reads them unconditionally.
"requests" => %{"install" => [], "uninstall" => []},
"apps" => Enum.map(by_package, fn {name, group} -> v1_app(name, group) end),
"packages" =>
Map.new(by_package, fn {name, group} ->
# Newest first: the client takes the head as the suggested
# version, so the order is load-bearing rather than cosmetic.
{name,
group
|> Enum.sort_by(& &1.info["version_code"], :desc)
|> Enum.map(&v1_package/1)}
end)
})
end
defp v1_app(name, group) do
added = group |> Enum.map(& &1.added) |> Enum.min()
updated = group |> Enum.map(& &1.added) |> Enum.max()
newest = Enum.max_by(group, & &1.info["version_code"])
%{
"packageName" => name,
"name" => app_label(name, group),
"summary" => "",
"description" => "",
"license" => "Unknown",
"categories" => [],
"added" => added,
"lastUpdated" => updated,
"suggestedVersionCode" => to_string(newest.info["version_code"]),
"suggestedVersionName" => newest.info["version_name"],
# v1 names the icon relative to <address>/icons/, so a bare
# filename rather than v2's path-and-hash object.
"icon" => app_icon_name(newest)
}
|> drop_nils()
end
defp v1_package(%{file: file, info: info, added: added}) do
%{
"apkName" => file.filename,
"hash" => file.sha256,
"hashType" => "sha256",
"versionCode" => info["version_code"],
"versionName" => info["version_name"],
"size" => file.size,
"minSdkVersion" => info["min_sdk"],
"targetSdkVersion" => info["target_sdk"],
# The certificate again. v1 spells it `signer`, and a client
# refuses an update whose signer differs from what's installed.
"signer" => info["signer"],
"added" => added,
"nativecode" => info["nativecode"] || [],
# `[name, maxSdkVersion]` pairs, which is the shape v1 uses —
# not v2's list of objects.
"uses-permission" => Enum.map(info["permissions"] || [], &[&1, nil])
}
|> drop_nils()
end
@doc false
# entry.json points at index-v2.json and pins it by hash, so the
# signature on entry.jar covers the index transitively without the
# index itself having to be wrapped in a JAR.
def entry(repos, label, ctx), do: entry(repos, label, ctx, nil)
@doc false
def entry(repos, label, ctx, scope) do
apks = apks(repos)
document = index_v2(repos, label, ctx, scope) |> IO.iodata_to_binary()
Jason.encode_to_iodata!(%{
"timestamp" => repo_timestamp(apks),
"version" => @index_version,
"index" => %{
"name" => "/index-v2.json",
"sha256" => :sha256 |> :crypto.hash(document) |> Base.encode16(case: :lower),
"size" => byte_size(document),
"numPackages" => apks |> Enum.map(& &1.info["package"]) |> Enum.uniq() |> length()
}
})
end
# The signed entry: entry.json wrapped in a JAR whose signature the
# client verifies against the fingerprint it pinned when the
# repository was added.
defp signed_entry(repos, label, scope, ctx) do
document = entry(repos, label, ctx, scope) |> IO.iodata_to_binary()
case Jar.build(scope, [{"entry.json", document}]) do
{:ok, jar, _fingerprint} ->
{:ok, "application/java-archive", jar}
{:error, reason} ->
{:error, {500, "could not sign the index: #{inspect(reason)}"}}
end
end
@doc """
The SHA-256 of this repository's signing certificate — the
`?fingerprint=` a client pins when adding the repository.
Generating the key if there isn't one yet, because the first person
to ask is usually the packages page trying to print the URL.
"""
@spec fingerprint(struct()) :: {:ok, String.t()} | {:error, term()}
def fingerprint(scope) do
with {:ok, der} <- Jar.certificate(scope), do: {:ok, Jar.fingerprint(der)}
end
# ── writes ───────────────────────────────────────────────────────────
@impl true
def store(repo, ctx, body)
def store(repo, %{method: "POST", path: path} = ctx, body)
when is_binary(body) and (path == ["upload"] or path == []) do
with {:ok, info} <- Apk.read(body) do
# Extracted once, here, rather than on every icon request — the
# alternative is pulling a 19 MB APK out of storage to serve
# 30 KB of PNG.
metadata = apk_metadata(info, body)
attrs = %{
filename: Apk.filename(info),
body: body,
content_type: content_type(nil),
metadata: metadata,
version_metadata: metadata,
uploader: uploader(ctx)
}
case Packages.publish_file(
repo,
@format,
info.package,
version_label(info),
attrs
) do
{:ok, _package, _version, _file} ->
{:ok, "application/json",
Jason.encode_to_iodata!(%{
"package" => info.package,
"versionCode" => info.version_code,
"versionName" => info.version_name,
"signer" => info.signer,
"filename" => Apk.filename(info)
})}
# The generic "a file with that name is already published"
# doesn't point anywhere useful here, and this has exactly one
# cause worth naming: a versionCode that didn't move. Android
# refuses an update that doesn't increase it and F-Droid orders
# by it, so a repeated one is a bug in the build rather than a
# publishing accident.
{:error, :file_exists} ->
{:error,
{409,
"#{Apk.filename(info)} is already published with different content — " <>
"versionCode #{info.version_code} has been used before. Bump it: Android " <>
"won't install an update whose versionCode hasn't increased."}}
{:error, reason} ->
{:error, {400, inspect(reason)}}
end
end
end
def store(_repo, %{method: "POST"}, _body),
do: {:error, {400, "POST the raw .apk to /upload"}}
def store(_repo, %{method: method}, _body),
do: {:error, {405, "#{method} is not supported by the F-Droid repository"}}
# ── internals ────────────────────────────────────────────────────────
# Every published APK across `repos`, with its decoded metadata.
defp apks(repos) do
Enum.flat_map(repos, fn %Repository{} = repo ->
repo
|> Packages.list_packages(@format)
|> Enum.flat_map(&Packages.list_package_files/1)
|> Enum.flat_map(fn %PackageFile{} = file ->
info = file.metadata || %{}
# A file whose metadata predates this adapter, or was written by
# something else, has no package name and cannot be indexed.
# Skipping beats emitting a null key that breaks the whole
# document for every other app in the repository.
if is_binary(info["package"]) do
[%{file: file, repo_name: repo.name, info: info, added: added_at(file)}]
else
[]
end
end)
end)
|> Enum.sort_by(&{&1.info["package"], &1.info["version_code"]})
end
defp added_at(%PackageFile{inserted_at: %DateTime{} = at}),
do: DateTime.to_unix(at, :millisecond)
defp added_at(_), do: 0
@doc false
# When this repository last changed — the newest publish, not `now`.
#
# It has to be derived from content, for two reasons. `entry.json`
# pins `index-v2.json` by hash, so a document that differs per
# request could never match the hash a client was given, and every
# fetch would look like tampering. And F-Droid compares this against
# the timestamp it already holds to refuse rollbacks, which only
# means anything if it moves when the content does and not otherwise.
#
# An empty repository reports 0 rather than the current time: it has
# never been updated, and saying "just now" every time anyone asks is
# both false and a rollback the next real publish has to climb out of.
defp repo_timestamp([]), do: 0
defp repo_timestamp(apks), do: apks |> Enum.map(& &1.added) |> Enum.max()
# `versionName (versionCode)` — the version string a human reads in
# the packages UI. versionCode alone sorts correctly but tells you
# nothing; versionName alone is not unique.
defp version_label(%{version_name: nil, version_code: code}), do: to_string(code)
defp version_label(%{version_name: name, version_code: code}), do: "#{name} (#{code})"
# Preloaded here rather than assumed: the controller does preload,
# but a missing association should not be a 500 from an index.
defp repo_label(%Repository{} = repo) do
repo = GitGud.Repo.preload(repo, [:owner, :organization])
"#{Storage.repo_handle(repo)}/#{repo.name}"
end
defp description([%Repository{description: d}], _label) when is_binary(d) and d != "", do: d
defp description(_repos, label), do: "Android packages published by #{label}."
defp string_keys(info), do: Map.new(info, fn {k, v} -> {to_string(k), stringify(v)} end)
# `signer_scheme` is an atom; everything else is already JSON-shaped.
defp stringify(value) when is_atom(value) and not is_boolean(value) and not is_nil(value),
do: Atom.to_string(value)
defp stringify(value), do: value
defp drop_nils(map), do: Map.reject(map, fn {_k, v} -> is_nil(v) end)
defp uploader(%{principal: {:user, user}}), do: user
defp uploader(_), do: nil
end
neiam
/gitgud
Git Gud
public · Issues · Pulls · Labels · Forks · Compare · Actions success · Badges · Packages
⭐
1
Log in to mark this repository.
26.0 KiB · text
History
8808a78