Add aircraft/exchange/market sources, IMAP mail ingest, fix GTFS import

1775e59 · Gabriel Morell · 2026-08-07 15:31

133 files +14715 -192
Message
{commit_body(@commit)}

Files changed

modified .gitignore
+6 −0
@@ -21,3 +21,9 @@ erl_crash.dump
21 21
22 22 # Temporary files, for example, from tests.
23 23 /tmp/
24 +
25 +# Mnesia store created at runtime by wheretz
26 +/Mnesia.*/
27 +
28 +# Server logs
29 +*.log
modified README.md
+57 −1
@@ -33,8 +33,64 @@ The public demo instance https://ac.gmp.io/ is here, and the landing data displa
33 33
34 34 ### Development
35 35
36 −Debian/Ubuntu users may need to isntall `erlang-xmerl`
36 +Debian/Ubuntu users may need to install `erlang-xmerl`
37 +
38 +Postgres is expected on **port 54321** (not 5432) as `postgres`/`postgres`, database
39 +`room_sanctum_dev` — see `config/dev.exs`. `docker-compose.yml` brings one up.
40 +
41 +#### First-time setup
42 +
43 +```sh
44 +mix deps.get
45 +mix download_data # REQUIRED — see below, the server will not boot without it
46 +```
47 +
48 +Then run the migrations, then `mix phx.server`. Two endpoints come up:
49 +
50 +- `room_hermes` on http://localhost:4001
51 +- `room_sanctum` on http://localhost:4002
52 +
53 +#### `mix download_data` — required before the first boot
54 +
55 +`wheretz` (timezone lookup by coordinate) ships **without** its geojson database, and
56 +`WhereTZ.Application.start/2` loads that data unconditionally at boot. If it is missing,
57 +the app fails to start and takes the whole umbrella down with it:
58 +
59 +```
60 +** (Mix) Could not start application wheretz: exited in: WhereTZ.Application.start(:normal, [])
61 + ** (EXIT) an exception was raised:
62 + ** (MatchError) no match of right hand side value:
63 + {:error, :enoent}
64 + (wheretz 0.1.16) lib/mix/tasks/download_data.ex:52: Mix.Tasks.DownloadData.load_from_json/0
65 +```
66 +
67 +`mix download_data` fetches a 42MB archive from the `timezone-boundary-builder` GitHub
68 +release and unpacks it to ~174MB. Gotchas:
69 +
70 +- The data lands in **`_build/$MIX_ENV/lib/wheretz/priv/data/`**, not in `deps/`. So it is
71 + per-environment (run it again under `MIX_ENV=prod`) and anything that clears `_build`
72 + — `mix clean`, a fresh clone, a Docker layer rebuild — means running it again.
73 +- **Boot is slow.** wheretz reparses the 139MB geojson into Mnesia on *every* start, so
74 + expect several minutes before the endpoints bind. It is not hung.
75 +- Startup writes a Mnesia directory (`Mnesia.nonode@nohost/`) into the repo root. It is
76 + not currently in `.gitignore`.
37 77
38 78 #### Migrations
39 79
40 80 Migrations need to be run from within the `room_sanctum`/`room_hermes` directories in that order.
81 +
82 +Until they are run the server *will* start and bind both ports, but every request returns
83 +**503** and the log fills with `Postgrex.Error ... (undefined_table)` for `oban_jobs`,
84 +`cfg_sources`, `cfg_visions`, `keryxiae`, and friends. A 503 on a fresh checkout means
85 +migrations, not a broken build.
86 +
87 +Both apps share one database, and therefore one `schema_migrations` table. So
88 +`mix ecto.migrations` run from either directory reports the *other* app's migrations as
89 +`** FILE NOT FOUND **` — they are recorded in the shared table but their files live under
90 +the sibling app. That is expected, not a corrupt migration history.
91 +
92 +#### Once it is up
93 +
94 +- `room_sanctum` (http://localhost:4002) serves the UI and should return 200.
95 +- `room_hermes` (http://localhost:4001) has no root route — its `get "/"` is commented out
96 + in the router — so **404 on `/` is normal** and means the app is healthy.
added apps/room_bourse/lib/room_bourse.ex
+3 −0

Click to load diff…

added apps/room_bourse/lib/room_bourse/worker.ex
+227 −0

Click to load diff…

added apps/room_bourse/mix.exs
+28 −0

Click to load diff…

added apps/room_cronos/lib/room_cronos/countdown.ex
+99 −0

Click to load diff…

modified apps/room_cronos/lib/room_cronos/cronos_worker.ex
+25 −0

Click to load diff…

modified apps/room_gtfs/lib/room_gtfs/gtfs_worker.ex
+26 −2

Click to load diff…

modified apps/room_gtfs/mix.exs
+1 −1

Click to load diff…

modified apps/room_hermes/.gitignore
+5 −0

Click to load diff…

added apps/room_hermes/lib/room_hermes/mail/headers.ex
+258 −0

Click to load diff…

added apps/room_hermes/lib/room_hermes/mail/imap.ex
+424 −0

Click to load diff…

added apps/room_hermes/lib/room_hermes/mail/imap_worker.ex
+237 −0

Click to load diff…

added apps/room_hermes/lib/room_hermes/mail/parser.ex
+109 −0

Click to load diff…

modified apps/room_hermes/lib/room_hermes/mail/server.ex
+6 −98

Click to load diff…

modified apps/room_hermes/lib/room_hermes/mail/supervisor.ex
+34 −3

Click to load diff…

added apps/room_icarus/lib/room_icarus.ex
+3 −0

Click to load diff…

added apps/room_icarus/lib/room_icarus/airlines.ex
+120 −0

Click to load diff…

added apps/room_icarus/lib/room_icarus/airports.ex
+106 −0

Click to load diff…

added apps/room_icarus/lib/room_icarus/classify.ex
+124 −0

Click to load diff…

added apps/room_icarus/lib/room_icarus/flight.ex
+227 −0

Click to load diff…

added apps/room_icarus/lib/room_icarus/worker.ex
+455 −0

Click to load diff…

added apps/room_icarus/mix.exs
+28 −0

Click to load diff…

added apps/room_icarus/priv/airports.tsv
+8801 −0

Click to load diff…

modified apps/room_packages/lib/room_packages/package_worker.ex
+57 −0

Click to load diff…

added apps/room_packages/lib/room_packages/usps.ex
+394 −0

Click to load diff…

modified apps/room_packages/mix.exs
+3 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum/condensers/basic_mqtt.ex
+12 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum/configuration.ex
+20 −9

Click to load diff…

added apps/room_sanctum/lib/room_sanctum/configuration/configs/bourse.ex
+20 −0

Click to load diff…

added apps/room_sanctum/lib/room_sanctum/configuration/configs/icarus.ex
+20 −0

Click to load diff…

added apps/room_sanctum/lib/room_sanctum/configuration/configs/mailbox.ex
+66 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum/configuration/configs/packages.ex
+16 −1

Click to load diff…

added apps/room_sanctum/lib/room_sanctum/configuration/configs/treasury.ex
+47 −0

Click to load diff…

added apps/room_sanctum/lib/room_sanctum/configuration/queries/bourse.ex
+39 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum/configuration/queries/cronos.ex
+29 −3

Click to load diff…

added apps/room_sanctum/lib/room_sanctum/configuration/queries/icarus.ex
+131 −0

Click to load diff…

added apps/room_sanctum/lib/room_sanctum/configuration/queries/mailbox.ex
+28 −0

Click to load diff…

added apps/room_sanctum/lib/room_sanctum/configuration/queries/treasury.ex
+73 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum/configuration/query.ex
+4 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum/configuration/source.ex
+8 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum/configuration/taxid.ex
+16 −1

Click to load diff…

added apps/room_sanctum/lib/room_sanctum/icarus.ex
+74 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum/queues/mail.ex
+84 −15

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum/queues/webhooks.ex
+36 −1

Click to load diff…

added apps/room_sanctum/lib/room_sanctum/storage/icarus/flight_watch.ex
+64 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum/workers/keryx_item_cache.ex
+17 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum/workers/keryx_worker.ex
+24 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum/workers/vision_worker.ex
+24 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/components/core_components.ex
+52 −3

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/controllers/data_controller.ex
+9 −1

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/icon_helpers.ex
+24 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/live/cfg_pythiae_live/public.ex
+1 −1

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/live/cfg_pythiae_live/public.html.heex
+4 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/live/cfg_pythiae_live/show.html.heex
+4 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/live/cfg_query_live/form_component.ex
+55 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/live/cfg_query_live/form_component.html.heex
+135 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/live/cfg_query_live/show.ex
+24 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/live/cfg_query_live/show.html.heex
+4 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/live/cfg_scribus_live/public.ex
+1 −1

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/live/cfg_scribus_live/public.html.heex
+4 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/live/cfg_source_live/form_component.ex
+36 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/live/cfg_source_live/form_component.html.heex
+114 −1

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/live/cfg_source_live/index.html.heex
+1 −1

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/live/cfg_source_live/show.ex
+164 −12

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/live/cfg_source_live/show.html.heex
+90 −1

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/live/cfg_vision_live/show.html.heex
+4 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/live/landing_live/index.html.heex
+4 −0

Click to load diff…

modified apps/room_sanctum/lib/room_sanctum_web/live/live_preview.ex
+396 −0

Click to load diff…

added apps/room_sanctum/lib/room_sanctum_web/source_type_info.ex
+60 −0

Click to load diff…

modified apps/room_sanctum/mix.exs
+5 −2

Click to load diff…

added apps/room_sanctum/priv/repo/migrations/20260806130000_create_icarus_flight_watches.exs
+44 −0

Click to load diff…

added apps/room_sanctum/priv/repo/migrations/20260806150000_add_imap_to_cfg_mailboxes.exs
+23 −0

Click to load diff…

added apps/room_sanctum/priv/repo/migrations/20260806160000_drop_imap_from_cfg_mailboxes.exs
+21 −0

Click to load diff…

added apps/room_treasury/lib/room_treasury.ex
+3 −0

Click to load diff…

added apps/room_treasury/lib/room_treasury/currencies.ex
+73 −0

Click to load diff…

added apps/room_treasury/lib/room_treasury/worker.ex
+245 −0

Click to load diff…

added apps/room_treasury/mix.exs
+28 −0

Click to load diff…

added apps/room_treasury/priv/currencies.tsv
+339 −0

Click to load diff…

modified apps/room_zeus/lib/supervision_worker.ex
+4 −0

Click to load diff…

modified mix.exs
+3 −0

Click to load diff…

modified mix.lock
+26 −34

Click to load diff…

Parents: 6e45ff4