neiam /apollos_crib

It's Apollo's Crib

public · Issues · Pulls 1 · Labels · Forks · Compare · Actions failure · Badges · Packages

Put the query timeouts back now that the queries are fast

c6f2de8 · Gabriel Morell · 2026-09-02 20:46

7 files +21 -17

Message

{commit_body(@commit)}

Files changed

modified apps/room_sanctum/lib/room_sanctum/storage.ex

+10 −6

@@ -1617,12 +1617,16 @@ defmodule RoomSanctum.Storage do

1617	1617	tz: ^tz
1618	1618	}
1619	1619	)
1620		− # Ecto's default of 15s was cancelling arrival lookups that the caller was
1621		− # perfectly willing to wait for -- the vision allows a query 25s -- and a
1622		− # cancelled statement raises, which used to publish "nothing is coming" on
1623		− # the strength of a timeout. The layers now agree; the lookups being this
1624		− # slow at all is a separate problem.
1625		− \|> Repo.all(timeout: 20_000)
	1620	+ # The layers have to agree about how long this may take, or a statement the
	1621	+ # caller is still waiting for gets cancelled underneath it -- and a
	1622	+ # cancelled statement raises, which once published "nothing is coming" on
	1623	+ # the strength of a timeout.
	1624	+ #
	1625	+ # This was 20s while the lookup took ten to twenty, which it did only
	1626	+ # because prod was missing the index for it. Rebuilt, the same query takes
	1627	+ # two milliseconds. Ecto's default is fine again, and the vision's own cap
	1628	+ # sits above it.
	1629	+ \|> Repo.all()
1626	1630
1627	1631	# q =
1628	1632	# from st in StopTime,

modified apps/room_sanctum/lib/room_sanctum/workers/vision_worker.ex

+11 −11

@@ -11,18 +11,18 @@ defmodule RoomSanctum.Worker.Vision do

11	11	# Each query is asked on its own, so this is the cap on one of them rather
12	12	# than on a round of them.
13	13	#
14		− # Ten seconds was too tight and made things worse rather than safer. GTFS
15		− # arrival lookups against prod's data take on the order of ten seconds -- the
16		− # same slowness that used to blow the thirty second tick when a round asked
17		− # three of them one after another -- so a ten second cap turned "sometimes
18		− # completes" into "never completes", and every panel sat on its previous
19		− # answer for ever.
	14	+ # This was briefly 25 seconds, which was right while an arrival lookup took
	15	+ # ten to twenty. It did, because prod had been running for four days without
	16	+ # the index those lookups need -- a CREATE INDEX CONCURRENTLY interrupted
	17	+ # partway leaves an index Postgres will not use, and the migration that
	18	+ # created it was recorded as applied, so nothing ever retried. With the index
	19	+ # rebuilt the same lookup takes two milliseconds.
20	20	#
21		− # Just under the tick instead: a query gets essentially the whole interval,
22		− # and the cap goes back to being a guard against a source that has genuinely
23		− # hung rather than a limit real work runs into. It is not a fix for the
24		− # slowness, which wants finding with an EXPLAIN against prod.
25		− @query_timeout_ms 25_000
	21	+ # So back to ten. A cap wants to be far enough above real work not to cut it
	22	+ # off, and near enough that a source which has genuinely hung is noticed
	23	+ # rather than tying up a slot for half a minute. Ten is five thousand times
	24	+ # the work it is guarding.
	25	+ @query_timeout_ms 10_000
26	26
27	27	# How many of a vision's queries may be out at once.
28	28	#

Parents: 36c4afc