neiam /apollos_crib

It's Apollo's Crib

Rescue orphaned import jobs instead of leaving them executing for ever

b85c005 · Gabriel Morell · 2026-09-01 20:49

2 files +19 -0

Message

{commit_body(@commit)}

modified config/config.exs

+19 −0

@@ -96,6 +96,25 @@ config :room_sanctum, Oban,

96	96	# buries Postgres. See RoomGtfs.ImportJob. Raising this trades database
97	97	# load for getting through the feeds sooner.
98	98	queues: [default: 10, webhooks: 20, emails: 20, gtfs_import: 1],
	99	+ # Nothing rescued a job whose process died. A VM crash mid-import left
	100	+ # the row in `executing` for good, and because ImportJob's uniqueness
	101	+ # counts `executing` with no expiry, that one row blocked every future
	102	+ # import of the feed -- the button on /cfg/offerings/work answered
	103	+ # "already outstanding" for ever, and psql was the only way out.
	104	+ #
	105	+ # rescue_after is the part to get right. Lifeline decides a job is
	106	+ # orphaned from how long ago it was attempted, not from whether it is
	107	+ # actually running, so a value shorter than a real import starts a
	108	+ # second one while the first is midway through truncating the same
	109	+ # tables. Measured over fifteen completed imports the worst case was
	110	+ # under two minutes and the average was thirty-three seconds; thirty
	111	+ # minutes is fifteen times the worst of those, which leaves room for a
	112	+ # feed much larger than any loaded so far and for a database having a
	113	+ # worse day than that one was.
	114	+ #
	115	+ # Erring long costs little now that a stuck job can be cleared by hand
	116	+ # on the queue page rather than waited out.
	117	+ plugins: [{Oban.Plugins.Lifeline, rescue_after: :timer.minutes(30)}],
99	118	repo: RoomSanctum.Repo
100	119	config :messenger, smtp_opts: [[port: 2525]]
101	120

Parents: 5f05b9a