GITGUD

neiam neiam /dms

public · Issues · Pulls · Labels · Forks · Compare · Actions failure · Badges · Packages

⭐

Log in to mark this repository.

14.2 KiB · text History 6fa3ef9

DMS

Stash images. Find them with a phrase.

DMS is a self-hosted document-management system aimed at images and screenshots. Every upload fans out to a queue of categorizers — thumbnails, OCR, image description, tag extraction — and everything they notice gets indexed as namespaced tags so you can search later by anything any pipeline saw. “McMaster catalog cover” finds the photo of the McMaster catalog cover.

Architecture

                 ┌──────────────┐                    ┌──────────────┐
  HTTP / API ──▶ │  webservice  │ ─── enqueue ─────▶ │   RabbitMQ   │
                 │  (axum, JWT- │                    │  per-kind Q  │
                 │   like)      │                    │  + DLQ       │
                 └──────┬───────┘                    └──────┬───────┘
                        │                                   │
                        ▼                                   ▼
                 ┌──────────────┐                    ┌──────────────┐
                 │   Postgres   │ ◀── result rows ── │ queue-worker │
                 │  (sea-orm)   │                    │  thumbnail / │
                 └──────────────┘                    │  ocr / desc /│
                                                     │  extract_tags│
                                                     └──────────────┘

Three Rust binaries, one Postgres, one RabbitMQ, content-addressed storage on disk. Workers are competing consumers — scale per-kind by running more replicas.

Repository layout

Path What
src/bin/webservice.rs HTTP + HTML server (axum + minijinja).
src/bin/queue-worker.rs Categorizer worker. Picks one kind via WORKER_KIND env / --kind.
src/bin/registration-token.rs CLI for generating invite tokens out of band.
src/bin/user-admin.rs CLI for managing users — password resets, activate/deactivate.
src/queue/ RabbitMQ topology (per-kind queues + DLX), publish/consume, typed JobMessage.
src/storage/ Storage trait + content-addressed LocalStorage (sha256 fan-out).
src/workers/ Concrete categorizers — thumbnail (image crate), OCR (ocrs), describe (pluggable), extract_tags.
src/jobs/ Per-upload, per-kind job-state tracking with aggregate roll-up onto the upload row.
src/handlers/ HTTP route handlers (HTML + JSON variants).
src/auth/ Sessions, password hashing (argon2), registration tokens, API tokens.
migration/ sea-orm-migration crate with the schema.
templates/ minijinja templates (root layout, uploads grid/detail, search, invites, api-tokens, …).
assets/ npm + Tailwind + DaisyUI sources; assets/f/ is the compiled output served at /assets.
tests/search_integration.rs End-to-end tests against a real Postgres (gated on DMS_TEST_DATABASE_URL).
android/ Companion Kotlin/Compose app — pair via QR, share-in/out, search, detail.
Dockerfile, compose.yml Multi-stage container build + full-stack compose (podman-friendly).

Run it

The fastest path is the bundled compose stack — postgres, rabbitmq, the webservice, and one worker per categorizer.

just build-image    # podman compose build webservice
just up             # podman compose up -d
just invite-in-pod  # bootstrap a registration token

Visit http://localhost:3007/register?token=<token>, sign up, generate an API token from /api-tokens for the Android app.

OCR models (ocrs detection + recognition .rten files) are bundled into the image at /opt/dms/models/ so the OCR worker is ready to go on first boot. Override OCRS_DETECTION_MODEL / OCRS_RECOGNITION_MODEL via env if you ever want to swap them — bigger model, fine-tuned variant, host-mounted file, etc.

Run it (without containers)

For development you’ll want the docker postgres + rabbitmq up and the Rust binaries running on the host:

podman compose up -d postgres rabbitmq
just deps           # one-time: npm install for tailwind
just assets         # build CSS + sync FontAwesome webfonts
just migrate        # apply pending migrations
just serve          # webservice on :3007 (auto-migrates by default)

# In another shell, one worker per kind:
WORKER_KIND=thumbnail   cargo run --bin queue-worker
WORKER_KIND=ocr         cargo run --bin queue-worker
WORKER_KIND=describe    cargo run --bin queue-worker
WORKER_KIND=extract_tags cargo run --bin queue-worker

just watch keeps Tailwind running in --watch mode while you edit templates.

Configuration

All env-driven — defaults make sense for local dev.

Env Default Notes
DATABASE_URL postgres://dms_user:dms_password@localhost:5433/dms sea-orm connection string.
AMQP_URL amqp://dms_user:dms_password@localhost:5674/%2f RabbitMQ.
BIND_ADDRESS 0.0.0.0:3007 Webservice listener.
STORAGE_ROOT ./storage Content-addressed file root.
MAX_UPLOAD_BYTES 104857600 (100 MiB) Per-upload cap; compose.yml raises it to 5 GiB. Bigger than this and the user gets a friendly redirect; the body limit layer matches. Keep the reverse proxy’s own body limit at least as large, or it rejects first.
MAX_ZIP_ENTRIES 1000 Most images read out of one zip. Past it the import stops and says so.
MAX_ZIP_TOTAL_BYTES 1073741824 (1 GiB) Total decompressed bytes read from one zip. A zip is bounded on the wire by MAX_UPLOAD_BYTES, but its contents are not — this is the zip-bomb guard.
SESSION_DURATION_HOURS 24 Browser session cookie lifetime.
REGISTRATION_TOKEN_DURATION_DAYS 7 Default invite expiry.
AUTO_MIGRATE true Run pending migrations on webservice boot. Disable in environments that prefer a separate migration step.
REGISTRATION_ENABLED false When off, plain /register 404s; /register?token=… invite links still work.
DMS_PUBLIC_URL (unset) Override what the QR pairing URL encodes when behind a proxy. Falls back to the request’s Host header, then the bind address.
WORKER_KIND (required for workers) thumbnail / ocr / describe / extract_tags. Same as --kind.
WORKER_PREFETCH 4 Per-worker concurrency (RabbitMQ basic_qos). Tune CPU-bound (OCR) lower than I/O-bound.
WORKER_MAX_ATTEMPTS 100 Hard cap on per-(upload, kind) DB-tracked attempts. Once exceeded, the worker DLQs the next redelivery instead of running it — protects the queue from a single poisoned message hogging consumers. Set 0 to disable.
OCRS_DETECTION_MODEL /opt/dms/models/text-detection.rten Bundled in the image; override to swap models. Required for OCR — outside Docker (e.g. cargo run), set this to a downloaded .rten file or the worker DLQs jobs with a clear log line.
OCRS_RECOGNITION_MODEL /opt/dms/models/text-recognition.rten Same as above.

Auth

Two credential types feed the same handlers via one middleware:

  • Browser session: argon2-hashed password + UUID session token in an HttpOnly; SameSite=Strict cookie. auth_middleware looks for it first.
  • API token: long-lived, generated from /api-tokens (also rendered as a dms://pair?… QR for the Android app). Sent as Authorization: Bearer <token>. Same middleware accepts it; downstream handlers can’t tell the difference.

Public registration is off by default. To onboard a user:

  1. As an existing user, visit /invites → generate a link with optional email pin and TTL.
  2. Or out-of-band: just invite (or just invite-in-pod for the containerized variant) to mint a token via the CLI.

Managing existing users

There is no self-service password reset, so recovery is a CLI operation. user-admin takes a username, email, or UUID interchangeably:

just user list                                  # who exists
just user "show alice"
just user "reset-password alice --generate"     # random password, printed once
just user "reset-password alice"                # prompt for it (hidden)
just user "deactivate alice"                    # revokes sessions too
just user "activate alice"

Resetting a password revokes every active session for that account, so a compromised login is cut off immediately. Prefer --generate or the prompt over --password: an argument is visible in shell history and ps. The password can also be piped in for scripted use — echo "$PW" | user-admin reset-password alice.

Inside the running container the binary is dms-user-admin; just user-in-pod "reset-password alice --generate" wraps podman compose exec.

Ingest

Three ways in, all converging on the same tail — content-address the bytes, dedupe per user against (file_hash, user_id), insert the row, attach any user tags, enqueue the default job set.

Path Web API
One file POST /uploads (multipart file) POST /api/uploads
A link POST /uploads/url (form url) POST /api/uploads/url
A zip of images POST /uploads/zip (multipart file) POST /api/uploads/zip

Pasting anywhere on the site works too: an image on the clipboard attaches to the quick-upload form (or uploads directly), and a pasted link is fetched server-side so animated GIFs and videos keep their original bytes.

Zip import spools the request body to STORAGE_ROOT/tmp as it arrives, then walks the archive from disk on a blocking thread, streaming each image through a small channel. Peak memory is a handful of images regardless of archive size — a 673 MiB zip of 120 photos imports at ~65 MiB RSS. The spool needs as much free disk as the upload and is removed however the request ends. It lives under STORAGE_ROOT rather than /tmp deliberately: /tmp is tmpfs on most hosts, which would put those gigabytes back in RAM.

Per entry:

  • Only entries whose name resolves to an image type are stored. Everything else is counted and skipped.
  • Directories, __MACOSX/ resource forks, and dotfiles are ignored. Nested paths flatten to the basename — storage is content-addressed, so the archive’s layout doesn’t survive.
  • MAX_UPLOAD_BYTES caps each entry, enforced against the bytes actually read rather than the size the central directory claims.
  • MAX_ZIP_ENTRIES and MAX_ZIP_TOTAL_BYTES bound the whole archive.

Every image inherits the description and tags submitted with the zip, and one AMQP connection dispatches jobs for the entire batch. A failing entry is counted, not fatal — the response reports imported / duplicate / skipped / failed counts, and says explicitly when a limit truncated the read.

Two caveats worth knowing:

  • The single-file paths (POST /uploads, POST /uploads/url) still buffer the whole body in memory, so a request there costs its own size in RSS. Only the zip path spools. Raising MAX_UPLOAD_BYTES raises that exposure.
  • An upload over the limit is rejected as soon as the server knows, which leaves the rest of the body in flight; a streaming reverse proxy reports that to the browser as a 502 rather than showing the redirect. The upload forms therefore check the file size client-side and refuse before sending a byte, and the proxy should carry its own client_max_body_size so it can answer with a proper 413.

Two surfaces, same engine.

  • /uploads?q=…&tags=…&namespace=… — listing-style filter.
  • /search — dedicated page with per-source toggles. Bare phrase, ns:value, or ns:* to list every tag in a namespace. Tag-based sources (user-tags, ocr:*, desc:*, other namespaces) can be excluded individually.
  • GET /api/search mirrors the dedicated page in JSON for the Android client; results carry matched_tags so the UI can show which tag triggered the hit.

The query parser splits ns:value into namespace + value-pattern; species:macro matches the species-namespaced tag with value matching %macro%. species:* is a namespace wildcard. Bare values match across whichever sources are enabled.

The whole pipeline is covered by unit tests for the parser plus integration tests that exercise the SQL against a real Postgres (tests/search_integration.rs).

Pipelines

Every upload row keeps a processing_status. Per-upload, per-kind state lives in upload_jobs(upload_id, kind, status, error, attempts, started_at, completed_at, …). Workers walk each row through pending → processing → complete (or failed); after every terminal transition a recompute flips uploads.processing_status to processing / complete / partial based on the aggregate.

Workers consult upload_jobs.is_complete for idempotency, so re-deliveries are no-ops. Manual re-runs from the detail page can pass force=on to clear the result and start fresh.

Android app

Self-hosted companion in android/. First slice ships:

  • QR pairing → (base_url, api_token) in EncryptedSharedPreferences.
  • Home grid with thumbnails (Coil, with original-file fallback when the thumbnail pipeline hasn’t run yet), pull-to-refresh, infinite scroll.
  • Search screen with parity to the web /search source toggles, hitting the JSON endpoint.
  • Detail screen — image, metadata, tag chips that pivot to a tag-filtered search, “Add tags” input wired to POST /api/uploads/{id}/tags.
  • Share-in: receive one or many images from the system share sheet, add tags, upload sequentially with a live progress indicator.
  • Share-out: from the detail screen, download to cache and hand to another app via FileProvider.
  • B612 mono typography + the same OKLCh palettes as the rest of the Casabeza Android apps, persisted via a separate prefs file.

See android/README.md for build/run details.

Testing

just check               # cargo check + unit tests (no DB needed)
just test-integration    # integration tests against the local postgres

The integration suite spins up a unique Postgres schema per test, runs migrations into it, exercises the search code, then drops the schema. Skipped silently if DMS_TEST_DATABASE_URL isn’t set — handy in CI without a database, opt-in locally.

License

See LICENSE.md.