GITGUD

neiam neiam /dms

public · Issues · Pulls · Labels · Forks · Compare · Actions failure · Badges · Packages

⭐

Log in to mark this repository.

9.9 KiB · text History 6908c22

DMS

Stash images. Find them with a phrase.

DMS is a self-hosted document-management system aimed at images and screenshots. Every upload fans out to a queue of categorizers — thumbnails, OCR, image description, tag extraction — and everything they notice gets indexed as namespaced tags so you can search later by anything any pipeline saw. “McMaster catalog cover” finds the photo of the McMaster catalog cover.

Architecture

                 ┌──────────────┐                    ┌──────────────┐
  HTTP / API ──▶ │  webservice  │ ─── enqueue ─────▶ │   RabbitMQ   │
                 │  (axum, JWT- │                    │  per-kind Q  │
                 │   like)      │                    │  + DLQ       │
                 └──────┬───────┘                    └──────┬───────┘
                        │                                   │
                        ▼                                   ▼
                 ┌──────────────┐                    ┌──────────────┐
                 │   Postgres   │ ◀── result rows ── │ queue-worker │
                 │  (sea-orm)   │                    │  thumbnail / │
                 └──────────────┘                    │  ocr / desc /│
                                                     │  extract_tags│
                                                     └──────────────┘

Three Rust binaries, one Postgres, one RabbitMQ, content-addressed storage on disk. Workers are competing consumers — scale per-kind by running more replicas.

Repository layout

Path What
src/bin/webservice.rs HTTP + HTML server (axum + minijinja).
src/bin/queue-worker.rs Categorizer worker. Picks one kind via WORKER_KIND env / --kind.
src/bin/registration-token.rs CLI for generating invite tokens out of band.
src/queue/ RabbitMQ topology (per-kind queues + DLX), publish/consume, typed JobMessage.
src/storage/ Storage trait + content-addressed LocalStorage (sha256 fan-out).
src/workers/ Concrete categorizers — thumbnail (image crate), OCR (ocrs), describe (pluggable), extract_tags.
src/jobs/ Per-upload, per-kind job-state tracking with aggregate roll-up onto the upload row.
src/handlers/ HTTP route handlers (HTML + JSON variants).
src/auth/ Sessions, password hashing (argon2), registration tokens, API tokens.
migration/ sea-orm-migration crate with the schema.
templates/ minijinja templates (root layout, uploads grid/detail, search, invites, api-tokens, …).
assets/ npm + Tailwind + DaisyUI sources; assets/f/ is the compiled output served at /assets.
tests/search_integration.rs End-to-end tests against a real Postgres (gated on DMS_TEST_DATABASE_URL).
android/ Companion Kotlin/Compose app — pair via QR, share-in/out, search, detail.
Dockerfile, compose.yml Multi-stage container build + full-stack compose (podman-friendly).

Run it

The fastest path is the bundled compose stack — postgres, rabbitmq, the webservice, and one worker per categorizer.

just build-image    # podman compose build webservice
just up             # podman compose up -d
just invite-in-pod  # bootstrap a registration token

Visit http://localhost:3007/register?token=<token>, sign up, generate an API token from /api-tokens for the Android app.

OCR models (ocrs detection + recognition .rten files) are bundled into the image at /opt/dms/models/ so the OCR worker is ready to go on first boot. Override OCRS_DETECTION_MODEL / OCRS_RECOGNITION_MODEL via env if you ever want to swap them — bigger model, fine-tuned variant, host-mounted file, etc.

Run it (without containers)

For development you’ll want the docker postgres + rabbitmq up and the Rust binaries running on the host:

podman compose up -d postgres rabbitmq
just deps           # one-time: npm install for tailwind
just assets         # build CSS + sync FontAwesome webfonts
just migrate        # apply pending migrations
just serve          # webservice on :3007 (auto-migrates by default)

# In another shell, one worker per kind:
WORKER_KIND=thumbnail   cargo run --bin queue-worker
WORKER_KIND=ocr         cargo run --bin queue-worker
WORKER_KIND=describe    cargo run --bin queue-worker
WORKER_KIND=extract_tags cargo run --bin queue-worker

just watch keeps Tailwind running in --watch mode while you edit templates.

Configuration

All env-driven — defaults make sense for local dev.

Env Default Notes
DATABASE_URL postgres://dms_user:dms_password@localhost:5433/dms sea-orm connection string.
AMQP_URL amqp://dms_user:dms_password@localhost:5674/%2f RabbitMQ.
BIND_ADDRESS 0.0.0.0:3007 Webservice listener.
STORAGE_ROOT ./storage Content-addressed file root.
MAX_UPLOAD_BYTES 104857600 (100 MiB) Per-upload cap. Bigger than this and the user gets a friendly redirect; the body limit layer matches.
SESSION_DURATION_HOURS 24 Browser session cookie lifetime.
REGISTRATION_TOKEN_DURATION_DAYS 7 Default invite expiry.
AUTO_MIGRATE true Run pending migrations on webservice boot. Disable in environments that prefer a separate migration step.
REGISTRATION_ENABLED false When off, plain /register 404s; /register?token=… invite links still work.
DMS_PUBLIC_URL (unset) Override what the QR pairing URL encodes when behind a proxy. Falls back to the request’s Host header, then the bind address.
WORKER_KIND (required for workers) thumbnail / ocr / describe / extract_tags. Same as --kind.
WORKER_PREFETCH 4 Per-worker concurrency (RabbitMQ basic_qos). Tune CPU-bound (OCR) lower than I/O-bound.
OCRS_DETECTION_MODEL /opt/dms/models/text-detection.rten Bundled in the image; override to swap models. Required for OCR — outside Docker (e.g. cargo run), set this to a downloaded .rten file or the worker DLQs jobs with a clear log line.
OCRS_RECOGNITION_MODEL /opt/dms/models/text-recognition.rten Same as above.

Auth

Two credential types feed the same handlers via one middleware:

  • Browser session: argon2-hashed password + UUID session token in an HttpOnly; SameSite=Strict cookie. auth_middleware looks for it first.
  • API token: long-lived, generated from /api-tokens (also rendered as a dms://pair?… QR for the Android app). Sent as Authorization: Bearer <token>. Same middleware accepts it; downstream handlers can’t tell the difference.

Public registration is off by default. To onboard a user:

  1. As an existing user, visit /invites → generate a link with optional email pin and TTL.
  2. Or out-of-band: just invite (or just invite-in-pod for the containerized variant) to mint a token via the CLI.

Two surfaces, same engine.

  • /uploads?q=…&tags=…&namespace=… — listing-style filter.
  • /search — dedicated page with per-source toggles. Bare phrase, ns:value, or ns:* to list every tag in a namespace. Tag-based sources (user-tags, ocr:*, desc:*, other namespaces) can be excluded individually.
  • GET /api/search mirrors the dedicated page in JSON for the Android client; results carry matched_tags so the UI can show which tag triggered the hit.

The query parser splits ns:value into namespace + value-pattern; species:macro matches the species-namespaced tag with value matching %macro%. species:* is a namespace wildcard. Bare values match across whichever sources are enabled.

The whole pipeline is covered by unit tests for the parser plus integration tests that exercise the SQL against a real Postgres (tests/search_integration.rs).

Pipelines

Every upload row keeps a processing_status. Per-upload, per-kind state lives in upload_jobs(upload_id, kind, status, error, attempts, started_at, completed_at, …). Workers walk each row through pending → processing → complete (or failed); after every terminal transition a recompute flips uploads.processing_status to processing / complete / partial based on the aggregate.

Workers consult upload_jobs.is_complete for idempotency, so re-deliveries are no-ops. Manual re-runs from the detail page can pass force=on to clear the result and start fresh.

Android app

Self-hosted companion in android/. First slice ships:

  • QR pairing → (base_url, api_token) in EncryptedSharedPreferences.
  • Home grid with thumbnails (Coil, with original-file fallback when the thumbnail pipeline hasn’t run yet), pull-to-refresh, infinite scroll.
  • Search screen with parity to the web /search source toggles, hitting the JSON endpoint.
  • Detail screen — image, metadata, tag chips that pivot to a tag-filtered search, “Add tags” input wired to POST /api/uploads/{id}/tags.
  • Share-in: receive one or many images from the system share sheet, add tags, upload sequentially with a live progress indicator.
  • Share-out: from the detail screen, download to cache and hand to another app via FileProvider.
  • B612 mono typography + the same OKLCh palettes as the rest of the Casabeza Android apps, persisted via a separate prefs file.

See android/README.md for build/run details.

Testing

just check               # cargo check + unit tests (no DB needed)
just test-integration    # integration tests against the local postgres

The integration suite spins up a unique Postgres schema per test, runs migrations into it, exercises the search code, then drops the schema. Skipped silently if DMS_TEST_DATABASE_URL isn’t set — handy in CI without a database, opt-in locally.

License

See LICENSE.md.