neiam /dms

public · Issues · Pulls · Labels · Forks · Compare · Actions failure · Badges · Packages

missing

3632264 · gmorell · 2026-05-08 02:44

2 files +212 -0

Files changed

added .gitlab-ci.yml

+11 −0

@@ -0,0 +1,11 @@

	1	+build-master:
	2	+ retry: 1
	3	+ stage: build
	4	+ image:
	5	+ name: gcr.io/kaniko-project/executor:v1.23.2-debug
	6	+ entrypoint: [""]
	7	+ only:
	8	+ - master
	9	+ script:
	10	+ - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
	11	+ - /kaniko/executor --force --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE --registry-mirror us0.regi.kube.gmp.io

modified README.md

+201 −0

@@ -1,2 +1,203 @@

1	1	# DMS
2	2
	3	+Stash images. Find them with a phrase.
	4	+
	5	+DMS is a self-hosted document-management system aimed at images and screenshots.
	6	+Every upload fans out to a queue of categorizers — thumbnails, OCR, image
	7	+description, tag extraction — and everything they notice gets indexed as
	8	+namespaced tags so you can search later by anything any pipeline saw.
	9	+"McMaster catalog cover" finds the photo of the McMaster catalog cover.
	10	+
	11	+## Architecture
	12	+
	13	+```
	14	+ ┌──────────────┐ ┌──────────────┐
	15	+ HTTP / API ──▶ │ webservice │ ─── enqueue ─────▶ │ RabbitMQ │
	16	+ │ (axum, JWT- │ │ per-kind Q │
	17	+ │ like) │ │ + DLQ │
	18	+ └──────┬───────┘ └──────┬───────┘
	19	+ │ │
	20	+ ▼ ▼
	21	+ ┌──────────────┐ ┌──────────────┐
	22	+ │ Postgres │ ◀── result rows ── │ queue-worker │
	23	+ │ (sea-orm) │ │ thumbnail / │
	24	+ └──────────────┘ │ ocr / desc /│
	25	+ │ extract_tags│
	26	+ └──────────────┘
	27	+```
	28	+
	29	+Three Rust binaries, one Postgres, one RabbitMQ, content-addressed storage on
	30	+disk. Workers are competing consumers — scale per-kind by running more
	31	+replicas.
	32	+
	33	+## Repository layout
	34	+
	35	+\| Path \| What \|
	36	+\|---\|---\|
	37	+\| `src/bin/webservice.rs` \| HTTP + HTML server (axum + minijinja). \|
	38	+\| `src/bin/queue-worker.rs` \| Categorizer worker. Picks one kind via `WORKER_KIND` env / `--kind`. \|
	39	+\| `src/bin/registration-token.rs` \| CLI for generating invite tokens out of band. \|
	40	+\| `src/queue/` \| RabbitMQ topology (per-kind queues + DLX), publish/consume, typed `JobMessage`. \|
	41	+\| `src/storage/` \| `Storage` trait + content-addressed `LocalStorage` (sha256 fan-out). \|
	42	+\| `src/workers/` \| Concrete categorizers — thumbnail (image crate), OCR (ocrs), describe (pluggable), extract_tags. \|
	43	+\| `src/jobs/` \| Per-upload, per-kind job-state tracking with aggregate roll-up onto the upload row. \|
	44	+\| `src/handlers/` \| HTTP route handlers (HTML + JSON variants). \|
	45	+\| `src/auth/` \| Sessions, password hashing (argon2), registration tokens, API tokens. \|
	46	+\| `migration/` \| sea-orm-migration crate with the schema. \|
	47	+\| `templates/` \| minijinja templates (root layout, uploads grid/detail, search, invites, api-tokens, …). \|
	48	+\| `assets/` \| npm + Tailwind + DaisyUI sources; `assets/f/` is the compiled output served at `/assets`. \|
	49	+\| `tests/search_integration.rs` \| End-to-end tests against a real Postgres (gated on `DMS_TEST_DATABASE_URL`). \|
	50	+\| `android/` \| Companion Kotlin/Compose app — pair via QR, share-in/out, search, detail. \|
	51	+\| `Dockerfile`, `compose.yml` \| Multi-stage container build + full-stack compose (podman-friendly). \|
	52	+
	53	+## Run it
	54	+
	55	+The fastest path is the bundled compose stack — postgres, rabbitmq, the
	56	+webservice, and one worker per categorizer.
	57	+
	58	+```bash
	59	+just build-image # podman compose build webservice
	60	+just up # podman compose up -d
	61	+just invite-in-pod # bootstrap a registration token
	62	+```
	63	+
	64	+Visit `http://localhost:3007/register?token=<token>`, sign up, generate an
	65	+API token from `/api-tokens` for the Android app.
	66	+
	67	+For OCR to actually run, drop the `ocrs` models into the `dms_data`
	68	+volume's `models/` directory. With a host-bind variant of compose:
	69	+
	70	+```bash
	71	+just ocr-models # downloads text-detection.rten + text-recognition.rten
	72	+```
	73	+
	74	+…then bind-mount `./models` to `/data/models` (with `:Z` if you're on
	75	+SELinux).
	76	+
	77	+## Run it (without containers)
	78	+
	79	+For development you'll want the docker postgres + rabbitmq up and the Rust
	80	+binaries running on the host:
	81	+
	82	+```bash
	83	+podman compose up -d postgres rabbitmq
	84	+just deps # one-time: npm install for tailwind
	85	+just assets # build CSS + sync FontAwesome webfonts
	86	+just migrate # apply pending migrations
	87	+just serve # webservice on :3007 (auto-migrates by default)
	88	+
	89	+# In another shell, one worker per kind:
	90	+WORKER_KIND=thumbnail cargo run --bin queue-worker
	91	+WORKER_KIND=ocr cargo run --bin queue-worker
	92	+WORKER_KIND=describe cargo run --bin queue-worker
	93	+WORKER_KIND=extract_tags cargo run --bin queue-worker
	94	+```
	95	+
	96	+`just watch` keeps Tailwind running in `--watch` mode while you edit
	97	+templates.
	98	+
	99	+## Configuration
	100	+
	101	+All env-driven — defaults make sense for local dev.
	102	+
	103	+\| Env \| Default \| Notes \|
	104	+\|---\|---\|---\|
	105	+\| `DATABASE_URL` \| `postgres://dms_user:dms_password@localhost:5433/dms` \| sea-orm connection string. \|
	106	+\| `AMQP_URL` \| `amqp://dms_user:dms_password@localhost:5674/%2f` \| RabbitMQ. \|
	107	+\| `BIND_ADDRESS` \| `0.0.0.0:3007` \| Webservice listener. \|
	108	+\| `STORAGE_ROOT` \| `./storage` \| Content-addressed file root. \|
	109	+\| `MAX_UPLOAD_BYTES` \| `104857600` (100 MiB) \| Per-upload cap. Bigger than this and the user gets a friendly redirect; the body limit layer matches. \|
	110	+\| `SESSION_DURATION_HOURS` \| `24` \| Browser session cookie lifetime. \|
	111	+\| `REGISTRATION_TOKEN_DURATION_DAYS` \| `7` \| Default invite expiry. \|
	112	+\| `AUTO_MIGRATE` \| `true` \| Run pending migrations on webservice boot. Disable in environments that prefer a separate migration step. \|
	113	+\| `REGISTRATION_ENABLED` \| `false` \| When off, plain `/register` 404s; `/register?token=…` invite links still work. \|
	114	+\| `DMS_PUBLIC_URL` \| (unset) \| Override what the QR pairing URL encodes when behind a proxy. Falls back to the request's Host header, then the bind address. \|
	115	+\| `WORKER_KIND` \| (required for workers) \| `thumbnail` / `ocr` / `describe` / `extract_tags`. Same as `--kind`. \|
	116	+\| `WORKER_PREFETCH` \| `4` \| Per-worker concurrency (RabbitMQ basic_qos). Tune CPU-bound (OCR) lower than I/O-bound. \|
	117	+\| `OCRS_DETECTION_MODEL` \| (required for OCR) \| Path to `text-detection.rten`. \|
	118	+\| `OCRS_RECOGNITION_MODEL` \| (required for OCR) \| Path to `text-recognition.rten`. \|
	119	+
	120	+## Auth
	121	+
	122	+Two credential types feed the same handlers via one middleware:
	123	+
	124	+- Browser session: `argon2`-hashed password + UUID session token in an
	125	+ `HttpOnly; SameSite=Strict` cookie. `auth_middleware` looks for it first.
	126	+- API token: long-lived, generated from `/api-tokens` (also rendered as
	127	+ a `dms://pair?…` QR for the Android app). Sent as `Authorization: Bearer
	128	+ <token>`. Same middleware accepts it; downstream handlers can't tell the
	129	+ difference.
	130	+
	131	+Public registration is off by default. To onboard a user:
	132	+
	133	+1. As an existing user, visit `/invites` → generate a link with optional
	134	+ email pin and TTL.
	135	+2. Or out-of-band: `just invite` (or `just invite-in-pod` for the
	136	+ containerized variant) to mint a token via the CLI.
	137	+
	138	+## Search
	139	+
	140	+Two surfaces, same engine.
	141	+
	142	+- `/uploads?q=…&tags=…&namespace=…` — listing-style filter.
	143	+- `/search` — dedicated page with per-source toggles. Bare phrase, `ns:value`,
	144	+ or `ns:*` to list every tag in a namespace. Tag-based sources (user-tags,
	145	+ `ocr:`, `desc:`, other namespaces) can be excluded individually.
	146	+- `GET /api/search` mirrors the dedicated page in JSON for the Android client;
	147	+ results carry `matched_tags` so the UI can show which tag triggered the hit.
	148	+
	149	+The query parser splits `ns:value` into namespace + value-pattern; `species:macro`
	150	+matches the `species`-namespaced tag with value matching `%macro%`. `species:*` is
	151	+a namespace wildcard. Bare values match across whichever sources are enabled.
	152	+
	153	+The whole pipeline is covered by unit tests for the parser plus integration tests
	154	+that exercise the SQL against a real Postgres (`tests/search_integration.rs`).
	155	+
	156	+## Pipelines
	157	+
	158	+Every upload row keeps a `processing_status`. Per-upload, per-kind state
	159	+lives in `upload_jobs(upload_id, kind, status, error, attempts, started_at,
	160	+completed_at, …)`. Workers walk each row through `pending → processing →
	161	+complete` (or `failed`); after every terminal transition a recompute flips
	162	+`uploads.processing_status` to `processing` / `complete` / `partial` based on
	163	+the aggregate.
	164	+
	165	+Workers consult `upload_jobs.is_complete` for idempotency, so re-deliveries
	166	+are no-ops. Manual re-runs from the detail page can pass `force=on` to clear
	167	+the result and start fresh.
	168	+
	169	+## Android app
	170	+
	171	+Self-hosted companion in `android/`. First slice ships:
	172	+
	173	+- QR pairing → `(base_url, api_token)` in `EncryptedSharedPreferences`.
	174	+- Home grid with thumbnails (Coil, with original-file fallback when the
	175	+ thumbnail pipeline hasn't run yet), pull-to-refresh, infinite scroll.
	176	+- Search screen with parity to the web `/search` source toggles, hitting the
	177	+ JSON endpoint.
	178	+- Detail screen — image, metadata, tag chips that pivot to a tag-filtered
	179	+ search, "Add tags" input wired to `POST /api/uploads/{id}/tags`.
	180	+- Share-in: receive one or many images from the system share sheet, add tags,
	181	+ upload sequentially with a live progress indicator.
	182	+- Share-out: from the detail screen, download to cache and hand to another
	183	+ app via FileProvider.
	184	+- B612 mono typography + the same OKLCh palettes as the rest of the
	185	+ Casabeza Android apps, persisted via a separate prefs file.
	186	+
	187	+See `android/README.md` for build/run details.
	188	+
	189	+## Testing
	190	+
	191	+```bash
	192	+just check # cargo check + unit tests (no DB needed)
	193	+just test-integration # integration tests against the local postgres
	194	+```
	195	+
	196	+The integration suite spins up a unique Postgres schema per test, runs
	197	+migrations into it, exercises the search code, then drops the schema.
	198	+Skipped silently if `DMS_TEST_DATABASE_URL` isn't set — handy in CI without
	199	+a database, opt-in locally.
	200	+
	201	+## License
	202	+
	203	+See `LICENSE.md`.

Parents: 1ce9641