rknn-describe-server

A small HTTP server that runs on the Rockchip board (RK3588 / RK3576 — e.g. r5bp-0) and exposes the airockchip/rknn-llm multimodal demo over a simple multipart endpoint that the DMS describe worker can call.

The DMS Rust worker stays on x86; this server is the only thing that needs to live on the NPU host.

Wire protocol

POST /describe — multipart/form-data:

field	required	notes
image	yes	binary image (jpg/png/webp/…)
prompt	no	overrides default prompt
model	no	name registered in models.yaml (default: first entry)
max_new_tokens	no	integer, default 256

Response:

{ "description": "A red barn at dusk with a tractor in front.",
  "model": "qwen2-vl-2b",
  "tokens": 27 }

GET /healthz — 200 ok once the model is loaded.

Approach

The reference implementation here just shells out to the upstream multimodal_demo binary per request. That keeps the integration minimal but reloads the model every call — fine for a smoke test, far too slow for production.

For sustained throughput, replace subprocess with a long-lived ctypes wrapper around librkllmrt.so (see rkllm.h in the upstream repo) that keeps the language model warm across requests. The request/response shape above does not need to change.

Setup on an RK3588 board

librknnrt.so and librkllmrt.so both ship inside the upstream repo (3rdparty/librknnrt/, rkllm-runtime/) and get installed next to the demo binary, so no system-wide RKNN runtime is required.

The bring-up below is what we ran on r5bp-0 and should reproduce on any RK3588 board with the same OS image.

sudo apt install -y git cmake g++ python3-venv python3-pip

# 1. clone the upstream toolkit and build the multimodal demo (the cmake
#    build-linux.sh assumes a cross-compiler — on-device just call cmake
#    directly with the system toolchain).
mkdir -p ~/dms-vlm && cd ~/dms-vlm
git clone --depth 1 https://github.com/airockchip/rknn-llm.git
cd rknn-llm
# rkllm's default CPU mask assumes 8 online cores. Required on boards with
# fewer (the CM5 at 172.16.15.136 has 6: 4x A55 + 2x A76) -- without it
# rkllm_init dies with "Mismatch between enabled CPUs mask and expected count".
patch -p1 < /path/to/dms/tools/rknn-describe-server/patches/0001-demo-cpu-mask-env.patch
cd examples/multimodal_model_demo/deploy
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release \
         -DCMAKE_SYSTEM_NAME=Linux \
         -DCMAKE_SYSTEM_PROCESSOR=aarch64
make -j8 install
# ⇒ ./install/demo_Linux_aarch64/{demo,imgenc,lib/}

# 2. fetch a converted VLM. The Qwen2-VL-2B build below totals ~3.3 GiB.
mkdir -p ~/dms-vlm/models/qwen2-vl-2b && cd ~/dms-vlm/models/qwen2-vl-2b
wget -O Qwen2-VL-2B-Instruct.rkllm \
  https://huggingface.co/thanhtantran/Qwen2-VL-2B-RKLLM/resolve/main/Qwen2-VL-2B-Instruct.rkllm
wget -O Qwen2-VL-2B-Instruct.rknn \
  https://huggingface.co/thanhtantran/Qwen2-VL-2B-RKLLM/resolve/main/Qwen2-VL-2B-Instruct.rknn

# 3. drop server.py + models.yaml into ~/dms-vlm, create a venv, install deps
cd ~/dms-vlm
python3 -m venv venv
./venv/bin/pip install fastapi "uvicorn[standard]" python-multipart pyyaml pillow
# optional: HEIC/AVIF uploads (server.py registers it if importable)
./venv/bin/pip install pillow-heif

# 4. install the user-level systemd unit and enable linger so it keeps
#    running across reboots without an active login session.
mkdir -p ~/.config/systemd/user
install -m 644 dms-vlm.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now dms-vlm.service
sudo loginctl enable-linger "$USER"

Smoke test from another host:

curl -s http://172.16.15.136:8080/healthz
curl -F image=@/path/to/sample.jpg http://172.16.15.136:8080/describe

First request takes ~10–15 s because the demo binary reloads the model every call. That’s the thing to fix when going to production — replace subprocess with a long-lived ctypes wrapper around librkllmrt.so so the LLM stays warm. The wire contract above does not change.

models.yaml — paths must point at the files as downloaded in step 2; the Qwen2-VL build ships one .rknn (vision) and one .rkllm (LLM), not vision.rknn / llm.rkllm:

qwen2-vl-2b:
  vision: /home/gmorell/dms-vlm/models/qwen2-vl-2b/Qwen2-VL-2B-Instruct.rknn
  llm:    /home/gmorell/dms-vlm/models/qwen2-vl-2b/Qwen2-VL-2B-Instruct.rkllm
  max_context_len: 2048
  rknn_core_num: 3
  platform: rk3588          # rk3588 | rk3576 | rk3562 | rv1126b
  enabled_cpus_num: 4       # must be >= rknn_core_num and match the mask popcount
  enabled_cpus_mask: "0x3C" # 6-core CM5: both A76 (cpu4,5) + two A55 (cpu2,3)
  img_start: "<|vision_start|>"
  img_end:   "<|vision_end|>"
  img_content: "<|image_pad|>"

platform is mandatory — the demo takes it as argv[7] and refuses to start without a value it recognises. enabled_cpus_num / enabled_cpus_mask are only honoured by a demo built with patches/0001-demo-cpu-mask-env.patch; omit them on a stock 8-core board.

Known limitation: short descriptions

The thanhtantran/Qwen2-VL-2B-RKLLM W8A8 conversion emits EOS after a single clause, e.g. "The image depicts an astronaut on the moon,". This is the model, not the wiring: raising max_new_tokens from 256 to 512 returns byte-identical output, and enabling the demo’s chat template changes nothing. Try a different conversion or a larger VLM if fuller captions are needed.

The DMS worker then talks to http://r5bp-0:8080/describe — see workers.yml for the env-var wiring.