rknn-describe-server

A small HTTP server that runs on the Rockchip board (RK3588 / RK3576 — e.g. r5bp-0) and exposes the airockchip/rknn-llm multimodal demo over a simple multipart endpoint that the DMS describe worker can call.

The DMS Rust worker stays on x86; this server is the only thing that needs to live on the NPU host.

Wire protocol

POST /describe — multipart/form-data:

field	required	notes
image	yes	binary image (jpg/png/webp/…)
prompt	no	overrides default prompt
model	no	name registered in models.yaml (default: first entry)
max_new_tokens	no	integer, default 256

Response:

{ "description": "A red barn at dusk with a tractor in front.",
  "model": "qwen2-vl-2b",
  "tokens": 27 }

GET /healthz — 200 ok once the model is loaded.

Approach

The reference implementation here just shells out to the upstream multimodal_demo binary per request. That keeps the integration minimal but reloads the model every call — fine for a smoke test, far too slow for production.

For sustained throughput, replace subprocess with a long-lived ctypes wrapper around librkllmrt.so (see rkllm.h in the upstream repo) that keeps the language model warm across requests. The request/response shape above does not need to change.

Setup on r5bp-0 (RK3588, librknnrt 2.3.0 already shipped by Armbian)

The bring-up below is what we ran on r5bp-0 and should reproduce on any RK3588 board with the same OS image.

sudo apt install -y git cmake g++ python3-venv python3-pip

# 1. clone the upstream toolkit and build the multimodal demo (the cmake
#    build-linux.sh assumes a cross-compiler — on-device just call cmake
#    directly with the system toolchain).
mkdir -p ~/dms-vlm && cd ~/dms-vlm
git clone --depth 1 https://github.com/airockchip/rknn-llm.git
cd rknn-llm/examples/multimodal_model_demo/deploy
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release \
         -DCMAKE_SYSTEM_NAME=Linux \
         -DCMAKE_SYSTEM_PROCESSOR=aarch64
make -j8 install
# ⇒ ./install/demo_Linux_aarch64/{demo,imgenc,lib/}

# 2. fetch a converted VLM. The Qwen2-VL-2B build below totals ~3.3 GiB.
mkdir -p ~/dms-vlm/models/qwen2-vl-2b && cd ~/dms-vlm/models/qwen2-vl-2b
wget -O Qwen2-VL-2B-Instruct.rkllm \
  https://huggingface.co/thanhtantran/Qwen2-VL-2B-RKLLM/resolve/main/Qwen2-VL-2B-Instruct.rkllm
wget -O Qwen2-VL-2B-Instruct.rknn \
  https://huggingface.co/thanhtantran/Qwen2-VL-2B-RKLLM/resolve/main/Qwen2-VL-2B-Instruct.rknn

# 3. drop server.py + models.yaml into ~/dms-vlm, create a venv, install deps
cd ~/dms-vlm
python3 -m venv venv
./venv/bin/pip install fastapi "uvicorn[standard]" python-multipart pyyaml

# 4. install the user-level systemd unit and enable linger so it keeps
#    running across reboots without an active login session.
mkdir -p ~/.config/systemd/user
install -m 644 dms-vlm.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now dms-vlm.service
sudo loginctl enable-linger "$USER"

Smoke test from another host:

curl -s http://r5bp-0:8080/healthz
curl -F image=@/path/to/sample.jpg http://r5bp-0:8080/describe

First request takes ~10–15 s because the demo binary reloads the model every call. That’s the thing to fix when going to production — replace subprocess with a long-lived ctypes wrapper around librkllmrt.so so the LLM stays warm. The wire contract above does not change.

models.yaml example:

qwen2-vl-2b:
  vision: /opt/rknn-llm/models/qwen2-vl-2b/vision.rknn
  llm:    /opt/rknn-llm/models/qwen2-vl-2b/llm.rkllm
  max_context_len: 2048
  rknn_core_num: 3
  img_start: "<|vision_start|>"
  img_end:   "<|vision_end|>"
  img_content: "<|image_pad|>"
minicpm-v-2_6:
  vision: /opt/rknn-llm/models/minicpm-v-2_6/vision.rknn
  llm:    /opt/rknn-llm/models/minicpm-v-2_6/llm.rkllm
  max_context_len: 2048
  rknn_core_num: 3
internvl2-1b:
  vision: /opt/rknn-llm/models/internvl2-1b/vision.rknn
  llm:    /opt/rknn-llm/models/internvl2-1b/llm.rkllm
  max_context_len: 2048
  rknn_core_num: 3

The DMS worker then talks to http://r5bp-0:8080/describe — see workers.yml for the env-var wiring.