feat: add OpenVINO NPU prototype services

2026-06-04 11:41:55 -07:00
parent d67c259187
commit 5b01b1bd11
26 changed files with 2165 additions and 4 deletions
@@ -0,0 +1,159 @@
+# OpenVINO NPU document/image triage prototype
+
+Local-only prototype for triaging screenshots, photos/scans, and PDF page images.
+It returns structured JSON metadata and explicitly reports CPU vs NPU stages.
+
+Location: `/home/will/lab/swarm/openvino-doc-image-triage-npu/`
+
+## Privacy and safety
+
+- No external uploads.
+- The only network call is optional localhost-only embeddings at `127.0.0.1:18817`.
+- Raw OCR/sidecar text is redacted by default and is not logged.
+- Full source paths are omitted by default; responses include basename and SHA-256.
+- Allowed roots are enforced for CLI/server requests.
+- This prototype does not mutate Obsidian, RAG, Chroma, vector collections, routing, or gateway services.
+
+## CPU vs NPU stages
+
+CPU:
+- file intake, allowed-root checks, size checks, hashing
+- image/PDF decoding/rendering and normalization
+- optional local text extraction from sidecars or PDF text libraries
+- regex metadata extraction and rule-based category fallback
+- final needs-attention rules
+
+NPU:
+- needs-attention semantic embedding, via existing local OpenVINO embeddings service on `:18817`
+- verified with `/sys/class/accel/accel0/device/npu_busy_time_us` before/after each embedding call
+
+Not configured in v1:
+- image category classifier on NPU. The JSON reports this as `CPU rule fallback (NPU model not configured in prototype v1)`. A future task can add a static-shape MobileNet/EfficientNet/ResNet OpenVINO IR model.
+- OCR on NPU. OCR remains CPU/local plumbing in v1.
+
+## Files
+
+- `triage.py` — core library and CLI.
+- `server.py` — stdlib HTTP server with `/healthz`, `/models`, `/triage`, `/triage/batch`.
+- `make_samples.py` — creates synthetic non-private image/PDF samples.
+- `tests/smoke_test.py` — end-to-end smoke test, including NPU busy-time verification when `:18817` is reachable.
+- `samples/` — generated synthetic fixtures.
+
+## Requirements
+
+Use the existing NPU venv when available:
+
+```bash
+cd /home/will/lab/swarm/openvino-doc-image-triage-npu
+/home/will/.venvs/npu/bin/python -m pip install pillow
+```
+
+`pillow` is already present in the discovered `/home/will/.venvs/npu`. Optional local PDF text/rendering improves PDF support:
+
+```bash
+/home/will/.venvs/npu/bin/python -m pip install pypdf pypdfium2
+```
+
+The smoke tests do not require external services except the existing localhost `:18817` embeddings service for positive NPU verification.
+
+## CLI usage
+
+Generate synthetic samples:
+
+```bash
+cd /home/will/lab/swarm/openvino-doc-image-triage-npu
+/home/will/.venvs/npu/bin/python make_samples.py
+```
+
+Triage local files:
+
+```bash
+/home/will/.venvs/npu/bin/python triage.py \
+  --allowed-root /home/will/lab/swarm/openvino-doc-image-triage-npu \
+  --pretty \
+  samples/synthetic_invoice.png samples/synthetic_invoice.pdf
+```
+
+Disable the local NPU embeddings call if needed:
+
+```bash
+/home/will/.venvs/npu/bin/python triage.py --no-embeddings --allowed-root "$PWD" samples/synthetic_receipt.png
+```
+
+Include OCR/sidecar text in a single response only when explicitly requested:
+
+```bash
+/home/will/.venvs/npu/bin/python triage.py --include-ocr-text --allowed-root "$PWD" samples/synthetic_invoice.png
+```
+
+## HTTP usage
+
+Check that port 18820 is free first:
+
+```bash
+ss -ltnp | grep ':18820\b' || true
+```
+
+Start local-only server:
+
+```bash
+cd /home/will/lab/swarm/openvino-doc-image-triage-npu
+/home/will/.venvs/npu/bin/python server.py --host 127.0.0.1 --port 18820 --allowed-root "$PWD"
+```
+
+Call it:
+
+```bash
+curl -sS http://127.0.0.1:18820/healthz | jq
+curl -sS http://127.0.0.1:18820/models | jq
+curl -sS -X POST http://127.0.0.1:18820/triage \
+  -H 'Content-Type: application/json' \
+  -d '{"path":"/home/will/lab/swarm/openvino-doc-image-triage-npu/samples/synthetic_invoice.png","options":{"allowed_roots":["/home/will/lab/swarm/openvino-doc-image-triage-npu"]}}' | jq
+```
+
+## Smoke test
+
+```bash
+cd /home/will/lab/swarm/openvino-doc-image-triage-npu
+/home/will/.venvs/npu/bin/python tests/smoke_test.py
+```
+
+Expected: JSON ending with `"ok": true`. If the embeddings service is up, the result should show positive NPU busy-time delta and each embedded page should report `verified_npu: true`.
+
+## Example output shape
+
+```json
+{
+  "file_id": "sha256:...",
+  "source_path_basename": "synthetic_invoice.png",
+  "media_type": "image",
+  "page_count": 1,
+  "pages": [
+    {
+      "page_index": 0,
+      "classification": {
+        "label": "bill_or_invoice",
+        "confidence": 0.71,
+        "device": "CPU",
+        "method": "rule_based_fallback"
+      },
+      "needs_attention": {
+        "value": true,
+        "device": "NPU+CPU",
+        "reasons": ["amount_due", "due_date_present"],
+        "embedding": {"verified_npu": true, "npu_busy_delta_us": 12345}
+      },
+      "metadata": {"dates_count": 1, "amounts_count": 1, "raw_values_redacted": true},
+      "ocr": {"available": true, "device": "CPU"}
+    }
+  ],
+  "processing_device_summary": {
+    "file_intake": "CPU",
+    "image_category_classification": "CPU rule fallback (NPU model not configured in prototype v1)",
+    "needs_attention_embedding": "NPU via local :18817",
+    "metadata_extraction": "CPU",
+    "npu_verified": true
+  },
+  "privacy": {"external_uploads": false, "raw_text_logged": false}
+}
+```