will/swarm-master
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
06cd49247af75ef520f6503bb5aca6a75a5b04a0
swarm-master/docs/swarm-infrastructure.md
T
William Valentin 06cd49247a chore(rag): enable NPU reranker by default
2026-06-04 15:01:26 -07:00

258 lines
11 KiB
Markdown
Raw Blame History

# Swarm Infrastructure
This document is the source-of-truth overview for Will's local swarm/agent infrastructure on the `zap` workstation. It focuses on the runtime services that support Atlas/Hermes, n8n automation, local model/search/voice tooling, Obsidian/RAG automation, and the new agentmon monitoring layer.
## High-level topology
```text
Telegram / Discord / Email
|
v
Hermes / Atlas gateway (default profile)
|
+--> local tools and specialist profiles
+--> n8n automation workflows on :18808
n8n automation
|
+--> direct watchdog probes for key service ports
+--> Agentmon Health Watchdog -> agentmon-query :8081
+--> Obsidian, RAG, voice memo, URL capture, digest workflows
agentmon
|
+--> agentmon-swarm-monitor -> Docker labels agentmon.monitor=true
+--> agentmon-openclaw-monitor -> OpenClaw VM snapshots
+--> NATS JetStream -> event processor -> Postgres
+--> query API / UI on :8081 / :8082
local AI/search/voice services
|
+--> LiteLLM :18804
+--> SearXNG :18803
+--> Brave MCP :18802
+--> llama.cpp :18806
+--> Ollama embeddings :18807 (legacy/CPU fallback)
+--> OpenVINO NPU embeddings :18817
+--> Kokoro TTS :18805
+--> Whisper NPU :18816
+--> approved/not-live NPU sidecars: reranker :18818, router/classifier :18819, GenAI worker :18820, doc/image triage optional :18829
```
See also:
- [`swarm-infrastructure.html`](./swarm-infrastructure.html) — visual architecture diagram
- [`diagram-maintenance.md`](./diagram-maintenance.md) — how to keep diagrams updated and when to create new ones
## Runtime layers
### 1. Messaging and agent gateway
- **Hermes / Atlas default profile** is the production messaging gateway.
- Connected platforms include Telegram, Discord, and email.
- Atlas uses local swarm services where suitable, especially search, local LLMs, embeddings, STT/TTS, n8n, and agentmon.
- Specialist Hermes profiles are available for delegated work, but the default profile remains the stable production gateway.
### 2. n8n automation
Container/service:
- `n8n-agent`
- Host URL: `http://127.0.0.1:18808`
- Container URL: `http://127.0.0.1:5678`
- Compose project: `/home/will/lab/swarm/docker-compose.yaml`
Important workflow source exports live under:
- `swarm-common/n8n-workflows/`
Current health/automation patterns:
- **Swarm Health Watchdog**: direct endpoint checks for search, LLM, voice, n8n, Docker health, etc.
- **Agentmon Health Watchdog**: polls agentmon aggregate snapshots and alerts on stale/degraded monitoring state.
- **RAG and Embedding Health Watchdog**: checks RAG/search/embedding path.
- Obsidian workflows: health/reindex, inbox triage, daily review, URL-to-note, chat summary capture, weekly decision/runbook extraction.
### 3. Agentmon monitoring layer
Repo:
- `/home/will/lab/agentmon`
Compose services:
- `agentmon-ingest` on `:8080` — ingestion gateway, `/healthz`
- `agentmon-query` on `:8081` — query API, `/healthz`, `/v1/events`, `/v1/stats/summary`
- `agentmon-ui` on `:8082` — web UI, `/healthz`
- `agentmon-processor` — NATS to Postgres event processor
- `agentmon-swarm-monitor` — monitors Docker containers labeled `agentmon.monitor=true`
- `agentmon-openclaw-monitor` — emits OpenClaw VM snapshots
- `agentmon-db` — Postgres
- `agentmon-nats` — NATS JetStream
Key query endpoints:
```text
http://127.0.0.1:8080/healthz
http://127.0.0.1:8081/healthz
http://127.0.0.1:8082/healthz
http://127.0.0.1:8081/v1/stats/summary
http://127.0.0.1:8081/v1/events?event_type=swarm.snapshot&limit=1
http://127.0.0.1:8081/v1/events?event_type=swarm.service.snapshot&limit=20
http://127.0.0.1:8081/v1/events?event_type=openclaw.snapshot&limit=3
```
From inside `n8n-agent`, use the Docker bridge gateway:
```text
http://172.19.0.1:8081/v1/events?event_type=swarm.snapshot&limit=1
```
### 4. Local AI, search, and voice services
Docker services:
- `litellm` — `:18804`, OpenAI-compatible LLM router
- `litellm-db` — Postgres backing LiteLLM
- `searxng` — `:18803`, local metasearch
- `brave-search` — `:18802`, Brave Search MCP server
- `kokoro-tts` — `:18805`, local TTS
- `whisper-server-npu` — `:18816`, OpenVINO NPU local transcription
- `n8n-agent` — `:18808`, automation
Host/user services:
- `llama-server.service` — `:18806`, local llama.cpp OpenAI-compatible LLM
- `ollama.service` — `:18807`, legacy/CPU embeddings API fallback
- `openvino-embeddings.service` — `:18817`, OpenVINO NPU embeddings API (`/v1/embeddings`, `/api/embed`, `/api/embeddings`)
- `docker-health-endpoint.service` — `:18809`, read-only container health for n8n
- `obsidian-reindex-endpoint.service` — `:18810`, Obsidian/RAG reindex trigger and `/semantic-search`; default collection `obsidian_bge_npu` using OpenVINO NPU embeddings, with optional request-time `:18818` reranking disabled by default
- `url-content-extractor.service` — `:18812`, YouTube/PDF/web extraction
- `voice-memo-processor.service` — `:18813`, voice memo processing
- `rag-embedding-health.service` — `:18814`, RAG/embedding health wrapper
Approved but not live-routed OpenVINO NPU sidecars:
| Port | Component | State | Safety boundary |
| ---: | --- | --- | --- |
| `18818` | reranker | approved prototype; optional foreground/user-systemd only | request-time only; no Chroma/vector mutation; no live RAG integration unless Will approves |
| `18819` | router/classifier | approved prototype; dry-run only | no Hermes/Atlas routing, memory writes, service restarts, or outbound messages |
| `18820` | bounded GenAI worker | approved prototype | background jobs only; not primary Atlas/Hermes model routing |
| `18829` | document/image triage | CLI-first; optional localhost server | synthetic/non-private smoke data only; no private directory processing; NPU stage is embeddings via `:18817` |
These sidecars must bind to `127.0.0.1` by default, must not be enabled persistently or wired into live Atlas/Hermes/RAG paths without explicit Will approval, and any NPU claim requires a positive `/sys/class/accel/accel0/device/npu_busy_time_us` delta before/after inference. HTTP 200 alone is not proof.
### 5. Obsidian and RAG
Vault:
- `/home/will/lab/swarm/swarm-common/obsidian-vault/will/will-shared-zap`
Local REST API:
- HTTP: `127.0.0.1:27123`
- HTTPS: `127.0.0.1:27124`
RAG/vector store:
- ChromaDB path: `~/.hermes/data/rag-search/chroma/`
- Reindex state/progress: active BGE/NPU state in `~/.hermes/data/rag-search/obsidian_bge_npu_index_state.json` and `obsidian_bge_npu_reindex_progress.json`; legacy Ollama state in `obsidian_index_state.json` remains for comparison/fallback.
- Active RAG query/reindex embedding backend: OpenVINO NPU embeddings service on `:18817`, currently `bge-base-en-v1.5-int8-ov`, collection `obsidian_bge_npu`.
- Legacy comparison/fallback collection: `obsidian`, built with Ollama on `:18807` using `nomic-embed-text`.
- Reindex/search endpoint: `POST :18810/reindex` for incremental updates, `POST :18810/reindex?full=true` for full semantic rebuilds, `GET :18810/semantic-health` to verify vectors plus a search smoke test, and `POST :18810/semantic-search` for n8n/Hermes semantic context lookup.
- Reranker path: `RAG_RERANK_ENABLED=true` for `:18810/semantic-search` after local bake testing. `/semantic-search` retrieves `RAG_RERANK_INITIAL_K` vector candidates, calls `RAG_RERANK_URL` (`http://127.0.0.1:18818/rerank`), returns reranked `RAG_RERANK_TOP_K`, requires positive `npu_busy_delta_us` by default (`RAG_RERANK_REQUIRE_NPU_PROOF=true`), and falls back to vector order with `rerank.error` metadata on timeout/error/non-positive NPU proof. Reranking is request-time only and must not mutate Chroma/vector collections.
## Monitoring model
The monitoring design is intentionally layered:
1. **n8n direct probes** check critical service endpoints and send deduped alerts.
2. **agentmon** continuously observes labeled Docker services and OpenClaw state, then writes snapshots through NATS/Postgres.
3. **n8n Agentmon Health Watchdog** polls agentmon's aggregate state and alerts if the monitoring pipeline itself becomes stale/degraded.
4. **Hermes/Atlas** can inspect both n8n and agentmon when troubleshooting, and can use the same endpoints as part of operational checks.
This means a single process being alive is not enough: the important signal is whether collection, ingestion, processing, storage, query, and alerting are all functioning.
## Agentmon Health Watchdog
Workflow source:
- `swarm-common/n8n-workflows/agentmon-health-watchdog.json`
Installed n8n workflow:
- Name: `Agentmon Health Watchdog`
- ID: `AgentmonHealthWatchdog`
- Schedule: every 5 minutes
Alert conditions:
- `agentmon-ingest`, `agentmon-query`, or `agentmon-ui` `/healthz` fails.
- Latest `swarm.snapshot` is missing.
- Latest `swarm.snapshot` is older than 3 minutes.
- Snapshot issues are non-empty.
- Required agentmon services are missing or not healthy/running:
- `agentmon-ingest`
- `agentmon-query`
- `agentmon-ui`
- `agentmon-processor`
- `agentmon-swarm-monitor`
- `agentmon-db`
- `agentmon-nats`
Deduplication:
- Alert after 2 failed checks.
- Reminder every 6 failed runs.
- Recovery message when state returns healthy.
## Operational quick checks
From the host:
```bash
cd /home/will/lab/swarm
make status
make local-ai-health
./scripts/npu-service-health.sh # read-only; includes sysfs busy-time proof for :18817
curl -fsS http://127.0.0.1:18810/semantic-health | jq '{status,state,search_ok,result_count}'
curl -fsS http://127.0.0.1:18810/semantic-search \
-H 'Content-Type: application/json' \
-d '{"query":"non-private semantic smoke","top_k":2}' \
| jq '{ok,index,top_k,search_k,rerank,result_count}'
curl -fsS http://127.0.0.1:18808/healthz
curl -fsS http://127.0.0.1:8081/healthz
curl -fsS 'http://127.0.0.1:8081/v1/events?event_type=swarm.snapshot&limit=1' | jq .
```
From inside `n8n-agent`:
```bash
docker exec n8n-agent /bin/sh -lc '
wget -qO- -T 5 http://172.19.0.1:18810/healthz
wget -qO- -T 5 http://172.19.0.1:18814/healthz
wget -qO- -T 5 http://172.19.0.1:18817/healthz | head -c 500
'
```
Verify n8n workflow activation:
```bash
docker exec -u node n8n-agent n8n export:workflow \
--id=AgentmonHealthWatchdog \
--output=/tmp/agentmon-export.json
docker cp n8n-agent:/tmp/agentmon-export.json /tmp/agentmon-export.json
jq '.[0] | {id,name,active,nodes:(.nodes|length)}' /tmp/agentmon-export.json
```
## Notes and pitfalls
- Do not commit `.env`, decrypted credentials, raw credential exports, or runtime DB files.
- n8n workflow backups can contain sensitive operational data; keep timestamped raw backups untracked unless intentionally sanitized.
- From host, use `127.0.0.1:<host-port>`.
- From `n8n-agent`, use `127.0.0.1:5678` for n8n itself and `172.19.0.1:<host-port>` for host-published swarm services.
- Agentmon `/healthz` only proves the web/API process is alive; pair it with snapshot freshness to prove the monitoring pipeline is flowing.
- OpenClaw is intentionally dormant unless explicitly re-enabled; do not alert on VMs being shut off by default.
- OpenVINO NPU sidecars on `:18819`, `:18820`, and optional `:18829` are prototypes/not-live unless a later approved change installs and routes them. The `:18818` reranker is live as a local request-time second stage for `:18810/semantic-search`; it still falls back to vector order on timeout/error/non-positive NPU proof. Do not draw live Atlas/Hermes/classifier/GenAI arrows to prototypes until approval and implementation actually exist.
Reference in New Issue View Git Blame Copy Permalink