Add workstation monitoring design
This commit is contained in:
OpenCode Test
296
docs/plans/2026-01-05-workstation-monitoring-design.md
Normal file
296
docs/plans/2026-01-05-workstation-monitoring-design.md
Normal file
@@ -0,0 +1,296 @@
|
|||||||
|
# Workstation Monitoring Design
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Deploy comprehensive monitoring for the Arch Linux workstation (willlaptop) by integrating with the existing k8s monitoring stack. This will enable proactive alerting for resource exhaustion, long-term capacity planning, and performance debugging.
|
||||||
|
|
||||||
|
**Reference:** Future consideration `fc-001` (workstation monitoring)
|
||||||
|
|
||||||
|
## Current Infrastructure
|
||||||
|
|
||||||
|
- **Workstation:** Arch Linux on MacBookPro9,2 (hostname: willlaptop)
|
||||||
|
- **K8s Cluster:** kube-prometheus-stack deployed with Prometheus, Alertmanager, Grafana
|
||||||
|
- **Network:** Direct network connectivity between workstation and cluster nodes
|
||||||
|
- **Existing Monitoring:** 3 node_exporters on cluster nodes, cluster-level alerts configured
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Components
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────┐ HTTP/9100 ┌──────────────────────┐
|
||||||
|
│ Workstation │ ──────────────────> │ K8s Prometheus │
|
||||||
|
│ (willlaptop) │ scrape every 15s │ (monitoring ns) │
|
||||||
|
│ │ │ │
|
||||||
|
│ node_exporter │ │ workstation rules │
|
||||||
|
│ systemd service│ │ + scrape config │
|
||||||
|
└─────────────────┘ └──────────────────────┘
|
||||||
|
│
|
||||||
|
v
|
||||||
|
┌──────────────────────┐
|
||||||
|
│ Alertmanager │
|
||||||
|
│ (existing setup) │
|
||||||
|
│ unified routing │
|
||||||
|
└──────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Flow
|
||||||
|
|
||||||
|
1. **node_exporter** exposes metrics on `http://willlaptop:9100/metrics`
|
||||||
|
2. **Prometheus** scrapes metrics every 15s via static target configuration
|
||||||
|
3. **PrometheusRule** evaluates workstation-specific alert rules
|
||||||
|
4. **Alertmanager** routes alerts to existing notification channels
|
||||||
|
|
||||||
|
## Workstation Deployment
|
||||||
|
|
||||||
|
### node_exporter Service
|
||||||
|
|
||||||
|
**Installation:**
|
||||||
|
```bash
|
||||||
|
pacman -S prometheus-node-exporter
|
||||||
|
```
|
||||||
|
|
||||||
|
**Systemd Configuration:**
|
||||||
|
- Service: `node_exporter.service`
|
||||||
|
- User: `node_exporter` (created by package)
|
||||||
|
- Listen address: `0.0.0.0:9100`
|
||||||
|
- Restart policy: `always` with 10s delay
|
||||||
|
- Logging: systemd journal (`journalctl -u node_exporter`)
|
||||||
|
|
||||||
|
**ExecStart flags:**
|
||||||
|
```bash
|
||||||
|
/usr/bin/node_exporter --collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($|/)
|
||||||
|
```
|
||||||
|
|
||||||
|
Excludes system mounts to reduce noise.
|
||||||
|
|
||||||
|
**Firewall Configuration:**
|
||||||
|
- Allow TCP 9100 from cluster nodes
|
||||||
|
- Use ufw or iptables to restrict access
|
||||||
|
|
||||||
|
**Metrics Collected:**
|
||||||
|
All default collectors except resource-intensive ones:
|
||||||
|
- CPU, memory, filesystem, network
|
||||||
|
- System stats (uptime, load average, systemd)
|
||||||
|
- Thermal (if available on hardware)
|
||||||
|
- Disk I/O
|
||||||
|
|
||||||
|
## Prometheus Configuration
|
||||||
|
|
||||||
|
### Static Scrape Target
|
||||||
|
|
||||||
|
**Job configuration:**
|
||||||
|
- Job name: `workstation/willlaptop`
|
||||||
|
- Target: `willlaptop:9100` (DNS resolution) or workstation IP
|
||||||
|
- Scrape interval: `15s` (matches cluster node_exporter)
|
||||||
|
- Scrape timeout: `10s`
|
||||||
|
- Metrics path: `/metrics`
|
||||||
|
- Honor labels: `true`
|
||||||
|
|
||||||
|
**Relabeling rules:**
|
||||||
|
- Add `env: "workstation"` label for identification
|
||||||
|
- Preserve `instance: "willlaptop"` from target
|
||||||
|
|
||||||
|
**Integration:**
|
||||||
|
Add to existing Prometheus CRD configuration in kube-prometheus-stack. This can be done via:
|
||||||
|
- PrometheusRule with additional scrape config, or
|
||||||
|
- Direct modification of Prometheus configuration
|
||||||
|
|
||||||
|
## Alert Rules
|
||||||
|
|
||||||
|
### PrometheusRule Resource
|
||||||
|
|
||||||
|
**Namespace:** `monitoring`
|
||||||
|
**Kind:** `PrometheusRule`
|
||||||
|
**Labels:** Standard discovery labels for Prometheus operator
|
||||||
|
|
||||||
|
### Alert Categories
|
||||||
|
|
||||||
|
#### Critical Alerts (Paging)
|
||||||
|
|
||||||
|
1. **WorkstationDiskSpaceCritical**
|
||||||
|
- Condition: `<5%` free on any mounted filesystem
|
||||||
|
- Duration: 5m
|
||||||
|
- Severity: critical
|
||||||
|
|
||||||
|
2. **WorkstationMemoryCritical**
|
||||||
|
- Condition: `>95%` memory usage
|
||||||
|
- Duration: 5m
|
||||||
|
- Severity: critical
|
||||||
|
|
||||||
|
3. **WorkstationCPUCritical**
|
||||||
|
- Condition: `>95%` CPU usage
|
||||||
|
- Duration: 10m
|
||||||
|
- Severity: critical
|
||||||
|
|
||||||
|
4. **WorkstationSystemdFailed**
|
||||||
|
- Condition: Failed systemd units detected
|
||||||
|
- Duration: 5m
|
||||||
|
- Severity: critical
|
||||||
|
|
||||||
|
#### Warning Alerts (Email/Slack)
|
||||||
|
|
||||||
|
1. **WorkstationDiskSpaceWarning**
|
||||||
|
- Condition: `<10%` free on any mounted filesystem
|
||||||
|
- Duration: 10m
|
||||||
|
- Severity: warning
|
||||||
|
|
||||||
|
2. **WorkstationMemoryWarning**
|
||||||
|
- Condition: `>85%` memory usage
|
||||||
|
- Duration: 10m
|
||||||
|
- Severity: warning
|
||||||
|
|
||||||
|
3. **WorkstationCPUWarning**
|
||||||
|
- Condition: `>80%` CPU usage
|
||||||
|
- Duration: 15m
|
||||||
|
- Severity: warning
|
||||||
|
|
||||||
|
4. **WorkstationLoadHigh**
|
||||||
|
- Condition: 5m load average > # CPU cores
|
||||||
|
- Duration: 10m
|
||||||
|
- Severity: warning
|
||||||
|
|
||||||
|
5. **WorkstationDiskInodeWarning**
|
||||||
|
- Condition: `<10%` inodes free
|
||||||
|
- Duration: 10m
|
||||||
|
- Severity: warning
|
||||||
|
|
||||||
|
6. **WorkstationNetworkErrors**
|
||||||
|
- Condition: High packet loss or error rate
|
||||||
|
- Duration: 10m
|
||||||
|
- Severity: warning
|
||||||
|
|
||||||
|
#### Info Alerts (Log Only)
|
||||||
|
|
||||||
|
1. **WorkstationDiskSpaceInfo**
|
||||||
|
- Condition: `<20%` free on any mounted filesystem
|
||||||
|
- Duration: 15m
|
||||||
|
- Severity: info
|
||||||
|
|
||||||
|
2. **WorkstationUptime**
|
||||||
|
- Condition: System uptime metric (recording rule)
|
||||||
|
- Severity: info
|
||||||
|
|
||||||
|
### Alert Annotations
|
||||||
|
|
||||||
|
Each alert includes:
|
||||||
|
- `summary`: Brief description
|
||||||
|
- `description`: Detailed explanation with metric values
|
||||||
|
- `runbook_url`: Link to troubleshooting documentation (if available)
|
||||||
|
|
||||||
|
## Versioning
|
||||||
|
|
||||||
|
### Repository Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
~/.claude/repos/homelab/charts/willlaptop-monitoring/
|
||||||
|
├── prometheus-rules.yaml # PrometheusRule for workstation alerts
|
||||||
|
├── values.yaml # Configuration values
|
||||||
|
└── README.md # Documentation
|
||||||
|
```
|
||||||
|
|
||||||
|
### Values.yaml Configuration
|
||||||
|
|
||||||
|
Configurable parameters:
|
||||||
|
```yaml
|
||||||
|
workstation:
|
||||||
|
hostname: willlaptop
|
||||||
|
ip: <workstation_ip> # optional, fallback to DNS
|
||||||
|
|
||||||
|
scrape:
|
||||||
|
interval: 15s
|
||||||
|
timeout: 10s
|
||||||
|
|
||||||
|
alerts:
|
||||||
|
disk:
|
||||||
|
critical_percent: 5
|
||||||
|
warning_percent: 10
|
||||||
|
info_percent: 20
|
||||||
|
memory:
|
||||||
|
critical_percent: 95
|
||||||
|
warning_percent: 85
|
||||||
|
cpu:
|
||||||
|
critical_percent: 95
|
||||||
|
critical_duration: 10m
|
||||||
|
warning_percent: 80
|
||||||
|
warning_duration: 15m
|
||||||
|
```
|
||||||
|
|
||||||
|
### Integration with ArgoCD
|
||||||
|
|
||||||
|
Follows existing GitOps pattern (charts/kube-prometheus-stack). Can be added to ArgoCD for automated deployments if desired.
|
||||||
|
|
||||||
|
## Testing and Verification
|
||||||
|
|
||||||
|
### Phase 1 - Workstation Deployment
|
||||||
|
|
||||||
|
1. Verify node_exporter installation:
|
||||||
|
```bash
|
||||||
|
pacman -Q prometheus-node-exporter
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Check systemd service status:
|
||||||
|
```bash
|
||||||
|
systemctl status node_exporter
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Verify metrics endpoint locally:
|
||||||
|
```bash
|
||||||
|
curl http://localhost:9100/metrics | head -20
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Test accessibility from cluster:
|
||||||
|
```bash
|
||||||
|
kubectl run -it --rm debug --image=curlimages/curl -- curl willlaptop:9100/metrics
|
||||||
|
```
|
||||||
|
|
||||||
|
### Phase 2 - Prometheus Integration
|
||||||
|
|
||||||
|
1. Verify Prometheus target:
|
||||||
|
- Access Prometheus UI → Targets → workstation/willlaptop
|
||||||
|
- Confirm target is UP
|
||||||
|
|
||||||
|
2. Verify metric ingestion:
|
||||||
|
```bash
|
||||||
|
# Query in Prometheus UI
|
||||||
|
node_up{instance="willlaptop"}
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Verify label injection:
|
||||||
|
- Confirm `env="workstation"` label appears on metrics
|
||||||
|
|
||||||
|
### Phase 3 - Alert Verification
|
||||||
|
|
||||||
|
1. Review PrometheusRule:
|
||||||
|
```bash
|
||||||
|
kubectl get prometheusrule workstation-alerts -n monitoring -o yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Verify rule evaluation:
|
||||||
|
- Access Prometheus UI → Rules
|
||||||
|
- Confirm workstation rules are active
|
||||||
|
|
||||||
|
3. Test critical alert:
|
||||||
|
- Temporarily trigger a low disk alert (or simulate)
|
||||||
|
- Verify alert fires in Prometheus UI
|
||||||
|
|
||||||
|
4. Verify Alertmanager integration:
|
||||||
|
- Check Alertmanager UI → Alerts
|
||||||
|
- Confirm workstation alerts are received
|
||||||
|
|
||||||
|
## Success Criteria
|
||||||
|
|
||||||
|
- [ ] node_exporter running on workstation
|
||||||
|
- [ ] Metrics accessible from cluster nodes
|
||||||
|
- [ ] Prometheus scraping workstation metrics
|
||||||
|
- [ ] Alert rules evaluated and firing correctly
|
||||||
|
- [ ] Alerts routing through Alertmanager
|
||||||
|
- [ ] Configuration versioned in homelab repository
|
||||||
|
- [ ] Documentation complete
|
||||||
|
|
||||||
|
## Future Enhancements
|
||||||
|
|
||||||
|
- Grafana dashboards for workstation metrics
|
||||||
|
- Alert tuning based on observed patterns
|
||||||
|
- Additional collectors (e.g., temperature sensors if available)
|
||||||
|
- Integration with morning-report skill for health status
|
||||||
Reference in New Issue
Block a user