unitforge/k8s/README.md

# UnitForge Raspberry Pi Kubernetes Deployment

Lightweight Kubernetes deployment optimized for Raspberry Pi 5 clusters.

## 🍓 Pi Cluster Requirements

### Hardware
- **Raspberry Pi 5** (4GB+ RAM recommended)
- **SD Card**: 32GB+ Class 10 or better
- **Network**: Gigabit Ethernet preferred
- **Power**: Official Pi 5 power supply

### Software
- **OS**: Raspberry Pi OS 64-bit or Ubuntu 22.04 LTS
- **Kubernetes**: 1.25+ (k3s recommended for Pi)
- **Container Runtime**: containerd or Docker
- **Ingress**: NGINX Ingress Controller

## 📁 Simplified Structure

```
k8s/
├── namespace.yaml      # Dedicated namespace
├── configmap.yaml      # Application configuration
├── deployment.yaml     # Single replica deployment
├── service.yaml        # ClusterIP service
├── ingress.yaml        # Local domain ingress
└── vpa.yaml            # Vertical Pod Autoscaler for Pi
```

**Simple kubectl deployment** - No kustomize, scripts, or complex tooling required!

## 🚀 Quick Deployment

### Apply All Resources
```bash
# Create namespace (optional)
kubectl create namespace unitforge

# Apply all resources to specific namespace
kubectl apply -f k8s/ -n unitforge

# Or apply to default namespace
kubectl apply -f k8s/

# Check status
kubectl get all -n unitforge
```

### Step-by-Step Deploy
```bash
# Create namespace (optional)
kubectl create namespace unitforge

# Apply resources in order
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/configmap.yaml -n unitforge
kubectl apply -f k8s/deployment.yaml -n unitforge
kubectl apply -f k8s/service.yaml -n unitforge
kubectl apply -f k8s/ingress.yaml -n unitforge
kubectl apply -f k8s/vpa.yaml -n unitforge
```

## 🔧 Pi Optimizations

### Resource Limits
- **Memory**: 128Mi request, 256Mi limit (VPA optimized)
- **CPU**: 100m request, 300m limit (VPA optimized)
- **Workers**: 2 (down from 4)
- **Replicas**: 1 (sufficient for Pi cluster)
- **VPA**: Enabled for automatic resource optimization

### Simplified Configuration
- **No HPA**: Single replica, manual scaling
- **No TLS**: HTTP only for simplicity
- **No NetworkPolicy**: Simplified networking
- **No Kustomize**: Standard kubectl commands only
- **Basic probes**: Longer timeouts for Pi performance
- **Minimal metrics**: Disabled to save resources

### Local Domains
- `unitforge.local` - Primary access
- `unitforge.pi` - Alternative domain

## 🌐 Access Setup

### Add to /etc/hosts
```bash
# Replace <pi-ip> with your Pi node IP
echo "<pi-ip> unitforge.local" >> /etc/hosts
echo "<pi-ip> unitforge.pi" >> /etc/hosts
```

### Get Node IP
```bash
# Find your Pi node IP
kubectl get nodes -o wide

# Or get ingress IP
kubectl get ingress unitforge -n unitforge
```

### Access Methods
```bash
# Browser access
http://unitforge.local
http://unitforge.pi

# Port forwarding
kubectl port-forward service/unitforge 8080:80 -n unitforge
# Then: http://localhost:8080

# Direct pod access
kubectl port-forward pod/<pod-name> 8080:8000 -n unitforge
```

## 📊 Pi Cluster Management

### Check Deployment
```bash
# Overall status (replace unitforge with your namespace)
kubectl get all -n unitforge

# Pod details
kubectl describe pod -l app=unitforge -n unitforge

# Logs
kubectl logs -f deployment/unitforge -n unitforge

# Resource usage
kubectl top pods -n unitforge

# If using current namespace, omit -n flag
kubectl get all
kubectl logs -f deployment/unitforge
```

### Common Operations
```bash
# Restart deployment (adjust namespace as needed)
kubectl rollout restart deployment/unitforge -n unitforge

# Scale (if needed)
kubectl scale deployment unitforge --replicas=2 -n unitforge

# Update image
kubectl set image deployment/unitforge unitforge=new-image:tag -n unitforge

# Delete everything
kubectl delete namespace unitforge
# Or if using default namespace
kubectl delete -f k8s/
```

### Troubleshooting
```bash
# Check events (adjust namespace as needed)
kubectl get events -n unitforge --sort-by='.lastTimestamp'

# Pod shell access
kubectl exec -it deployment/unitforge -n unitforge -- /bin/bash

# Check node resources
kubectl describe node <pi-node-name>

# Check disk space
kubectl exec -it deployment/unitforge -n unitforge -- df -h

# For current namespace, omit -n flag
kubectl get events --sort-by='.lastTimestamp'
kubectl exec -it deployment/unitforge -- /bin/bash
```

## 🤖 Vertical Pod Autoscaler (VPA)

### VPA Benefits for Pi Clusters
- **Automatic optimization**: Adjusts resource requests based on actual usage
- **Pi hardware awareness**: Conservative limits for ARM64 architecture
- **Memory efficiency**: Critical for Pi nodes with limited RAM
- **Cost optimization**: Right-sizing for Pi cluster resources

### VPA Configuration
```bash
# Check if VPA is installed
kubectl get crd verticalpodautoscalers.autoscaling.k8s.io

# Apply VPA configuration
kubectl apply -f vpa.yaml -n unitforge

# Monitor VPA recommendations
kubectl get vpa unitforge-vpa -n unitforge
kubectl describe vpa unitforge-vpa -n unitforge

# Check current pod resources
kubectl get pods -l app=unitforge -n unitforge -o wide
kubectl top pods -l app=unitforge -n unitforge
```

### VPA Resource Bounds (Pi Optimized)
```yaml
# CPU bounds for Pi 5
minAllowed:
  cpu: 50m        # Minimal baseline
  memory: 64Mi    # Absolute minimum
maxAllowed:
  cpu: 500m       # Conservative Pi 5 limit
  memory: 512Mi   # Safe for 4-8GB Pi
```

### Installing VPA on Pi Cluster
```bash
# Install VPA components (if not already installed)
kubectl apply -f https://github.com/kubernetes/autoscaler/releases/latest/download/vpa-release.yaml

# Verify VPA installation
kubectl get pods -n kube-system | grep vpa

# Check VPA CRDs
kubectl get crd | grep verticalpodautoscaler
```

### VPA Management Commands
```bash
# Check VPA status
kubectl get vpa unitforge-vpa -n unitforge -o wide

# Apply VPA to unitforge namespace
kubectl apply -f vpa.yaml -n unitforge

# Monitor resource usage
kubectl top pods -l app=unitforge -n unitforge
kubectl describe pods -l app=unitforge -n unitforge

# Restart deployment to apply recommendations
kubectl rollout restart deployment/unitforge -n unitforge

# Delete VPA if needed
kubectl delete vpa unitforge-vpa -n unitforge
```

### VPA Monitoring
```bash
# View VPA recommendations
kubectl describe vpa unitforge-vpa -n unitforge

# Check current vs recommended resources
kubectl get vpa unitforge-vpa -n unitforge -o yaml

# Monitor pod resource usage
kubectl top pods -l app=unitforge -n unitforge --containers

# Pi-specific monitoring
vcgencmd measure_temp    # Pi temperature
htop                     # System resources
iostat -x 1 5           # I/O performance

# Simple VPA status check
kubectl get vpa -n unitforge
```

## 🔧 Configuration

### Environment Variables (ConfigMap)
```yaml
# Key Pi-optimized settings
WORKERS: "2"                    # Reduced for Pi
MAX_CONNECTIONS: "50"           # Lower limit
REQUEST_TIMEOUT: "60"           # Longer for Pi
ENABLE_API_METRICS: "false"     # Disabled to save resources
COMPRESS_RESPONSES: "true"      # Reduce bandwidth
```

### Resource Tuning
```yaml
# For Pi 4 (4GB) - reduce further
resources:
  requests:
    memory: "96Mi"
    cpu: "50m"
  limits:
    memory: "192Mi"
    cpu: "200m"

# For Pi 5 (8GB) - can increase
resources:
  requests:
    memory: "256Mi"
    cpu: "200m"
  limits:
    memory: "512Mi"
    cpu: "500m"
```

## 🍓 Pi-Specific Tips

### Performance
- **Use fast SD cards**: Class 10, U3, or A1/A2 rated
- **Enable cgroups**: Add to `/boot/cmdline.txt`:
  ```
  cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
  ```
- **Increase swap**: For memory-constrained Pis
- **Use SSD**: Boot from SSD for better I/O performance

### Networking
- **Use wired connections**: Ethernet preferred over WiFi
- **Local DNS**: Consider Pi-hole for local resolution
- **MetalLB**: For LoadBalancer services in bare metal

### Storage
- **EmptyDir volumes**: Used for temporary files
- **Local storage**: Consider local-path-provisioner
- **NFS**: For shared storage across Pi nodes

## 🔍 Monitoring on Pi

### Basic Monitoring
```bash
# Node resources
kubectl top nodes

# Pod resources (adjust namespace as needed)
kubectl top pods -n unitforge
# Or current namespace
kubectl top pods

# System resources on Pi
htop
iostat
free -h
df -h
```

### Simple Metrics
```bash
# Application health
curl http://unitforge.local/health

# Basic load test
for i in {1..10}; do
  curl -s http://unitforge.local/health > /dev/null
  echo "Request $i completed"
done

```bash
# VPA resource monitoring
kubectl get vpa -n unitforge
kubectl describe vpa unitforge-vpa -n unitforge

# Watch VPA recommendations
watch kubectl describe vpa unitforge-vpa -n unitforge
```

## 🚨 Common Pi Issues

### Memory Pressure
```bash
# Check memory usage
kubectl describe node <pi-node>

# Check pod memory usage
kubectl top pods -n unitforge

# Check VPA recommendations
kubectl describe vpa unitforge-vpa -n unitforge

# Apply VPA recommendations
kubectl rollout restart deployment/unitforge -n unitforge

# Reduce resource requests manually if needed
# Edit deployment.yaml resources section
```

### Storage Full
```bash
# Check disk usage (adjust namespace as needed)
kubectl exec -it deployment/unitforge -n unitforge -- df -h

# Clean Docker images on nodes
docker system prune -f
```

### Slow Performance
```bash
# Check I/O wait
iostat -x 1 5

# Consider moving to SSD
# Check SD card health
```

### Network Issues
```bash
# Check ingress controller
kubectl get pods -n ingress-nginx

# Test internal networking (adjust namespace as needed)
kubectl exec -it deployment/unitforge -n unitforge -- wget -qO- http://unitforge/health

# Test service connectivity
kubectl exec -it deployment/unitforge -n unitforge -- wget -qO- http://unitforge.unitforge.svc.cluster.local/health
```

## 📚 Standard Kubernetes Workflows

### Deployment Management
```bash
# Check deployment status
kubectl rollout status deployment/unitforge -n unitforge

# View deployment history
kubectl rollout history deployment/unitforge -n unitforge

# Rollback deployment
kubectl rollout undo deployment/unitforge -n unitforge

# Restart deployment
kubectl rollout restart deployment/unitforge -n unitforge
```

### Configuration Updates
```bash
# Update configmap
kubectl apply -f k8s/configmap.yaml -n unitforge

# Force pod restart to pick up config changes
kubectl rollout restart deployment/unitforge -n unitforge

# Edit configmap directly
kubectl edit configmap unitforge-config -n unitforge
```

### Resource Management
```bash
# Scale deployment
kubectl scale deployment unitforge --replicas=2 -n unitforge

# VPA-based resource optimization
kubectl apply -f vpa.yaml -n unitforge
kubectl rollout restart deployment/unitforge -n unitforge

# Manual resource updates (if VPA not used)
kubectl patch deployment unitforge -n unitforge -p '{"spec":{"template":{"spec":{"containers":[{"name":"unitforge","resources":{"limits":{"memory":"512Mi"}}}]}}}}'

# Get resource usage and VPA recommendations
kubectl top pods -n unitforge
kubectl describe vpa unitforge-vpa -n unitforge
kubectl describe deployment unitforge -n unitforge
```

## 📚 Pi Cluster Resources

### K3s Installation
```bash
# Master node
curl -sfL https://get.k3s.io | sh -

# Worker nodes
curl -sfL https://get.k3s.io | K3S_URL=https://<master-ip>:6443 K3S_TOKEN=<token> sh -
```

### Useful Pi Tools
- **k3s**: Lightweight Kubernetes for Pi
- **kubectl**: Standard Kubernetes CLI (no kustomize needed)
- **k9s**: Terminal UI for Kubernetes
- **htop**: System monitoring
- **vcgencmd**: Pi-specific commands

## 🎯 Production on Pi

### High Availability
- **Multiple Pi nodes**: 3+ for redundancy
- **Shared storage**: NFS or distributed storage
- **Load balancing**: MetalLB or external LB
- **Backup strategy**: Regular etcd backups

### Security
- **Network segmentation**: VLANs for cluster traffic
- **Firewall rules**: iptables or ufw configuration
- **Regular updates**: Keep Pi OS and k3s updated
- **Monitoring**: Basic Prometheus setup

### Scaling
- **Horizontal**: Add more Pi nodes
- **Vertical**: Upgrade to Pi 5 with more RAM
- **Storage**: Add USB SSDs for better performance
- **Network**: Ensure gigabit switches

---

**UnitForge on Pi** - Making systemd management portable! 🍓🚀