# Inference Cost Optimization Plan

**Goal**: Reduce LLM inference costs without quality loss using OpenClaw's built-in configuration knobs + smarter subagent model selection. No code changes to OpenClaw — config-only, fully upstream-compatible.

**Date**: 2026-03-05
**Status**: Planning

---

## Current State

| Item | Value |
|------|-------|
| Main session model | `litellm/copilot-claude-opus-4.6` (via GitHub Copilot) |
| Default agent model | `litellm/copilot-claude-sonnet-4.6` |
| Prompt caching | **NOT SET** (no `cacheRetention` configured) |
| Context pruning | **NOT SET** (no `contextPruning` configured) |
| Heartbeat | 30m (main agent only) |
| Subagent model | Inherits session model (expensive!) |
| Free models available | `zai/glm-4.7`, `zai/glm-4.7-flash`, `zai/glm-4.7-flashx`, `zai/glm-5` (all $0) |
| Copilot models | Flat-rate via GitHub Copilot subscription (effectively $0 marginal cost per token) |

### Cost Structure
- **Copilot models** (litellm/copilot-*): Covered by GitHub Copilot subscription — no per-token cost, but subject to rate limits and quotas. Using Opus when Sonnet suffices wastes quota.
- **ZAI models** (zai/glm-*): Free tier, no per-token cost. Quality varies by task type.
- The real "cost" is: (a) Copilot quota burn on expensive models, (b) latency, (c) quality risk on cheaper models.

---

## Phase 1: Enable Prompt Caching

**What**: Configure `cacheRetention` on Anthropic-backed models so repeated system prompts and stable context get cached by the provider.

**Why**: Our system prompt (AGENTS.md + SOUL.md + USER.md + TOOLS.md + IDENTITY.md + HEARTBEAT.md + skills list) is large and mostly static. Without caching, every turn reprocesses ~15-20k tokens of identical prefix. With caching, subsequent turns pay ~10% for cached tokens (Anthropic pricing).

**Applies only to Claude-backed models.** GPT, GLM, and Gemini models do NOT support Anthropic's `cacheRetention` mechanism. OpenAI caching is automatic (no config needed). ZAI/GLM has no caching mechanism.

**Cache TTL reality check** (from official Anthropic docs):
- `short` (`cacheRetention`) = **5-minute TTL** (default — refreshed on each use within the window)
- `long` = **1-hour TTL** (at higher write cost: 2x base input price vs 1.25x for short)
- Cache reads cost 0.1x (10%) of base input — so a cache hit on a 15k-token system prompt costs 90% less
- First turn writes cache (slightly more expensive), subsequent turns read it (very cheap)

**Implication**: With 5-minute default TTL and a 30-minute heartbeat, the cache expires between every heartbeat. Either:
1. Use `long` (1h TTL) and set heartbeat to 55m to keep warm — best for cost savings
2. Use `short` (5m TTL) with no heartbeat adjustment — cache only helps within active bursts

**Recommendation**: Use `long` on main session Claude models + 25m heartbeat (well within 1h). Cache writes are slightly more expensive but the read savings dominate for any active session.

**Config change** (`~/.openclaw/openclaw.json`):
```json
{
  "agents": {
    "defaults": {
      "models": {
        "litellm/copilot-claude-opus-4.6": {
          "params": {
            "cacheRetention": "long"
          }
        },
        "litellm/copilot-claude-sonnet-4.6": {
          "params": {
            "cacheRetention": "long"
          }
        },
        "litellm/copilot-claude-opus-4.5": {
          "params": {
            "cacheRetention": "long"
          }
        },
        "litellm/copilot-claude-sonnet-4.5": {
          "params": {
            "cacheRetention": "long"
          }
        },
        "litellm/copilot-claude-haiku-4.5": {
          "params": {
            "cacheRetention": "short"
          }
        }
      }
    }
  }
}
```

Note: No config needed for GPT models — OpenAI caches automatically for free on prompts ≥1024 tokens. No config available for ZAI/GLM models (no caching support).

**Verification**:
1. After applying, check `/status` or `/usage full` for `cacheRead` vs `cacheWrite` tokens.
2. Enable cache trace diagnostics temporarily:
   ```json
   { "diagnostics": { "cacheTrace": { "enabled": true } } }
   ```
3. First turn will show high `cacheWrite` (populating cache). Subsequent turns should show high `cacheRead` with much lower `cacheWrite`.
4. Target: >60% cache hit rate within 2-3 turns of a session.

**Risk**: Zero. Caching doesn't change outputs — it's purely a provider-side optimization.

**Expected impact**: 40-60% reduction in input token processing cost for sessions with multiple turns.

---

## Phase 2: Heartbeat Cache Warming

**What**: Align heartbeat interval to keep the 1-hour prompt cache warm across idle gaps.

**Why**: With `cacheRetention: "long"` (1h TTL), the cache expires after 1 hour of no activity. A heartbeat just under 1h ensures the cache is touched before it expires, so the next real interaction reads from cache instead of rewriting it. Our current 30m heartbeat already works, but 25m gives a safety margin.

**Important**: Heartbeat keep-warm only applies to **Claude models**. GPT/GLM models don't benefit — their caching is either automatic (OpenAI) or non-existent (ZAI).

**Config change** (`~/.openclaw/openclaw.json`):
```json
{
  "agents": {
    "defaults": {
      "heartbeat": {
        "every": "55m"
      }
    },
    "list": [
      {
        "id": "main",
        "heartbeat": {
          "every": "25m"
        }
      }
    ]
  }
}
```

**Rationale**:
- Main agent: keep at 25m (well within 1h TTL, ensures cache stays warm during active use)
- Other agents (claude, codex, copilot, opencode): 55m default (just under 1h TTL, minimal quota burn when idle)
- If an agent is rarely used, its heartbeat won't fire (disabled agents skip heartbeat)

**Verification**:
1. After a 30-minute idle gap, check that the next interaction shows `cacheRead` (not all `cacheWrite`).
2. Monitor heartbeat token cost via `/usage full` on a heartbeat response.

**Risk**: Low. Slightly more frequent heartbeat = slightly more baseline token usage, but the cache savings on real interactions outweigh this.

**Expected impact**: Maintains the Phase 1 cache savings across idle periods instead of losing them after TTL expiry.

---

## Phase 3: Context Pruning

**What**: Enable `cache-ttl` context pruning so old tool results and conversation history get pruned after the cache window expires.

**Why**: Long sessions accumulate tool results, file reads, and old conversation turns that bloat the context. Without pruning, post-idle requests re-cache the entire oversized history. Cache-TTL pruning trims stale context so re-caching after idle is smaller and cheaper.

**Config change** (`~/.openclaw/openclaw.json`):
```json
{
  "agents": {
    "defaults": {
      "contextPruning": {
        "mode": "cache-ttl",
        "ttl": "1h"
      }
    }
  }
}
```

**Rationale**:
- `cache-ttl` mode: prunes old tool-result context after the cache TTL expires
- `ttl: "1h"`: matches Anthropic's `long` cache retention window
- After 1h of no interaction, old tool results and conversation history are pruned, so the next request re-caches a smaller context

**Verification**:
1. Use `/context list` or `/context detail` to check context size before and after pruning.
2. After a >1h idle gap, verify the context window is smaller than before the gap.
3. Ensure no critical context is lost — compaction summaries should preserve key information.

**Risk**: Low-medium. Pruning removes old tool results, which means the model can't reference exact earlier tool outputs after pruning. Compaction summaries mitigate this. Test by asking about earlier conversation after a pruning event.

**Expected impact**: 20-30% reduction in context size for long sessions, which reduces both input token cost and improves response quality (less noise in context).

---

## Phase 4: Cheaper Models for Subagents

**What**: Route subagent tasks to cheaper models based on task complexity, with quality verification.

**Why**: Currently ALL subagents inherit the session model (Opus 4.6 or whatever the session is on). Most subagent tasks (council advisors, research queries, simple generation) don't need frontier-model quality. ZAI GLM-4.7 is free and handles many tasks well. Copilot Sonnet/Haiku are much cheaper quota-wise than Opus.

### Model Tier Strategy

| Tier | Model | Use Case | Cost |
|------|-------|----------|------|
| **Free** | `zai/glm-4.7` | Bulk subagent work: council advisors, brainstorming, summarization, classification | $0 |
| **Free-fast** | `zai/glm-4.7-flash` | Simple/short subagent tasks: acknowledgments, formatting, quick lookups | $0 |
| **Cheap** | `litellm/copilot-claude-haiku-4.5` | Tasks needing Claude quality but not heavy reasoning | Low quota |
| **Standard** | `litellm/copilot-claude-sonnet-4.6` | Tasks needing strong reasoning, code generation, analysis | Medium quota |
| **Frontier** | `litellm/copilot-claude-opus-4.6` | Only for: main session, referee/meta-arbiter, critical decisions | High quota |

### Implementation

#### 4a. Council Skill — Default to GLM-4.7

Update council skill to use cheaper models by default:

| Council Role | Default Model | Override for `tier=heavy` |
|-------------|---------------|--------------------------|
| Personality advisors | `zai/glm-4.7` | `litellm/copilot-claude-sonnet-4.6` |
| D/P Freethinkers | `zai/glm-4.7` | `litellm/copilot-claude-sonnet-4.6` |
| D/P Arbiters | `zai/glm-4.7` | `litellm/copilot-claude-sonnet-4.6` |
| Referee / Meta-Arbiter | `litellm/copilot-claude-sonnet-4.6` | `litellm/copilot-claude-opus-4.6` |

When spawning subagents via `sessions_spawn`, pass the `model` parameter:
```json
{
  "task": "...",
  "mode": "run",
  "label": "council-pragmatist",
  "model": "zai/glm-4.7"
}
```

#### 4b. General Subagent Routing Guidelines

Encode these in AGENTS.md or a workspace convention file so all future subagent spawns follow the pattern:

**Use `zai/glm-4.7` (free) when**:
- Task is well-defined with clear constraints
- Output format is specified in the prompt
- Task is one of: summarization, brainstorming, classification, translation, formatting, simple Q&A
- Task doesn't require tool use or complex multi-step reasoning

**Use `litellm/copilot-claude-sonnet-4.6` (standard) when**:
- Task requires nuanced reasoning or analysis
- Task involves code generation or review
- Output quality is user-facing and high-stakes
- Task requires understanding subtle context

**Use `litellm/copilot-claude-opus-4.6` (frontier) when**:
- Main interactive session only
- Final synthesis / referee / meta-arbiter roles
- Tasks where the user explicitly asked for highest quality

### Subagent Model Notes

- When spawning subagents for Claude tasks, caching applies if the subagent model is also a Claude model. But subagents are typically short-lived (single-turn `mode=run`), so caching benefit is minimal — they don't accumulate conversation history.
- The main caching win is in the **main session**, which has a large, growing context across many turns.
- For GLM-4.7 subagents: no caching benefit, but no cost either ($0 model). Prompts must be self-contained and tightly framed.
- For GPT subagents: OpenAI caches automatically if prompt ≥1024 tokens, no action needed.

### Prompt Structure for Maximum Cache Efficiency (Claude models)

Per official Anthropic best practices:
- **Static first, dynamic last**: System prompt, role, instructions → then topic/task (dynamic part).
- This structure is what OpenClaw already does (system prompt built once, user message varies).
- OpenClaw handles the `cache_control` injection automatically via `cacheRetention` config.
- Our prompts already follow this structure correctly.

Before switching council and subagents to GLM-4.7, run a quality comparison:

1. **Same-topic test**: Run the personality council on a topic we've already tested with Sonnet, but using GLM-4.7 for advisors. Compare output quality side by side.
2. **Structured output test**: Verify GLM-4.7 follows prompt templates correctly (word count guidance, section headers, role staying).
3. **Scoring rubric**:
   - Does the advisor stay in character? (yes/no)
   - Is the output substantive (not generic platitudes)? (1-5)
   - Does it follow word count guidance? (within 50% of target)
   - Does it reference specific aspects of the topic? (1-5)
4. **Minimum quality bar**: If GLM-4.7 scores ≥3.5/5 average on the rubric, it's good enough for advisor roles. Referee always stays on Sonnet+.

#### 4d. Prompt Engineering Per Model — Official Best Practices

From official Anthropic and OpenAI docs (see `memory/references/`):

**For Claude models (Haiku, Sonnet — subagent advisors)**
- Give an explicit role in the system prompt: `You are the Skeptic advisor on a council...`
- Use XML tags to separate role, instructions, context, topic: `<instructions>`, `<context>`, `<topic>`
- Put static instructions first, variable topic at the end (maximizes cache hit rate on repeated spawns)
- 3-5 `<example>` tags for structured output formats
- Use "tell what to do" not "don't do X": `Write in flowing prose` not `Don't use bullet points`

**For GLM-4.7 (free tier subagents)**
- Be MORE explicit than with Claude — GLM needs tighter constraints
- Constrain output length tightly: "Respond in exactly 3 paragraphs" not "200-400 words"
- Use numbered lists or explicit section headers in the prompt
- Front-load the most critical instruction (role + constraint first, context second)
- Include a format-check reminder: "Before responding, verify your output matches the format above"
- Request structured output over open-ended generation when possible
- Avoid complex multi-step reasoning chains — GLM handles simpler, well-defined tasks best

**For GPT models (gpt-5-mini, gpt-4.1 subagents)**
- Include explicit step-by-step instructions (GPT benefits from "think step by step" guidance)
- Use `response_format: json_schema` for any scored/structured output — eliminates format retries entirely
- Use `developer` role for system/role instructions (higher priority than `user`)
- Don't over-specify for reasoning models (o3, o4-mini) — they reason internally
- Pin to specific model snapshots if quality consistency matters (`gpt-4.1-2025-04-14`)

---

## Implementation Order

### Step 1: Config changes (Phases 1-3) — Do together, single commit

Apply all three config changes to `~/.openclaw/openclaw.json`:
- `cacheRetention: "long"` on Claude models
- `heartbeat.every: "25m"` for main, `"55m"` default
- `contextPruning.mode: "cache-ttl"` with `ttl: "1h"`

Restart gateway: `openclaw gateway restart`

Verify with `/status` and `/usage full` over next few interactions.

### Step 2: Quality test GLM-4.7 for subagent work

Run a single council advisor (e.g., Pragmatist) on a known topic using `model: "zai/glm-4.7"` in `sessions_spawn`. Compare output quality against the Sonnet run we already have saved.

### Step 3: Update council skill for model tiers

If GLM-4.7 passes quality bar, update `skills/council/SKILL.md` and `scripts/council.sh` with the model tier routing table. Update `references/prompts.md` with tighter prompt variants for cheaper models if needed.

### Step 4: Update AGENTS.md with subagent routing guidelines

Add a section documenting when to use which model tier for subagents, so the convention is followed consistently.

### Step 5: Monitor and tune

- Track cache hit rates over 1-2 days
- Monitor if context pruning causes any information loss
- Adjust heartbeat timing if cache misses are too frequent
- Tune GLM-4.7 prompts based on observed output quality

---

## Upstream Safety Rules

These are hard constraints. Any implementation that violates them is out of scope.

### ❌ Never do
- Edit files under `~/.npm-global/lib/node_modules/openclaw/` directly (dist, src, docs)
- Patch or monkey-patch OpenClaw's runtime code, even for emergencies (exception: the existing TUI patch has a tracked upstream PR — document any new ones immediately)
- Add config keys not documented in OpenClaw's own docs (guessing at undocumented keys can silently break on upgrade)
- Modify `~/.openclaw/openclaw.json` in a way that would be overwritten or invalidated by `openclaw update`
- Introduce any middleware, proxy, or hook that intercepts OpenClaw's internal request path

### ✅ Safe to do
- Edit `~/.openclaw/openclaw.json` using documented config knobs (agents, models, diagnostics, contextPruning, etc.)
- Add/edit workspace files (`~/.openclaw/workspace/`) freely — these are never touched by OpenClaw updates
- Install/update skills via `clawhub` — skills are workspace-local
- Run `openclaw gateway restart` after config changes
- Use `openclaw update status` / `scripts/openclaw-update-safe.sh` to check for upstream updates

### Checking before applying
Before implementing any config change:
1. Verify the key exists in `/home/openclaw/.npm-global/lib/node_modules/openclaw/docs/` or `https://docs.openclaw.ai`
2. If undocumented: skip it or open a question/issue — don't guess
3. After `openclaw update`, re-verify config keys still work (check gateway logs for config parse errors)

### Update workflow
```bash
# Before updating OpenClaw
openclaw update status           # check what version is available
# Review changelog for breaking config changes
openclaw update                  # update (safe scripts handle local compat)
openclaw gateway restart         # restart to pick up new version
# Verify gateway health + session model still resolves correctly
```

## What This Does NOT Change

- **No OpenClaw code changes**: Everything is config-only in `openclaw.json`
- **No upstream divergence**: All settings use documented OpenClaw config knobs
- **No new infrastructure**: No proxy servers, routers, or middleware
- **Main session stays on Opus**: Only subagents move to cheaper models
- **Fully reversible**: Remove the config keys to revert to current behavior

---

## Expected Combined Impact

| Optimization | Estimated Savings | Confidence |
|-------------|-------------------|------------|
| Prompt caching | 40-60% input token reduction | High |
| Cache warming via heartbeat | Maintains cache savings across idle | High |
| Context pruning | 20-30% context size reduction for long sessions | Medium |
| Subagent model routing | 60-80% subagent cost (free model for bulk work) | Medium (pending quality test) |

**Combined**: Significant reduction in Copilot quota burn. Main session quality unchanged. Subagent quality maintained through tighter prompts + quality verification.
