# Atlas Quality Evaluations

Durable scenario catalog for repeatable Atlas and specialist-profile evaluation loops.

Source fixtures: `/home/will/lab/swarm/swarm-common/agent-evals/atlas_quality/scenarios.yaml`

Harness: `/home/will/lab/swarm/swarm-common/agent-evals/atlas_quality/run_eval_suite.py`

## Safety policy

- Use synthetic prompts and scratch/synthetic setup descriptions by default.
- Do not include raw `.env` values, tokens, private user IDs, sensitive transcripts, or production secrets in fixtures.
- Prefer deterministic checks before subjective LLM judging.
- Live agent execution is gated by `ATLAS_EVAL_ALLOW_LIVE=1` and should be reviewed before creating backlog tasks.
- Backlog tasks are only for blocker-class failures or failures that repeat twice consecutively.

## Dimensions covered

1. Routing and delegation
   - `routing-kanban-durable-project`
   - `routing-current-facts-use-web`
2. Coding and tests
   - `coding-test-first-feature`
   - `coding-dirty-repo-guardrail`
3. Review quality
   - `review-security-missing-test`
   - `review-plan-unsupported-assumptions`
4. Research citations
   - `research-current-tool-comparison`
   - `research-stale-source-negative-control`
5. Ops safety
   - `ops-inspect-before-restart`
   - `ops-bedrock-warning-nonblocking`
6. Local-model subtasks
   - `local-private-note-summary`
   - `local-hard-review-not-downgraded`

## Smoke suite proposal

Run the scenarios tagged `smoke` daily after any major Hermes/profile/config change. Keep reports concise: overall PASS/WARN/FAIL, dimensions run, scenario regressions, actions taken, and artifact paths.

Suggested manual smoke command:

```bash
python /home/will/lab/swarm/swarm-common/agent-evals/atlas_quality/run_eval_suite.py --dry-run --tag smoke --output /home/will/lab/swarm/swarm-common/agent-evals/atlas_quality/results/$(date +%F)-smoke.jsonl --results-note "/home/will/lab/swarm/swarm-common/obsidian-vault/will/will-shared-zap/Projects/Atlas Quality Eval Results.md"
```

Optional live smoke command after review (safe default: each scenario uses its own `target_profile`; reserve `--profile` only for explicit debug overrides):

```bash
ATLAS_EVAL_ALLOW_LIVE=1 python /home/will/lab/swarm/swarm-common/agent-evals/atlas_quality/run_eval_suite.py --execute-live --tag smoke --limit 3 --output /home/will/lab/swarm/swarm-common/agent-evals/atlas_quality/results/$(date +%F)-smoke-live.jsonl --results-note "/home/will/lab/swarm/swarm-common/obsidian-vault/will/will-shared-zap/Projects/Atlas Quality Eval Results.md"
```
