# Phase 1 PR #3 Checklist: Memory Category Structure

Created: 2026-02-12
Owner: Flynn core
Status: ready to implement

## Goal

Introduce structured memory categories (facts/preferences/decisions/projects) to organize persistent memory storage while maintaining full backward compatibility with the existing flat namespace system. Add category-aware retrieval helpers and minimal integration points without changing user-facing commands yet.

Categories:
- **facts**: Verified facts about user, environment, tools, or systems
- **preferences**: User preferences, workflow patterns, communication style
- **decisions**: Important decisions made, their rationale, and outcomes
- **projects**: Project-specific context, goals, and progress notes

## PR Boundary

In scope:
- Category metadata structure and validation
- Backward-compatible category namespacing (e.g., `user/facts`, `user/preferences`)
- Category-aware read/write/search helpers in MemoryStore
- Retrieval helpers for category filtering
- Tests for category operations with backward compatibility validation
- Wire minimal integration points for auto-extraction (no command changes)

Out of scope:
- User-facing command changes (`/memory` commands)
- Auto-categorization of existing memory content
- Category-specific extraction prompts in AgentOrchestrator
- Category-based UI in web dashboard
- Category migration tools for existing deployments
- Category-specific embedding strategies

## File-by-File Implementation Plan

### 1. `src/memory/categories.ts` (new)

**Purpose:** Define category metadata, validation, and namespace mapping.

**Implementation:**
```typescript
/**
 * Supported memory categories for structured organization.
 */
export const MEMORY_CATEGORIES = {
  facts: {
    name: 'facts',
    description: 'Verified facts about user, environment, tools, or systems',
    namespace: 'facts',
  },
  preferences: {
    name: 'preferences',
    description: 'User preferences, workflow patterns, communication style',
    namespace: 'preferences',
  },
  decisions: {
    name: 'decisions',
    description: 'Important decisions made, their rationale, and outcomes',
    namespace: 'decisions',
  },
  projects: {
    name: 'projects',
    description: 'Project-specific context, goals, and progress notes',
    namespace: 'projects',
  },
} as const;

export type MemoryCategory = keyof typeof MEMORY_CATEGORIES;

export const VALID_CATEGORIES = Object.keys(MEMORY_CATEGORIES) as MemoryCategory[];

/**
 * Validate if a string is a valid memory category.
 */
export function isValidCategory(category: string): category is MemoryCategory {
  return VALID_CATEGORIES.includes(category as MemoryCategory);
}

/**
 * Map a category to its namespace path within a base namespace.
 * Examples:
 *   - buildCategoryNamespace('user', 'facts') → 'user/facts'
 *   - buildCategoryNamespace('sessions/abc', 'preferences') → 'sessions/abc/preferences'
 */
export function buildCategoryNamespace(
  baseNamespace: string,
  category: MemoryCategory
): string {
  return `${baseNamespace}/${MEMORY_CATEGORIES[category].namespace}`;
}

/**
 * Parse a namespace path to extract base and category if present.
 * Examples:
 *   - 'user/facts' → { base: 'user', category: 'facts' }
 *   - 'user' → { base: 'user', category: undefined }
 *   - 'sessions/abc/decisions' → { base: 'sessions/abc', category: 'decisions' }
 */
export function parseNamespace(namespace: string): {
  base: string;
  category?: MemoryCategory;
} {
  const parts = namespace.split('/');
  const lastPart = parts[parts.length - 1];
  
  if (isValidCategory(lastPart)) {
    const base = parts.slice(0, -1).join('/');
    return { base, category: lastPart };
  }
  
  return { base: namespace, category: undefined };
}

/**
 * List all category namespaces under a base namespace.
 * Example: 'user' → ['user/facts', 'user/preferences', 'user/decisions', 'user/projects']
 */
export function listCategoryNamespaces(baseNamespace: string): string[] {
  return VALID_CATEGORIES.map((cat) => buildCategoryNamespace(baseNamespace, cat));
}
```

**Tests:**
- Validate category constants and types
- Test `isValidCategory()` with valid/invalid inputs
- Test `buildCategoryNamespace()` with various base paths
- Test `parseNamespace()` with flat and nested paths
- Test `listCategoryNamespaces()` returns all categories

**Estimated effort:** 1 hour

---

### 2. `src/memory/store.ts` (modify)

**Purpose:** Add category-aware methods while preserving existing flat namespace behavior.

**Changes:**
1. Add imports:
```typescript
import type { MemoryCategory } from './categories.js';
import { isValidCategory, buildCategoryNamespace, parseNamespace } from './categories.js';
```

2. Add new methods (after existing public API, before private helpers):
```typescript
/**
 * Read content from a specific category within a base namespace.
 * Backward-compatible: if category file doesn't exist, returns empty string.
 * 
 * @param baseNamespace - Base namespace (e.g., 'user', 'sessions/abc123')
 * @param category - Category to read from
 */
readCategory(baseNamespace: string, category: MemoryCategory): string {
  const namespace = buildCategoryNamespace(baseNamespace, category);
  return this.read(namespace);
}

/**
 * Write content to a specific category within a base namespace.
 * 
 * @param baseNamespace - Base namespace (e.g., 'user', 'sessions/abc123')
 * @param category - Category to write to
 * @param content - Content to write
 * @param mode - 'append' or 'replace'
 */
writeCategory(
  baseNamespace: string,
  category: MemoryCategory,
  content: string,
  mode: 'append' | 'replace'
): void {
  const namespace = buildCategoryNamespace(baseNamespace, category);
  this.write(namespace, content, mode);
}

/**
 * List all categories present under a base namespace.
 * Returns only categories that have existing files.
 * 
 * @param baseNamespace - Base namespace to check
 */
listCategories(baseNamespace: string): MemoryCategory[] {
  const allNamespaces = this.listNamespaces();
  const categories: MemoryCategory[] = [];
  
  for (const ns of allNamespaces) {
    const parsed = parseNamespace(ns);
    if (parsed.base === baseNamespace && parsed.category) {
      categories.push(parsed.category);
    }
  }
  
  return categories;
}

/**
 * Read all category content under a base namespace.
 * Returns a map of category → content for all existing category files.
 * 
 * @param baseNamespace - Base namespace to read from
 */
readAllCategories(baseNamespace: string): Map<MemoryCategory, string> {
  const result = new Map<MemoryCategory, string>();
  const categories = this.listCategories(baseNamespace);
  
  for (const category of categories) {
    const content = this.readCategory(baseNamespace, category);
    if (content.length > 0) {
      result.set(category, content);
    }
  }
  
  return result;
}

/**
 * Search with optional category filtering.
 * If categories are specified, only searches within those category namespaces.
 * 
 * @param query - Search query
 * @param options - Optional filters
 */
searchWithCategories(
  query: string,
  options?: {
    baseNamespace?: string;
    categories?: MemoryCategory[];
  }
): SearchResult[] {
  const allResults = this.search(query);
  
  if (!options?.baseNamespace && !options?.categories) {
    return allResults;
  }
  
  return allResults.filter((result) => {
    const parsed = parseNamespace(result.namespace);
    
    // Filter by base namespace if specified
    if (options.baseNamespace && !result.namespace.startsWith(options.baseNamespace)) {
      return false;
    }
    
    // Filter by categories if specified
    if (options.categories && parsed.category) {
      return options.categories.includes(parsed.category);
    }
    
    // If categories filter is specified but this namespace has no category, exclude it
    if (options.categories && !parsed.category) {
      return false;
    }
    
    return true;
  });
}
```

3. Update `getContextForPrompt()` to include categories:
```typescript
/**
 * Build memory context suitable for injection into a system prompt.
 *
 * Reads both flat `user.md`/`global.md` (for backward compat) and 
 * structured categories under 'user' namespace.
 * 
 * Truncates to stay within {@link MemoryStoreConfig.maxContextTokens}.
 */
getContextForPrompt(): string {
  const sections: string[] = [];
  
  // Read legacy flat files for backward compatibility
  const userMemory = this.read('user');
  const globalMemory = this.read('global');
  
  if (userMemory.length > 0) {
    sections.push(`## User Memory\n\n${userMemory}`);
  }
  if (globalMemory.length > 0) {
    sections.push(`## Global Memory\n\n${globalMemory}`);
  }
  
  // Read structured categories under 'user' namespace
  const categories = this.readAllCategories('user');
  for (const [category, content] of categories) {
    if (content.length > 0) {
      const categoryLabel = category.charAt(0).toUpperCase() + category.slice(1);
      sections.push(`## User ${categoryLabel}\n\n${content}`);
    }
  }
  
  // Nothing to inject
  if (sections.length === 0) {
    return '';
  }
  
  const full = sections.join('\n\n');
  
  // Truncate to fit within the token budget (estimate: 4 chars ≈ 1 token)
  const maxChars = this._config.maxContextTokens * 4;
  if (full.length <= maxChars) {
    return full;
  }
  return full.slice(0, maxChars);
}
```

**Tests:** (in `src/memory/store.test.ts`)
- Test `readCategory()` returns empty for non-existent categories
- Test `writeCategory()` creates proper namespace paths
- Test `listCategories()` returns only existing categories
- Test `readAllCategories()` returns map of all category content
- Test `searchWithCategories()` with no filters (returns all)
- Test `searchWithCategories()` with base namespace filter
- Test `searchWithCategories()` with categories filter
- Test `searchWithCategories()` with both filters
- Test `getContextForPrompt()` includes both legacy and category content
- Test backward compatibility: flat `user.md` still works alongside categories

**Estimated effort:** 2-3 hours

---

### 3. `src/memory/categories.test.ts` (new)

**Purpose:** Comprehensive tests for category utilities.

**Coverage:**
- Category validation
- Namespace building and parsing
- Edge cases (empty strings, special characters, nested paths)
- Backward compatibility with flat namespaces

**Estimated effort:** 1 hour

---

### 4. `src/memory/index.ts` (modify)

**Purpose:** Export category types and utilities.

**Changes:**
```typescript
export { MemoryStore } from './store.js';
export type { MemoryStoreConfig, SearchResult } from './store.js';
export { chunkText } from './chunker.js';
export type { Chunk, ChunkOptions } from './chunker.js';
export { createEmbeddingProvider, OpenAIEmbeddingProvider, GeminiEmbeddingProvider, OllamaEmbeddingProvider, LlamaCppEmbeddingProvider } from './embeddings.js';
export type { EmbeddingProvider } from './embeddings.js';
export { VectorStore, cosineSimilarity, contentHash } from './vector-store.js';
export type { VectorSearchResult, EmbeddingRow } from './vector-store.js';
export { HybridSearch } from './hybrid-search.js';
export type { HybridSearchResult } from './hybrid-search.js';
export { MEMORY_CATEGORIES, VALID_CATEGORIES, isValidCategory, buildCategoryNamespace, parseNamespace, listCategoryNamespaces } from './categories.js';
export type { MemoryCategory } from './categories.js';
```

**Estimated effort:** 5 minutes

---

### 5. `src/tools/builtin/memory-read.ts` (modify)

**Purpose:** Update tool description to mention categories (no functional change yet).

**Changes:**
Update description to document category support:
```typescript
description:
  'Read a persistent memory file by namespace. Available namespaces include "user" (user preferences and facts), "global" (cross-session knowledge), and session-specific namespaces. ' +
  'Supports structured categories: append "/facts", "/preferences", "/decisions", or "/projects" to a base namespace (e.g., "user/facts", "sessions/abc123/preferences"). ' +
  'Returns the full contents of the memory file.',
```

**Estimated effort:** 5 minutes

---

### 6. `src/tools/builtin/memory-write.ts` (modify)

**Purpose:** Update tool description to mention categories (no functional change yet).

**Changes:**
Update description:
```typescript
description:
  'Write to a persistent memory file. Use mode="append" to add new information without overwriting existing content, or mode="replace" to overwrite the entire namespace. ' +
  'Supports structured categories: use namespaces like "user/facts", "user/preferences", "user/decisions", or "user/projects" for organized storage.',
```

**Estimated effort:** 5 minutes

---

### 7. `src/tools/builtin/memory-search.ts` (modify - optional for this PR)

**Purpose:** Document category filtering support in description (implementation deferred to follow-up).

**Changes:**
Update description:
```typescript
description:
  'Search across all memory files for a keyword or phrase. Returns matching lines with surrounding context from every namespace.' +
  (hybridSearch ? ' Uses semantic vector search combined with keyword matching for better results.' : '') +
  ' Future: will support category filtering (e.g., search only within facts or preferences).',
```

**Estimated effort:** 2 minutes

---

### 8. Integration wiring (no user-facing changes)

**Files to review (no changes required for this PR):**
- `src/backends/native/orchestrator.ts` — memory extraction hook point (already uses `memoryStore.write()`, categories will work automatically)
- `src/daemon/memory.ts` — memory initialization (no changes needed, categories are opt-in)

**Validation:** Existing memory tools continue to work with flat namespaces; category namespaces are treated as regular namespaces by existing code.

**Estimated effort:** 30 minutes review

---

## Implementation Steps

**Phase 1: Foundation (3-4 hours)**
1. Create `src/memory/categories.ts` with types, constants, and utilities
2. Create `src/memory/categories.test.ts` with comprehensive tests
3. Run: `pnpm typecheck && pnpm test:run src/memory/categories.test.ts`

**Phase 2: Store Integration (3-4 hours)**
4. Add category-aware methods to `src/memory/store.ts`
5. Update `getContextForPrompt()` to include category content
6. Add tests to `src/memory/store.test.ts` for new methods and backward compat
7. Run: `pnpm typecheck && pnpm test:run src/memory/store.test.ts`

**Phase 3: Exports and Documentation (1 hour)**
8. Update `src/memory/index.ts` with new exports
9. Update tool descriptions in `memory-read.ts`, `memory-write.ts`, `memory-search.ts`
10. Run: `pnpm typecheck && pnpm test:run`

**Phase 4: Validation (30 minutes)**
11. Build and lint: `pnpm build && pnpm lint`
12. Full test suite: `pnpm test:run`
13. Manual smoke test: verify flat `user.md` and `user/facts.md` both work

---

## Validation Commands

Run in order:

```bash
# Type checking
pnpm typecheck

# Unit tests for categories
pnpm test:run src/memory/categories.test.ts

# Unit tests for store
pnpm test:run src/memory/store.test.ts

# Full test suite
pnpm test:run

# Linting
pnpm lint

# Build
pnpm build
```

---

## Acceptance Criteria

### Functional
- [ ] Category constants defined with proper TypeScript types
- [ ] `buildCategoryNamespace()` creates proper paths (e.g., `user/facts`)
- [ ] `parseNamespace()` correctly extracts base and category
- [ ] `MemoryStore.readCategory()` reads category-specific content
- [ ] `MemoryStore.writeCategory()` writes to category namespaces
- [ ] `MemoryStore.listCategories()` returns only existing categories
- [ ] `MemoryStore.readAllCategories()` returns map of all category content
- [ ] `searchWithCategories()` filters by base namespace and/or categories
- [ ] `getContextForPrompt()` includes both legacy flat files and categories

### Backward Compatibility
- [ ] Existing flat `user.md` and `global.md` continue to work
- [ ] Existing memory tools work unchanged with flat namespaces
- [ ] Existing tests pass without modification
- [ ] Category namespaces are opt-in, not required
- [ ] Legacy namespace patterns (e.g., `sessions/abc123`) unaffected

### Quality
- [ ] All new functions have JSDoc comments
- [ ] Test coverage ≥90% for new code
- [ ] No type errors
- [ ] No lint warnings
- [ ] Build succeeds

---

## Risk Assessment and Mitigations

### Risk 1: Breaking existing memory access patterns
**Likelihood:** Low  
**Impact:** High  
**Mitigation:**
- Keep all existing MemoryStore methods unchanged
- Add category methods as new API surface
- Test backward compatibility explicitly
- Ensure `getContextForPrompt()` includes both legacy and category content

### Risk 2: Namespace collision between flat and category paths
**Likelihood:** Low  
**Impact:** Medium  
**Example:** User creates `user/facts` manually before categories exist  
**Mitigation:**
- Document category namespaces in tool descriptions
- `parseNamespace()` handles both flat and category paths gracefully
- No automatic migration means existing files remain untouched

### Risk 3: Token budget exceeded due to category content injection
**Likelihood:** Medium  
**Impact:** Low  
**Mitigation:**
- Existing `maxContextTokens` truncation logic handles all content
- Categories are additive, not multiplicative (replaces flat structure over time)
- Follow-up PR can add per-category token allocation if needed

### Risk 4: Test regressions in memory-related tests
**Likelihood:** Low  
**Impact:** Medium  
**Mitigation:**
- Run full test suite before and after changes
- Add explicit backward compatibility test cases
- Use test fixtures that mix flat and category namespaces

---

## Quality Gates

### Code Quality
- [ ] All functions have TypeScript types (no `any`)
- [ ] All public APIs have JSDoc comments
- [ ] Error handling for invalid category names
- [ ] Consistent naming conventions (camelCase functions, PascalCase types)

### Testing
- [ ] Unit tests for all category utilities
- [ ] Integration tests for category read/write/search
- [ ] Backward compatibility tests with legacy namespaces
- [ ] Edge case tests (empty content, missing files, invalid categories)

### Performance
- [ ] No measurable performance regression in `MemoryStore.search()`
- [ ] `listCategories()` scans namespace list once (no redundant I/O)
- [ ] `getContextForPrompt()` maintains O(n) complexity

### Documentation
- [ ] Tool descriptions updated to mention categories
- [ ] JSDoc comments explain category namespace format
- [ ] Examples in JSDoc show category usage patterns

---

## Follow-up PRs (Out of Scope)

1. **Category-specific memory tools:**
   - Add `memory.read_category` and `memory.write_category` tools
   - Add category filtering to `memory.search` tool
   - Update TUI commands to support category selection

2. **Auto-categorization in AgentOrchestrator:**
   - Update memory extraction prompts to suggest categories
   - Add heuristics for auto-routing content to appropriate categories

3. **Category migration tool:**
   - CLI command to analyze existing flat memory files
   - Suggest categorization of existing content
   - Batch migration with user confirmation

4. **Web dashboard category UI:**
   - Category tabs in memory viewer
   - Per-category search filters
   - Category statistics and visualization

5. **Category-specific embedding strategies:**
   - Different embedding models per category
   - Category-weighted hybrid search scoring
   - Cross-category semantic links

---

## Suggested Commit Message

```
feat(memory): add structured memory categories with backward compatibility

Introduces four memory categories (facts/preferences/decisions/projects)
for organized persistent storage:

- Add category types, constants, and namespace utilities
- Add category-aware read/write/search methods to MemoryStore
- Update getContextForPrompt() to include category content
- Maintain full backward compatibility with flat namespace files
- Update tool descriptions to document category support

Categories use namespace paths (e.g., 'user/facts', 'sessions/abc/preferences')
and are opt-in. Existing flat files ('user.md', 'global.md') continue to work
unchanged.

Tests: 15+ new tests for category operations and backward compatibility
```

---

## PR Description Template

```markdown
## Summary

Adds structured memory categories (facts/preferences/decisions/projects) to organize persistent memory while maintaining full backward compatibility with existing flat namespace files.

## Changes

- **New:** `src/memory/categories.ts` — category types, utilities, namespace mapping
- **Modified:** `src/memory/store.ts` — category-aware read/write/search methods
- **Modified:** `src/memory/index.ts` — export category types
- **Modified:** Tool descriptions updated to document category support
- **Tests:** 15+ new tests for category operations and backward compatibility

## Category Structure

Categories are implemented as namespace paths:
- `user/facts` — verified facts about user, environment, tools
- `user/preferences` — user preferences, workflow patterns, style
- `user/decisions` — important decisions, rationale, outcomes
- `user/projects` — project-specific context and progress

Flat files (`user.md`, `global.md`) continue to work unchanged.

## Testing

All existing tests pass. New tests cover:
- Category namespace building and parsing
- Category-specific read/write operations
- Category filtering in search
- Backward compatibility with flat files
- Mixed flat + category content in prompts

## Backward Compatibility

✅ Existing `user.md` and `global.md` files work unchanged  
✅ Existing memory tools work with flat namespaces  
✅ Categories are opt-in, not required  
✅ No breaking changes to public API

## Follow-up Work

- Category-specific memory tools (`memory.read_category`, etc.)
- Auto-categorization in memory extraction
- Category UI in web dashboard
- Migration tool for existing flat files
```

---

## Time Estimate

Total: **8-10 hours**

Breakdown:
- Categories foundation: 3-4 hours
- Store integration: 3-4 hours
- Exports and docs: 1 hour
- Validation and testing: 1-2 hours

---

## Success Metrics

- [ ] PR merged without breaking changes
- [ ] Test count increases by 15+
- [ ] All existing memory tests pass unchanged
- [ ] Documentation updated and clear
- [ ] No performance regression in memory operations
- [ ] state.json updated with phase completion
