will/flynn

Files

T

William Valentin 5a4d4b2e21 docs: archive stale plan checklists and clarify canonical tracking

2026-02-18 08:15:39 -08:00

34 KiB

Raw Permalink Blame History

Gmail Push Notifications Revisit — Implementation Plan

Archived (2026-02-18): Historical implementation checklist. Canonical status is tracked in docs/plans/state.json; unchecked boxes here are not active backlog unless explicitly re-opened.

Date: 2026-02-13
Status: Planned
Goal: Make Gmail notifications operationally reliable and well-documented

Overview

The Gmail watcher currently supports two modes:

Pub/Sub push notifications (Google sends HTTP POST to /gmail/push)
Polling fallback (History API queries every N seconds)

Current implementation has operational ambiguity around:

Deployment patterns (public webhook vs Pub/Sub pull vs hybrid)
GCP resource setup (topic/subscription/IAM relationships)
Network reachability constraints (Tailscale-only gateways)
Configuration surface (implicit vs explicit controls)
Error diagnostics (vague failure messages)

This plan addresses these gaps without changing core functionality.

Problem Analysis

Current Behavior

Push setup flow (src/automation/gmail.ts:222-253):

resolvePubSubTopicName() reads pubsub_topic config or FLYNN_GMAIL_PUBSUB_TOPIC env var
Calls gmail.users.watch() with the topic
If watch fails → logs warning, continues with polling only
If watch succeeds → schedules renewal every 6 days

Issues:

Topic ownership mismatch: OAuth credentials' project_id may differ from the Pub/Sub topic's project
Network reachability: Google cannot reach Tailscale-only endpoints (server.tailscale_only: true)
Implicit disable: No explicit way to disable push without deleting pubsub_topic field
Poor error messages: "Invalid topicName" error includes no actionable guidance
Undocumented subscription: Users must manually create Pub/Sub subscription → push endpoint mapping
Doctor check gap: flynn doctor validates OAuth but not Pub/Sub setup

Deployment Patterns

Pattern	Reachability	Flynn Config	GCP Setup	Use Case
Public webhook	Public IP + DNS	`pubsub_topic` set	Topic + push subscription → `POST https://example.com/gmail/push`	Production deployment with public ingress
Tailscale Serve	Tailscale funnel	`pubsub_topic` set, `server.tailscale.funnel: true`	Topic + push subscription → `POST https://flynn.ts.net/gmail/push`	Private deployment with Tailscale funnel enabled
Pub/Sub pull	Any (no inbound)	`pubsub_subscription_id` set	Topic + pull subscription	Private deployment, Flynn polls subscription
Polling only	Any (no inbound)	`pubsub_topic` omitted	None	Development, no GCP setup
Hybrid (recommended)	Any	`pubsub_topic` + `pubsub_subscription_id`	Topic + push subscription + pull subscription	Push when reachable, pull fallback, History API tertiary

Recommended Changes

1. Config Schema Additions

File: src/config/schema.ts

Add new fields to gmailSchema:

const gmailSchema = z.object({
  enabled: z.boolean().default(false),
  credentials_file: z.string().optional(),
  token_file: z.string().default('~/.config/flynn/gmail-token.json'),
  
  // ── Push notifications ────────────────────────────────────────────────
  /**
   * Pub/Sub topic for Gmail push notifications (watch API).
   * Format: projects/<project-id>/topics/<topic> or just <topic> (auto-prefixed).
   * If omitted, push notifications are disabled (polling only).
   */
  pubsub_topic: z.string().optional(),
  
  /**
   * Pub/Sub subscription ID for pull-based message retrieval.
   * Format: projects/<project-id>/subscriptions/<subscription> or just <subscription>.
   * When set, Flynn periodically pulls messages from this subscription in addition
   * to waiting for push notifications at POST /gmail/push.
   * Enables hybrid mode: push when reachable, pull fallback.
   */
  pubsub_subscription_id: z.string().optional(),
  
  /**
   * Pull interval for Pub/Sub subscription polling (hybrid mode).
   * Only used when pubsub_subscription_id is set. Default: 60s.
   */
  pubsub_pull_interval: z.string().default('60s'),
  
  /**
   * Maximum messages to pull per request when using pull mode. Default: 10.
   */
  pubsub_max_messages: z.number().min(1).max(100).default(10),
  
  /**
   * Disable watch API even if pubsub_topic is set.
   * Useful for debugging or when push endpoint is temporarily unreachable.
   */
  disable_push: z.boolean().default(false),
  
  // ── Existing fields ────────────────────────────────────────────────────
  watch_labels: z.array(z.string()).default(['INBOX']),
  poll_interval: z.string().default('300s'),  // History API fallback
  history_start: z.string().optional(),
  output: z.object({
    channel: z.string().min(1),
    peer: z.string().min(1),
  }),
  message: z.string().default('New email from {{from}}: {{subject}}\n\n{{snippet}}'),
}).optional();

Defaults:

pubsub_topic: undefined (no push)
pubsub_subscription_id: undefined (no pull)
pubsub_pull_interval: '60s'
pubsub_max_messages: 10
disable_push: false
poll_interval: '300s' (History API fallback)

2. Implementation Changes

2.1 Add Pub/Sub Pull Support

File: src/automation/gmail.ts

Add new method setupPullSubscription():

private pullSubscriptionTimer?: ReturnType<typeof setInterval>;
private subscriptionPath?: string;

/**
 * Set up Pub/Sub pull subscription for periodic message retrieval.
 * Alternative/fallback to push notifications when endpoint is unreachable.
 */
private async setupPullSubscription(): Promise<void> {
  if (!this.config.pubsub_subscription_id) {
    return; // Pull mode not configured
  }

  const subId = this.config.pubsub_subscription_id.trim();
  
  // Allow shorthand: just the subscription id (e.g. "gmail-pull")
  if (!subId.includes('/')) {
    if (!this.googleProjectId) {
      console.warn(
        `GmailWatcher: pubsub_subscription_id '${subId}' must be fully qualified ` +
        `(projects/<project-id>/subscriptions/<sub>) because project_id was not found in credentials`
      );
      return;
    }
    this.subscriptionPath = `projects/${this.googleProjectId}/subscriptions/${subId}`;
  } else {
    const isValid = /^projects\/[^/]+\/subscriptions\/[^/]+$/.test(subId);
    if (!isValid) {
      console.warn(
        `GmailWatcher: Invalid pubsub_subscription_id '${subId}'. ` +
        `Expected: projects/<project-id>/subscriptions/<sub>`
      );
      return;
    }
    this.subscriptionPath = subId;
  }

  console.log(`GmailWatcher: Pull subscription registered (${this.subscriptionPath})`);

  // Start periodic pull
  const pullMs = parseInterval(this.config.pubsub_pull_interval ?? '60s');
  this.pullSubscriptionTimer = setInterval(() => {
    this.pullSubscriptionMessages().catch((err) => {
      console.error('GmailWatcher: Pull subscription error —', err instanceof Error ? err.message : err);
    });
  }, pullMs);
}

/**
 * Pull messages from the Pub/Sub subscription and process them.
 */
private async pullSubscriptionMessages(): Promise<void> {
  if (!this.subscriptionPath || !this.oauth2Client) {
    return;
  }

  try {
    const { PubSub } = await import('@google-cloud/pubsub');
    const pubsub = new PubSub({
      projectId: this.googleProjectId,
      authClient: this.oauth2Client,
    });

    const subscription = pubsub.subscription(this.subscriptionPath);
    const [messages] = await subscription.pull({
      maxMessages: this.config.pubsub_max_messages ?? 10,
    });

    for (const message of messages) {
      if (message.data) {
        await this.handlePushNotification(message.data.toString('base64'));
      }
      
      // Acknowledge the message
      await subscription.ack(message);
    }
  } catch (error) {
    console.error(
      'GmailWatcher: Failed to pull Pub/Sub messages —',
      error instanceof Error ? error.message : error
    );
  }
}

Update connect() to call setupPullSubscription():

async connect(): Promise<void> {
  this._status = 'connecting';

  try {
    this.oauth2Client = await this.authorize();
  } catch (error) {
    const errMsg = error instanceof Error ? error.message : 'Unknown error';
    console.error(`GmailWatcher: Authorization failed — ${errMsg}`);
    console.error('GmailWatcher: Run "flynn gmail-auth" to set up Gmail credentials.');
    this._status = 'error';
    return;
  }

  // Set up Gmail push watch (Pub/Sub) if not disabled
  if (!this.config.disable_push) {
    try {
      await this.setupWatch();
    } catch (error) {
      const errMsg = error instanceof Error ? error.message : 'Unknown error';
      const hint = this.buildWatchErrorHint(errMsg);
      console.warn(`GmailWatcher: Watch setup failed (push disabled) — ${errMsg}${hint}`);
    }
  } else {
    console.log('GmailWatcher: Push notifications disabled (disable_push: true)');
  }

  // Set up Pub/Sub pull subscription (hybrid mode)
  await this.setupPullSubscription();

  // Start History API polling fallback
  const pollMs = parseInterval(this.config.poll_interval ?? '300s');
  this.pollTimer = setInterval(() => {
    this.pollForNewMessages().catch((err) => {
      console.error('GmailWatcher: Poll error —', err instanceof Error ? err.message : err);
    });
  }, pollMs);

  this._status = 'connected';
  console.log(
    `GmailWatcher: Connected (` +
    `push=${!this.config.disable_push && !!this.config.pubsub_topic}, ` +
    `pull=${!!this.config.pubsub_subscription_id}, ` +
    `poll_interval=${this.config.poll_interval ?? '300s'})`
  );
  auditLogger?.systemStart('GmailWatcher', {
    push: !this.config.disable_push && !!this.config.pubsub_topic,
    pull: !!this.config.pubsub_subscription_id,
    poll_interval: this.config.poll_interval,
  });
}

Update disconnect() to clear pull timer:

async disconnect(): Promise<void> {
  if (this.pollTimer) {
    clearInterval(this.pollTimer);
    this.pollTimer = undefined;
  }
  if (this.watchTimer) {
    clearTimeout(this.watchTimer);
    this.watchTimer = undefined;
  }
  if (this.pullSubscriptionTimer) {
    clearInterval(this.pullSubscriptionTimer);
    this.pullSubscriptionTimer = undefined;
  }
  this.oauth2Client = undefined;
  this._status = 'disconnected';
  auditLogger?.systemStop('GmailWatcher');
}

2.2 Improve Error Messages

File: src/automation/gmail.ts

Add helper method buildWatchErrorHint():

/**
 * Build a helpful error hint based on the watch failure message.
 */
private buildWatchErrorHint(errMsg: string): string {
  if (errMsg.includes('Invalid topicName')) {
    const topicHint = this.googleProjectId
      ? `projects/${this.googleProjectId}/topics/gmail-push`
      : '<project-id>/topics/gmail-push';
    
    return `\n` +
      `  Tip: Set automation.gmail.pubsub_topic to "${topicHint}"\n` +
      `  Ensure the topic exists in GCP and the subscription is configured to POST to /gmail/push\n` +
      `  Or set automation.gmail.pubsub_subscription_id for pull-based mode (no webhook required)`;
  }
  
  if (errMsg.includes('Permission denied') || errMsg.includes('Forbidden')) {
    return `\n` +
      `  Tip: Grant Gmail API permission to publish to the Pub/Sub topic:\n` +
      `    gcloud pubsub topics add-iam-policy-binding <topic> \\\n` +
      `      --member=serviceAccount:gmail-api-push@system.gserviceaccount.com \\\n` +
      `      --role=roles/pubsub.publisher`;
  }
  
  if (errMsg.includes('Not found') || errMsg.includes('404')) {
    return `\n` +
      `  Tip: Create the Pub/Sub topic first:\n` +
      `    gcloud pubsub topics create gmail-push --project=<project-id>`;
  }
  
  return '';
}

Update setupWatch() error handling:

try {
  await this.setupWatch();
} catch (error) {
  const errMsg = error instanceof Error ? error.message : 'Unknown error';
  const hint = this.buildWatchErrorHint(errMsg);
  console.warn(`GmailWatcher: Watch setup failed (push disabled) — ${errMsg}${hint}`);
}

3. Doctor Checks

File: src/cli/doctor.ts

Enhance checkGmail to validate Pub/Sub setup:

const checkGmail: Check = async (ctx) => {
  if (!ctx.config) {
    return { status: 'skip', label: 'Gmail configured', detail: '(config invalid)' };
  }
  const gmail = ctx.config.automation.gmail;
  if (!gmail?.enabled) {
    return { status: 'skip', label: 'Gmail configured', detail: '(not enabled)' };
  }

  // 1. Check credentials file
  const credentialsPath = expandPath(gmail.credentials_file ?? '~/.config/flynn/gmail-credentials.json');
  if (!existsSync(credentialsPath)) {
    return { status: 'fail', label: 'Gmail configured', detail: `credentials file not found: ${credentialsPath}` };
  }

  // 2. Check token file
  const tokenPath = expandPath(gmail.token_file ?? '~/.config/flynn/gmail-token.json');
  if (!existsSync(tokenPath)) {
    return { status: 'warn', label: 'Gmail configured', detail: 'run `flynn gmail-auth` to authenticate' };
  }

  // 3. Parse credentials to get project_id
  let projectId: string | undefined;
  try {
    const creds = JSON.parse(readFileSync(credentialsPath, 'utf-8'));
    projectId = creds.installed?.project_id ?? creds.web?.project_id;
  } catch {
    // Ignore parse errors — already validated above
  }

  // 4. Check Pub/Sub configuration
  const modes: string[] = [];
  const warnings: string[] = [];

  if (gmail.pubsub_topic && !gmail.disable_push) {
    const topic = gmail.pubsub_topic.trim();
    if (!topic.includes('/') && !projectId) {
      warnings.push('pubsub_topic requires project_id in credentials');
    } else {
      modes.push('push');
      
      // Warn if tailscale_only is enabled (Google cannot reach tailnet-only endpoints)
      if (ctx.config.server?.tailscale_only && !ctx.config.server?.tailscale?.funnel) {
        warnings.push('push requires public endpoint or Tailscale funnel');
      }
    }
  }

  if (gmail.pubsub_subscription_id) {
    const sub = gmail.pubsub_subscription_id.trim();
    if (!sub.includes('/') && !projectId) {
      warnings.push('pubsub_subscription_id requires project_id in credentials');
    } else {
      modes.push('pull');
    }
  }

  if (modes.length === 0) {
    modes.push('poll');
  } else {
    modes.push('poll-fallback');
  }

  const modeStr = modes.join(' + ');
  const detail = warnings.length > 0
    ? `${modeStr} (⚠️  ${warnings.join(', ')})`
    : `${modeStr} → ${gmail.output.channel}/${gmail.output.peer}`;

  return {
    status: warnings.length > 0 ? 'warn' : 'pass',
    label: 'Gmail configured',
    detail,
  };
};

4. Documentation Updates

4.1 README.md

File: README.md

Update Gmail section (lines 461-512):

### Gmail Watcher

Monitor a Gmail inbox for new messages and trigger agent responses. Flynn supports three retrieval modes:

1. **Push notifications** (Pub/Sub watch) — Google sends HTTP POST to `/gmail/push`
2. **Pull subscription** (Pub/Sub pull) — Flynn periodically pulls messages from a subscription
3. **History API polling** (fallback) — Flynn queries the History API every N seconds

**Recommended deployment**: Hybrid mode (push + pull + polling) for maximum reliability.

#### Prerequisites

1. **Google Cloud Project** with Gmail API and Pub/Sub API enabled
2. **OAuth2 credentials** (Desktop application) — download JSON from Cloud Console
3. **Pub/Sub topic** (for push or pull):
   ```bash
   gcloud pubsub topics create gmail-push --project=your-project

Grant Gmail API publish permission:

gcloud pubsub topics add-iam-policy-binding projects/your-project/topics/gmail-push \
  --member=serviceAccount:gmail-api-push@system.gserviceaccount.com \
  --role=roles/pubsub.publisher

Authenticate:
```
flynn gmail-auth
```

Deployment Patterns

Mode	Config	GCP Setup	Network	Use Case
Push only	`pubsub_topic`	Topic + push subscription → `POST https://example.com/gmail/push`	Public IP or Tailscale funnel	Production with ingress
Pull only	`pubsub_subscription_id`	Topic + pull subscription	Any (no inbound)	Private deployment
Hybrid ⭐	Both	Topic + push + pull subscriptions	Any	Maximum reliability
Polling only	Neither	None	Any	Development/testing

Push Notifications Setup

When pubsub_topic is set, Flynn calls gmail.users.watch() to register for push notifications.

Create a push subscription in GCP:

gcloud pubsub subscriptions create gmail-push-sub \
  --topic=gmail-push \
  --push-endpoint=https://your-domain.com/gmail/push

If using Tailscale Serve, enable funnel:

server:
  tailscale:
    serve: true
    funnel: true  # ← Required for Google to reach your endpoint

Set pubsub_topic in config:

automation:
  gmail:
    enabled: true
    pubsub_topic: projects/your-project/topics/gmail-push

Important: Push notifications require a publicly reachable endpoint. If server.tailscale_only: true and funnel is disabled, push will fail (Flynn falls back to pull/poll).

Pull Subscription Setup

When pubsub_subscription_id is set, Flynn periodically pulls messages from the subscription.

Create a pull subscription in GCP:

gcloud pubsub subscriptions create gmail-pull-sub \
  --topic=gmail-push \
  --ack-deadline=60

Set pubsub_subscription_id in config:

automation:
  gmail:
    enabled: true
    pubsub_subscription_id: projects/your-project/topics/gmail-pull-sub
    pubsub_pull_interval: "60s"

Note: Pull mode works with any network setup (no inbound connections required).

Hybrid Mode (Recommended)

Combine push + pull for maximum reliability:

automation:
  gmail:
    enabled: true
    credentials_file: ~/.config/flynn/gmail-credentials.json
    token_file: ~/.config/flynn/gmail-token.json
    
    # Push notifications (when endpoint is reachable)
    pubsub_topic: projects/your-project/topics/gmail-push
    
    # Pull fallback (always works)
    pubsub_subscription_id: projects/your-project/subscriptions/gmail-pull-sub
    pubsub_pull_interval: "60s"
    
    # History API tertiary fallback
    poll_interval: "300s"
    
    watch_labels: [INBOX]
    message: "New email from {{from}}: {{subject}}\n\n{{snippet}}"
    output:
      channel: telegram
      peer: "123456789"

Configuration Reference

Field	Required	Default	Description
`enabled`	no	`false`	Enable the Gmail watcher
`credentials_file`	yes	—	Path to OAuth2 credentials JSON
`token_file`	no	`~/.config/flynn/gmail-token.json`	Path to stored refresh token
`pubsub_topic`	no	—	Pub/Sub topic for push notifications (format: `projects/<project>/topics/<topic>`)
`pubsub_subscription_id`	no	—	Pub/Sub subscription for pull mode (format: `projects/<project>/subscriptions/<sub>`)
`pubsub_pull_interval`	no	`60s`	Pull interval when using subscription
`pubsub_max_messages`	no	`10`	Max messages per pull request (1-100)
`disable_push`	no	`false`	Disable watch API even if `pubsub_topic` is set
`watch_labels`	no	`[INBOX]`	Gmail labels to watch
`poll_interval`	no	`300s`	History API fallback interval
`history_start`	no	—	ISO date — only process emails after this date
`message`	no	`New email from {{from}}: {{subject}}\n\n{{snippet}}`	Template for agent message
`output.channel`	yes	—	Channel to route responses (e.g. `telegram`)
`output.peer`	yes	—	Peer/chat ID on output channel

Template Variables

Variable	Description
`{{from}}`	Sender email address
`{{to}}`	Recipient email address
`{{subject}}`	Email subject line
`{{snippet}}`	Gmail-provided snippet (first ~200 chars)
`{{date}}`	Email date header
`{{id}}`	Gmail message ID
`{{labels}}`	Comma-separated label names

Troubleshooting

"Watch setup failed — Invalid topicName"

Ensure pubsub_topic format is correct: projects/<project>/topics/<topic>
Verify the topic exists: gcloud pubsub topics list
Check IAM permissions (Gmail API must be able to publish)

"Permission denied" or "Forbidden"

Grant publish permission to Gmail API:

gcloud pubsub topics add-iam-policy-binding <topic> \
  --member=serviceAccount:gmail-api-push@system.gserviceaccount.com \
  --role=roles/pubsub.publisher

Push notifications not arriving

Verify endpoint is reachable: curl -X POST https://your-domain.com/gmail/push
Check subscription configuration: gcloud pubsub subscriptions describe <sub>
Enable Tailscale funnel if using tailscale_only: true
Fall back to pull or polling mode

Check status:

flynn doctor  # Validates config and reports active modes (push/pull/poll)


#### 4.2 config/default.yaml

**File**: `config/default.yaml`

Update Gmail example (lines 128-136):

```yaml
#   gmail:
#     enabled: false
#     credentials_file: ~/.config/flynn/gmail-credentials.json
#     token_file: ~/.config/flynn/gmail-token.json
#     
#     # Push notifications (requires public endpoint or Tailscale funnel)
#     pubsub_topic: projects/<your-project>/topics/gmail-push
#     # pubsub_topic: gmail-push  # Shorthand when project_id is in credentials
#     
#     # Pull subscription (works with any network setup)
#     # pubsub_subscription_id: projects/<your-project>/subscriptions/gmail-pull-sub
#     # pubsub_pull_interval: "60s"
#     # pubsub_max_messages: 10
#     
#     # Disable push even if pubsub_topic is set (for debugging)
#     # disable_push: false
#     
#     watch_labels: [INBOX]
#     poll_interval: "60s"           # History API fallback interval
#     message: "New email from {{from}}: {{subject}}\n\n{{snippet}}"
#     output:
#       channel: telegram
#       peer: "123456789"

5. Dependencies

File: package.json

Add @google-cloud/pubsub for pull subscription support:

{
  "dependencies": {
    "@google-cloud/pubsub": "^4.0.0"
  }
}

Test Plan

5.1 Unit Tests

File: src/automation/gmail.test.ts

Add tests for new functionality:

describe('GmailWatcher - Pub/Sub Pull', () => {
  it('should set up pull subscription with full subscription path', async () => {
    const config = {
      ...baseConfig,
      pubsub_subscription_id: 'projects/test-project/subscriptions/gmail-pull',
      pubsub_pull_interval: '30s',
    };
    const watcher = new GmailWatcher(config, channelLookup);
    await watcher.connect();
    
    // Verify subscriptionPath is set
    expect(watcher['subscriptionPath']).toBe('projects/test-project/subscriptions/gmail-pull');
    expect(watcher['pullSubscriptionTimer']).toBeDefined();
    
    await watcher.disconnect();
    expect(watcher['pullSubscriptionTimer']).toBeUndefined();
  });

  it('should auto-prefix subscription with project_id when shorthand is used', async () => {
    const config = {
      ...baseConfig,
      pubsub_subscription_id: 'gmail-pull',
    };
    const watcher = new GmailWatcher(config, channelLookup);
    await watcher.connect();
    
    expect(watcher['subscriptionPath']).toBe('projects/test-project/subscriptions/gmail-pull');
    await watcher.disconnect();
  });

  it('should skip pull setup when pubsub_subscription_id is not set', async () => {
    const watcher = new GmailWatcher(baseConfig, channelLookup);
    await watcher.connect();
    
    expect(watcher['subscriptionPath']).toBeUndefined();
    expect(watcher['pullSubscriptionTimer']).toBeUndefined();
    await watcher.disconnect();
  });

  it('should warn on invalid subscription path format', async () => {
    const consoleSpy = vi.spyOn(console, 'warn').mockImplementation(() => {});
    
    const config = {
      ...baseConfig,
      pubsub_subscription_id: 'invalid-format',
      credentials_file: '~/.config/flynn/gmail-credentials-no-project.json',
    };
    
    // Mock credentials without project_id
    vi.mocked(readFileSync).mockImplementation((path: string) => {
      if (path.includes('credentials')) {
        return JSON.stringify({
          installed: {
            client_id: 'test-client-id',
            client_secret: 'test-client-secret',
            // No project_id
          },
        });
      }
      return JSON.stringify({ access_token: 'token' });
    });
    
    const watcher = new GmailWatcher(config, channelLookup);
    await watcher.connect();
    
    expect(consoleSpy).toHaveBeenCalledWith(
      expect.stringContaining('pubsub_subscription_id \'invalid-format\' must be fully qualified')
    );
    
    await watcher.disconnect();
    consoleSpy.mockRestore();
  });
});

describe('GmailWatcher - disable_push', () => {
  it('should skip watch setup when disable_push is true', async () => {
    const config = {
      ...baseConfig,
      pubsub_topic: 'projects/test-project/topics/gmail-push',
      disable_push: true,
    };
    
    const watcher = new GmailWatcher(config, channelLookup);
    await watcher.connect();
    
    // Watch should not be called
    const mockWatch = vi.mocked(google.gmail().users.watch);
    expect(mockWatch).not.toHaveBeenCalled();
    
    await watcher.disconnect();
  });

  it('should log push disabled message when disable_push is true', async () => {
    const consoleSpy = vi.spyOn(console, 'log').mockImplementation(() => {});
    
    const config = {
      ...baseConfig,
      pubsub_topic: 'gmail-push',
      disable_push: true,
    };
    
    const watcher = new GmailWatcher(config, channelLookup);
    await watcher.connect();
    
    expect(consoleSpy).toHaveBeenCalledWith(
      'GmailWatcher: Push notifications disabled (disable_push: true)'
    );
    
    await watcher.disconnect();
    consoleSpy.mockRestore();
  });
});

describe('GmailWatcher - buildWatchErrorHint', () => {
  it('should return topic hint for Invalid topicName error', () => {
    const watcher = new GmailWatcher(baseConfig, channelLookup);
    const hint = watcher['buildWatchErrorHint']('Invalid topicName');
    
    expect(hint).toContain('Set automation.gmail.pubsub_topic');
    expect(hint).toContain('projects/test-project/topics/gmail-push');
  });

  it('should return IAM hint for Permission denied error', () => {
    const watcher = new GmailWatcher(baseConfig, channelLookup);
    const hint = watcher['buildWatchErrorHint']('Permission denied');
    
    expect(hint).toContain('gcloud pubsub topics add-iam-policy-binding');
    expect(hint).toContain('gmail-api-push@system.gserviceaccount.com');
  });

  it('should return creation hint for Not found error', () => {
    const watcher = new GmailWatcher(baseConfig, channelLookup);
    const hint = watcher['buildWatchErrorHint']('Not found');
    
    expect(hint).toContain('gcloud pubsub topics create');
  });
});

describe('GmailWatcher - connection logging', () => {
  it('should log all active modes on connect', async () => {
    const consoleSpy = vi.spyOn(console, 'log').mockImplementation(() => {});
    
    const config = {
      ...baseConfig,
      pubsub_topic: 'gmail-push',
      pubsub_subscription_id: 'gmail-pull',
    };
    
    const watcher = new GmailWatcher(config, channelLookup);
    await watcher.connect();
    
    expect(consoleSpy).toHaveBeenCalledWith(
      expect.stringContaining('push=true, pull=true, poll_interval=')
    );
    
    await watcher.disconnect();
    consoleSpy.mockRestore();
  });
});

5.2 Doctor Check Tests

File: src/cli/doctor.test.ts

Add tests for enhanced Gmail validation:

describe('doctor - checkGmail enhanced', () => {
  it('should detect push mode', async () => {
    const config = {
      ...baseConfig,
      automation: {
        gmail: {
          enabled: true,
          credentials_file: '/mock/credentials.json',
          token_file: '/mock/token.json',
          pubsub_topic: 'projects/test/topics/gmail-push',
          output: { channel: 'telegram', peer: '123' },
        },
      },
    };
    
    const result = await checkGmail({ config });
    expect(result.status).toBe('pass');
    expect(result.detail).toContain('push + poll-fallback');
  });

  it('should detect pull mode', async () => {
    const config = {
      ...baseConfig,
      automation: {
        gmail: {
          enabled: true,
          credentials_file: '/mock/credentials.json',
          token_file: '/mock/token.json',
          pubsub_subscription_id: 'projects/test/subscriptions/gmail-pull',
          output: { channel: 'telegram', peer: '123' },
        },
      },
    };
    
    const result = await checkGmail({ config });
    expect(result.status).toBe('pass');
    expect(result.detail).toContain('pull + poll-fallback');
  });

  it('should detect hybrid mode', async () => {
    const config = {
      ...baseConfig,
      automation: {
        gmail: {
          enabled: true,
          credentials_file: '/mock/credentials.json',
          token_file: '/mock/token.json',
          pubsub_topic: 'gmail-push',
          pubsub_subscription_id: 'gmail-pull',
          output: { channel: 'telegram', peer: '123' },
        },
      },
    };
    
    const result = await checkGmail({ config });
    expect(result.status).toBe('pass');
    expect(result.detail).toContain('push + pull + poll-fallback');
  });

  it('should warn when push requires public endpoint but tailscale_only is enabled', async () => {
    const config = {
      ...baseConfig,
      server: {
        tailscale_only: true,
      },
      automation: {
        gmail: {
          enabled: true,
          credentials_file: '/mock/credentials.json',
          token_file: '/mock/token.json',
          pubsub_topic: 'gmail-push',
          output: { channel: 'telegram', peer: '123' },
        },
      },
    };
    
    const result = await checkGmail({ config });
    expect(result.status).toBe('warn');
    expect(result.detail).toContain('push requires public endpoint or Tailscale funnel');
  });

  it('should pass when tailscale funnel is enabled', async () => {
    const config = {
      ...baseConfig,
      server: {
        tailscale_only: true,
        tailscale: { funnel: true },
      },
      automation: {
        gmail: {
          enabled: true,
          credentials_file: '/mock/credentials.json',
          token_file: '/mock/token.json',
          pubsub_topic: 'gmail-push',
          output: { channel: 'telegram', peer: '123' },
        },
      },
    };
    
    const result = await checkGmail({ config });
    expect(result.status).toBe('pass');
    expect(result.detail).not.toContain('⚠️');
  });
});

5.3 Integration Tests

Manual testing checklist:

Push only: Set pubsub_topic, start daemon, send test email, verify POST arrives at /gmail/push, verify agent response
Pull only: Set pubsub_subscription_id, start daemon, send test email, verify pull retrieves message, verify agent response
Hybrid: Set both, start daemon, send test email, verify either push or pull handles it (no duplicate processing)
Polling only: Omit both, start daemon, send test email, verify History API poll finds it
Disable push: Set disable_push: true with pubsub_topic, verify watch is not called
Doctor check: Run flynn doctor with each mode, verify output shows correct modes
Error messages: Trigger watch failure (invalid topic), verify hint contains actionable guidance
Tailscale warning: Set tailscale_only: true without funnel, verify doctor warns about push

Files to Modify

Core Implementation

File	Changes
`src/config/schema.ts`	Add `pubsub_subscription_id`, `pubsub_pull_interval`, `pubsub_max_messages`, `disable_push` fields
`src/automation/gmail.ts`	Add `setupPullSubscription()`, `pullSubscriptionMessages()`, `buildWatchErrorHint()`, update `connect()` / `disconnect()`
`src/cli/doctor.ts`	Enhance `checkGmail` with Pub/Sub mode detection and Tailscale reachability check
`package.json`	Add `@google-cloud/pubsub` dependency

Documentation

File	Changes
`README.md`	Rewrite Gmail section with deployment patterns, setup instructions, troubleshooting
`config/default.yaml`	Update Gmail example with new fields and comments
`docs/plans/state.json`	Mark `gmail-push-revisit` as completed

Tests

File	Changes
`src/automation/gmail.test.ts`	Add tests for pull mode, `disable_push`, error hints, connection logging
`src/cli/doctor.test.ts`	Add tests for enhanced Gmail validation (mode detection, Tailscale warning)

Summary

This implementation plan makes Gmail notifications operationally reliable by:

Supporting three deployment patterns: Push (public endpoint), Pull (private), Hybrid (both)
Clarifying config surface: Explicit pubsub_subscription_id, pubsub_pull_interval, disable_push fields
Improving error messages: Actionable hints for common setup failures (invalid topic, missing IAM, etc.)
Enhancing diagnostics: flynn doctor reports active modes and warns about reachability issues
Comprehensive documentation: Setup guides for each pattern, GCP resource instructions, troubleshooting

Recommended default: Hybrid mode (push + pull + polling) for maximum reliability across all network configurations.

Estimated effort: 1-2 days (implementation + testing + docs)

No breaking changes: Existing pubsub_topic-only configs continue to work (push + polling)

34 KiB Raw Permalink Blame History

Gmail Push Notifications Revisit — Implementation Plan

Overview

Problem Analysis

Current Behavior

Deployment Patterns

Recommended Changes

1. Config Schema Additions

2. Implementation Changes

2.1 Add Pub/Sub Pull Support

2.2 Improve Error Messages

3. Doctor Checks

4. Documentation Updates

4.1 README.md

Deployment Patterns

Push Notifications Setup

Pull Subscription Setup

Hybrid Mode (Recommended)

Configuration Reference

Template Variables

Troubleshooting

5. Dependencies

Test Plan

5.1 Unit Tests

5.2 Doctor Check Tests

5.3 Integration Tests

Files to Modify

Core Implementation

Documentation

Tests

Summary

34 KiB

Raw Permalink Blame History