1008 lines
33 KiB
Markdown
1008 lines
33 KiB
Markdown
# Gmail Push Notifications Revisit — Implementation Plan
|
|
|
|
**Date**: 2026-02-13
|
|
**Status**: Planned
|
|
**Goal**: Make Gmail notifications operationally reliable and well-documented
|
|
|
|
## Overview
|
|
|
|
The Gmail watcher currently supports two modes:
|
|
1. **Pub/Sub push notifications** (Google sends HTTP POST to `/gmail/push`)
|
|
2. **Polling fallback** (History API queries every N seconds)
|
|
|
|
Current implementation has operational ambiguity around:
|
|
- Deployment patterns (public webhook vs Pub/Sub pull vs hybrid)
|
|
- GCP resource setup (topic/subscription/IAM relationships)
|
|
- Network reachability constraints (Tailscale-only gateways)
|
|
- Configuration surface (implicit vs explicit controls)
|
|
- Error diagnostics (vague failure messages)
|
|
|
|
This plan addresses these gaps without changing core functionality.
|
|
|
|
---
|
|
|
|
## Problem Analysis
|
|
|
|
### Current Behavior
|
|
|
|
**Push setup flow** (`src/automation/gmail.ts:222-253`):
|
|
1. `resolvePubSubTopicName()` reads `pubsub_topic` config or `FLYNN_GMAIL_PUBSUB_TOPIC` env var
|
|
2. Calls `gmail.users.watch()` with the topic
|
|
3. If watch fails → logs warning, continues with polling only
|
|
4. If watch succeeds → schedules renewal every 6 days
|
|
|
|
**Issues**:
|
|
- **Topic ownership mismatch**: OAuth credentials' `project_id` may differ from the Pub/Sub topic's project
|
|
- **Network reachability**: Google cannot reach Tailscale-only endpoints (`server.tailscale_only: true`)
|
|
- **Implicit disable**: No explicit way to disable push without deleting `pubsub_topic` field
|
|
- **Poor error messages**: "Invalid topicName" error includes no actionable guidance
|
|
- **Undocumented subscription**: Users must manually create Pub/Sub subscription → push endpoint mapping
|
|
- **Doctor check gap**: `flynn doctor` validates OAuth but not Pub/Sub setup
|
|
|
|
### Deployment Patterns
|
|
|
|
| Pattern | Reachability | Flynn Config | GCP Setup | Use Case |
|
|
|---------|--------------|--------------|-----------|----------|
|
|
| **Public webhook** | Public IP + DNS | `pubsub_topic` set | Topic + push subscription → `POST https://example.com/gmail/push` | Production deployment with public ingress |
|
|
| **Tailscale Serve** | Tailscale funnel | `pubsub_topic` set, `server.tailscale.funnel: true` | Topic + push subscription → `POST https://flynn.ts.net/gmail/push` | Private deployment with Tailscale funnel enabled |
|
|
| **Pub/Sub pull** | Any (no inbound) | `pubsub_subscription_id` set | Topic + pull subscription | Private deployment, Flynn polls subscription |
|
|
| **Polling only** | Any (no inbound) | `pubsub_topic` omitted | None | Development, no GCP setup |
|
|
| **Hybrid (recommended)** | Any | `pubsub_topic` + `pubsub_subscription_id` | Topic + push subscription + pull subscription | Push when reachable, pull fallback, History API tertiary |
|
|
|
|
---
|
|
|
|
## Recommended Changes
|
|
|
|
### 1. Config Schema Additions
|
|
|
|
**File**: `src/config/schema.ts`
|
|
|
|
Add new fields to `gmailSchema`:
|
|
|
|
```typescript
|
|
const gmailSchema = z.object({
|
|
enabled: z.boolean().default(false),
|
|
credentials_file: z.string().optional(),
|
|
token_file: z.string().default('~/.config/flynn/gmail-token.json'),
|
|
|
|
// ── Push notifications ────────────────────────────────────────────────
|
|
/**
|
|
* Pub/Sub topic for Gmail push notifications (watch API).
|
|
* Format: projects/<project-id>/topics/<topic> or just <topic> (auto-prefixed).
|
|
* If omitted, push notifications are disabled (polling only).
|
|
*/
|
|
pubsub_topic: z.string().optional(),
|
|
|
|
/**
|
|
* Pub/Sub subscription ID for pull-based message retrieval.
|
|
* Format: projects/<project-id>/subscriptions/<subscription> or just <subscription>.
|
|
* When set, Flynn periodically pulls messages from this subscription in addition
|
|
* to waiting for push notifications at POST /gmail/push.
|
|
* Enables hybrid mode: push when reachable, pull fallback.
|
|
*/
|
|
pubsub_subscription_id: z.string().optional(),
|
|
|
|
/**
|
|
* Pull interval for Pub/Sub subscription polling (hybrid mode).
|
|
* Only used when pubsub_subscription_id is set. Default: 60s.
|
|
*/
|
|
pubsub_pull_interval: z.string().default('60s'),
|
|
|
|
/**
|
|
* Maximum messages to pull per request when using pull mode. Default: 10.
|
|
*/
|
|
pubsub_max_messages: z.number().min(1).max(100).default(10),
|
|
|
|
/**
|
|
* Disable watch API even if pubsub_topic is set.
|
|
* Useful for debugging or when push endpoint is temporarily unreachable.
|
|
*/
|
|
disable_push: z.boolean().default(false),
|
|
|
|
// ── Existing fields ────────────────────────────────────────────────────
|
|
watch_labels: z.array(z.string()).default(['INBOX']),
|
|
poll_interval: z.string().default('300s'), // History API fallback
|
|
history_start: z.string().optional(),
|
|
output: z.object({
|
|
channel: z.string().min(1),
|
|
peer: z.string().min(1),
|
|
}),
|
|
message: z.string().default('New email from {{from}}: {{subject}}\n\n{{snippet}}'),
|
|
}).optional();
|
|
```
|
|
|
|
**Defaults**:
|
|
- `pubsub_topic`: `undefined` (no push)
|
|
- `pubsub_subscription_id`: `undefined` (no pull)
|
|
- `pubsub_pull_interval`: `'60s'`
|
|
- `pubsub_max_messages`: `10`
|
|
- `disable_push`: `false`
|
|
- `poll_interval`: `'300s'` (History API fallback)
|
|
|
|
---
|
|
|
|
### 2. Implementation Changes
|
|
|
|
#### 2.1 Add Pub/Sub Pull Support
|
|
|
|
**File**: `src/automation/gmail.ts`
|
|
|
|
Add new method `setupPullSubscription()`:
|
|
|
|
```typescript
|
|
private pullSubscriptionTimer?: ReturnType<typeof setInterval>;
|
|
private subscriptionPath?: string;
|
|
|
|
/**
|
|
* Set up Pub/Sub pull subscription for periodic message retrieval.
|
|
* Alternative/fallback to push notifications when endpoint is unreachable.
|
|
*/
|
|
private async setupPullSubscription(): Promise<void> {
|
|
if (!this.config.pubsub_subscription_id) {
|
|
return; // Pull mode not configured
|
|
}
|
|
|
|
const subId = this.config.pubsub_subscription_id.trim();
|
|
|
|
// Allow shorthand: just the subscription id (e.g. "gmail-pull")
|
|
if (!subId.includes('/')) {
|
|
if (!this.googleProjectId) {
|
|
console.warn(
|
|
`GmailWatcher: pubsub_subscription_id '${subId}' must be fully qualified ` +
|
|
`(projects/<project-id>/subscriptions/<sub>) because project_id was not found in credentials`
|
|
);
|
|
return;
|
|
}
|
|
this.subscriptionPath = `projects/${this.googleProjectId}/subscriptions/${subId}`;
|
|
} else {
|
|
const isValid = /^projects\/[^/]+\/subscriptions\/[^/]+$/.test(subId);
|
|
if (!isValid) {
|
|
console.warn(
|
|
`GmailWatcher: Invalid pubsub_subscription_id '${subId}'. ` +
|
|
`Expected: projects/<project-id>/subscriptions/<sub>`
|
|
);
|
|
return;
|
|
}
|
|
this.subscriptionPath = subId;
|
|
}
|
|
|
|
console.log(`GmailWatcher: Pull subscription registered (${this.subscriptionPath})`);
|
|
|
|
// Start periodic pull
|
|
const pullMs = parseInterval(this.config.pubsub_pull_interval ?? '60s');
|
|
this.pullSubscriptionTimer = setInterval(() => {
|
|
this.pullSubscriptionMessages().catch((err) => {
|
|
console.error('GmailWatcher: Pull subscription error —', err instanceof Error ? err.message : err);
|
|
});
|
|
}, pullMs);
|
|
}
|
|
|
|
/**
|
|
* Pull messages from the Pub/Sub subscription and process them.
|
|
*/
|
|
private async pullSubscriptionMessages(): Promise<void> {
|
|
if (!this.subscriptionPath || !this.oauth2Client) {
|
|
return;
|
|
}
|
|
|
|
try {
|
|
const { PubSub } = await import('@google-cloud/pubsub');
|
|
const pubsub = new PubSub({
|
|
projectId: this.googleProjectId,
|
|
authClient: this.oauth2Client,
|
|
});
|
|
|
|
const subscription = pubsub.subscription(this.subscriptionPath);
|
|
const [messages] = await subscription.pull({
|
|
maxMessages: this.config.pubsub_max_messages ?? 10,
|
|
});
|
|
|
|
for (const message of messages) {
|
|
if (message.data) {
|
|
await this.handlePushNotification(message.data.toString('base64'));
|
|
}
|
|
|
|
// Acknowledge the message
|
|
await subscription.ack(message);
|
|
}
|
|
} catch (error) {
|
|
console.error(
|
|
'GmailWatcher: Failed to pull Pub/Sub messages —',
|
|
error instanceof Error ? error.message : error
|
|
);
|
|
}
|
|
}
|
|
```
|
|
|
|
Update `connect()` to call `setupPullSubscription()`:
|
|
|
|
```typescript
|
|
async connect(): Promise<void> {
|
|
this._status = 'connecting';
|
|
|
|
try {
|
|
this.oauth2Client = await this.authorize();
|
|
} catch (error) {
|
|
const errMsg = error instanceof Error ? error.message : 'Unknown error';
|
|
console.error(`GmailWatcher: Authorization failed — ${errMsg}`);
|
|
console.error('GmailWatcher: Run "flynn gmail-auth" to set up Gmail credentials.');
|
|
this._status = 'error';
|
|
return;
|
|
}
|
|
|
|
// Set up Gmail push watch (Pub/Sub) if not disabled
|
|
if (!this.config.disable_push) {
|
|
try {
|
|
await this.setupWatch();
|
|
} catch (error) {
|
|
const errMsg = error instanceof Error ? error.message : 'Unknown error';
|
|
const hint = this.buildWatchErrorHint(errMsg);
|
|
console.warn(`GmailWatcher: Watch setup failed (push disabled) — ${errMsg}${hint}`);
|
|
}
|
|
} else {
|
|
console.log('GmailWatcher: Push notifications disabled (disable_push: true)');
|
|
}
|
|
|
|
// Set up Pub/Sub pull subscription (hybrid mode)
|
|
await this.setupPullSubscription();
|
|
|
|
// Start History API polling fallback
|
|
const pollMs = parseInterval(this.config.poll_interval ?? '300s');
|
|
this.pollTimer = setInterval(() => {
|
|
this.pollForNewMessages().catch((err) => {
|
|
console.error('GmailWatcher: Poll error —', err instanceof Error ? err.message : err);
|
|
});
|
|
}, pollMs);
|
|
|
|
this._status = 'connected';
|
|
console.log(
|
|
`GmailWatcher: Connected (` +
|
|
`push=${!this.config.disable_push && !!this.config.pubsub_topic}, ` +
|
|
`pull=${!!this.config.pubsub_subscription_id}, ` +
|
|
`poll_interval=${this.config.poll_interval ?? '300s'})`
|
|
);
|
|
auditLogger?.systemStart('GmailWatcher', {
|
|
push: !this.config.disable_push && !!this.config.pubsub_topic,
|
|
pull: !!this.config.pubsub_subscription_id,
|
|
poll_interval: this.config.poll_interval,
|
|
});
|
|
}
|
|
```
|
|
|
|
Update `disconnect()` to clear pull timer:
|
|
|
|
```typescript
|
|
async disconnect(): Promise<void> {
|
|
if (this.pollTimer) {
|
|
clearInterval(this.pollTimer);
|
|
this.pollTimer = undefined;
|
|
}
|
|
if (this.watchTimer) {
|
|
clearTimeout(this.watchTimer);
|
|
this.watchTimer = undefined;
|
|
}
|
|
if (this.pullSubscriptionTimer) {
|
|
clearInterval(this.pullSubscriptionTimer);
|
|
this.pullSubscriptionTimer = undefined;
|
|
}
|
|
this.oauth2Client = undefined;
|
|
this._status = 'disconnected';
|
|
auditLogger?.systemStop('GmailWatcher');
|
|
}
|
|
```
|
|
|
|
#### 2.2 Improve Error Messages
|
|
|
|
**File**: `src/automation/gmail.ts`
|
|
|
|
Add helper method `buildWatchErrorHint()`:
|
|
|
|
```typescript
|
|
/**
|
|
* Build a helpful error hint based on the watch failure message.
|
|
*/
|
|
private buildWatchErrorHint(errMsg: string): string {
|
|
if (errMsg.includes('Invalid topicName')) {
|
|
const topicHint = this.googleProjectId
|
|
? `projects/${this.googleProjectId}/topics/gmail-push`
|
|
: '<project-id>/topics/gmail-push';
|
|
|
|
return `\n` +
|
|
` Tip: Set automation.gmail.pubsub_topic to "${topicHint}"\n` +
|
|
` Ensure the topic exists in GCP and the subscription is configured to POST to /gmail/push\n` +
|
|
` Or set automation.gmail.pubsub_subscription_id for pull-based mode (no webhook required)`;
|
|
}
|
|
|
|
if (errMsg.includes('Permission denied') || errMsg.includes('Forbidden')) {
|
|
return `\n` +
|
|
` Tip: Grant Gmail API permission to publish to the Pub/Sub topic:\n` +
|
|
` gcloud pubsub topics add-iam-policy-binding <topic> \\\n` +
|
|
` --member=serviceAccount:gmail-api-push@system.gserviceaccount.com \\\n` +
|
|
` --role=roles/pubsub.publisher`;
|
|
}
|
|
|
|
if (errMsg.includes('Not found') || errMsg.includes('404')) {
|
|
return `\n` +
|
|
` Tip: Create the Pub/Sub topic first:\n` +
|
|
` gcloud pubsub topics create gmail-push --project=<project-id>`;
|
|
}
|
|
|
|
return '';
|
|
}
|
|
```
|
|
|
|
Update `setupWatch()` error handling:
|
|
|
|
```typescript
|
|
try {
|
|
await this.setupWatch();
|
|
} catch (error) {
|
|
const errMsg = error instanceof Error ? error.message : 'Unknown error';
|
|
const hint = this.buildWatchErrorHint(errMsg);
|
|
console.warn(`GmailWatcher: Watch setup failed (push disabled) — ${errMsg}${hint}`);
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### 3. Doctor Checks
|
|
|
|
**File**: `src/cli/doctor.ts`
|
|
|
|
Enhance `checkGmail` to validate Pub/Sub setup:
|
|
|
|
```typescript
|
|
const checkGmail: Check = async (ctx) => {
|
|
if (!ctx.config) {
|
|
return { status: 'skip', label: 'Gmail configured', detail: '(config invalid)' };
|
|
}
|
|
const gmail = ctx.config.automation.gmail;
|
|
if (!gmail?.enabled) {
|
|
return { status: 'skip', label: 'Gmail configured', detail: '(not enabled)' };
|
|
}
|
|
|
|
// 1. Check credentials file
|
|
const credentialsPath = expandPath(gmail.credentials_file ?? '~/.config/flynn/gmail-credentials.json');
|
|
if (!existsSync(credentialsPath)) {
|
|
return { status: 'fail', label: 'Gmail configured', detail: `credentials file not found: ${credentialsPath}` };
|
|
}
|
|
|
|
// 2. Check token file
|
|
const tokenPath = expandPath(gmail.token_file ?? '~/.config/flynn/gmail-token.json');
|
|
if (!existsSync(tokenPath)) {
|
|
return { status: 'warn', label: 'Gmail configured', detail: 'run `flynn gmail-auth` to authenticate' };
|
|
}
|
|
|
|
// 3. Parse credentials to get project_id
|
|
let projectId: string | undefined;
|
|
try {
|
|
const creds = JSON.parse(readFileSync(credentialsPath, 'utf-8'));
|
|
projectId = creds.installed?.project_id ?? creds.web?.project_id;
|
|
} catch {
|
|
// Ignore parse errors — already validated above
|
|
}
|
|
|
|
// 4. Check Pub/Sub configuration
|
|
const modes: string[] = [];
|
|
const warnings: string[] = [];
|
|
|
|
if (gmail.pubsub_topic && !gmail.disable_push) {
|
|
const topic = gmail.pubsub_topic.trim();
|
|
if (!topic.includes('/') && !projectId) {
|
|
warnings.push('pubsub_topic requires project_id in credentials');
|
|
} else {
|
|
modes.push('push');
|
|
|
|
// Warn if tailscale_only is enabled (Google cannot reach tailnet-only endpoints)
|
|
if (ctx.config.server?.tailscale_only && !ctx.config.server?.tailscale?.funnel) {
|
|
warnings.push('push requires public endpoint or Tailscale funnel');
|
|
}
|
|
}
|
|
}
|
|
|
|
if (gmail.pubsub_subscription_id) {
|
|
const sub = gmail.pubsub_subscription_id.trim();
|
|
if (!sub.includes('/') && !projectId) {
|
|
warnings.push('pubsub_subscription_id requires project_id in credentials');
|
|
} else {
|
|
modes.push('pull');
|
|
}
|
|
}
|
|
|
|
if (modes.length === 0) {
|
|
modes.push('poll');
|
|
} else {
|
|
modes.push('poll-fallback');
|
|
}
|
|
|
|
const modeStr = modes.join(' + ');
|
|
const detail = warnings.length > 0
|
|
? `${modeStr} (⚠️ ${warnings.join(', ')})`
|
|
: `${modeStr} → ${gmail.output.channel}/${gmail.output.peer}`;
|
|
|
|
return {
|
|
status: warnings.length > 0 ? 'warn' : 'pass',
|
|
label: 'Gmail configured',
|
|
detail,
|
|
};
|
|
};
|
|
```
|
|
|
|
---
|
|
|
|
### 4. Documentation Updates
|
|
|
|
#### 4.1 README.md
|
|
|
|
**File**: `README.md`
|
|
|
|
Update Gmail section (lines 461-512):
|
|
|
|
```markdown
|
|
### Gmail Watcher
|
|
|
|
Monitor a Gmail inbox for new messages and trigger agent responses. Flynn supports three retrieval modes:
|
|
|
|
1. **Push notifications** (Pub/Sub watch) — Google sends HTTP POST to `/gmail/push`
|
|
2. **Pull subscription** (Pub/Sub pull) — Flynn periodically pulls messages from a subscription
|
|
3. **History API polling** (fallback) — Flynn queries the History API every N seconds
|
|
|
|
**Recommended deployment**: Hybrid mode (push + pull + polling) for maximum reliability.
|
|
|
|
#### Prerequisites
|
|
|
|
1. **Google Cloud Project** with Gmail API and Pub/Sub API enabled
|
|
2. **OAuth2 credentials** (Desktop application) — download JSON from Cloud Console
|
|
3. **Pub/Sub topic** (for push or pull):
|
|
```bash
|
|
gcloud pubsub topics create gmail-push --project=your-project
|
|
```
|
|
4. **Grant Gmail API publish permission**:
|
|
```bash
|
|
gcloud pubsub topics add-iam-policy-binding projects/your-project/topics/gmail-push \
|
|
--member=serviceAccount:gmail-api-push@system.gserviceaccount.com \
|
|
--role=roles/pubsub.publisher
|
|
```
|
|
5. **Authenticate**:
|
|
```bash
|
|
flynn gmail-auth
|
|
```
|
|
|
|
#### Deployment Patterns
|
|
|
|
| Mode | Config | GCP Setup | Network | Use Case |
|
|
|------|--------|-----------|---------|----------|
|
|
| **Push only** | `pubsub_topic` | Topic + push subscription → `POST https://example.com/gmail/push` | Public IP or Tailscale funnel | Production with ingress |
|
|
| **Pull only** | `pubsub_subscription_id` | Topic + pull subscription | Any (no inbound) | Private deployment |
|
|
| **Hybrid** ⭐ | Both | Topic + push + pull subscriptions | Any | Maximum reliability |
|
|
| **Polling only** | Neither | None | Any | Development/testing |
|
|
|
|
#### Push Notifications Setup
|
|
|
|
When `pubsub_topic` is set, Flynn calls `gmail.users.watch()` to register for push notifications.
|
|
|
|
1. Create a **push subscription** in GCP:
|
|
```bash
|
|
gcloud pubsub subscriptions create gmail-push-sub \
|
|
--topic=gmail-push \
|
|
--push-endpoint=https://your-domain.com/gmail/push
|
|
```
|
|
|
|
2. If using **Tailscale Serve**, enable funnel:
|
|
```yaml
|
|
server:
|
|
tailscale:
|
|
serve: true
|
|
funnel: true # ← Required for Google to reach your endpoint
|
|
```
|
|
|
|
3. Set `pubsub_topic` in config:
|
|
```yaml
|
|
automation:
|
|
gmail:
|
|
enabled: true
|
|
pubsub_topic: projects/your-project/topics/gmail-push
|
|
```
|
|
|
|
**Important**: Push notifications require a publicly reachable endpoint. If `server.tailscale_only: true` and funnel is disabled, push will fail (Flynn falls back to pull/poll).
|
|
|
|
#### Pull Subscription Setup
|
|
|
|
When `pubsub_subscription_id` is set, Flynn periodically pulls messages from the subscription.
|
|
|
|
1. Create a **pull subscription** in GCP:
|
|
```bash
|
|
gcloud pubsub subscriptions create gmail-pull-sub \
|
|
--topic=gmail-push \
|
|
--ack-deadline=60
|
|
```
|
|
|
|
2. Set `pubsub_subscription_id` in config:
|
|
```yaml
|
|
automation:
|
|
gmail:
|
|
enabled: true
|
|
pubsub_subscription_id: projects/your-project/topics/gmail-pull-sub
|
|
pubsub_pull_interval: "60s"
|
|
```
|
|
|
|
**Note**: Pull mode works with any network setup (no inbound connections required).
|
|
|
|
#### Hybrid Mode (Recommended)
|
|
|
|
Combine push + pull for maximum reliability:
|
|
|
|
```yaml
|
|
automation:
|
|
gmail:
|
|
enabled: true
|
|
credentials_file: ~/.config/flynn/gmail-credentials.json
|
|
token_file: ~/.config/flynn/gmail-token.json
|
|
|
|
# Push notifications (when endpoint is reachable)
|
|
pubsub_topic: projects/your-project/topics/gmail-push
|
|
|
|
# Pull fallback (always works)
|
|
pubsub_subscription_id: projects/your-project/subscriptions/gmail-pull-sub
|
|
pubsub_pull_interval: "60s"
|
|
|
|
# History API tertiary fallback
|
|
poll_interval: "300s"
|
|
|
|
watch_labels: [INBOX]
|
|
message: "New email from {{from}}: {{subject}}\n\n{{snippet}}"
|
|
output:
|
|
channel: telegram
|
|
peer: "123456789"
|
|
```
|
|
|
|
#### Configuration Reference
|
|
|
|
| Field | Required | Default | Description |
|
|
|-------|----------|---------|-------------|
|
|
| `enabled` | no | `false` | Enable the Gmail watcher |
|
|
| `credentials_file` | yes | — | Path to OAuth2 credentials JSON |
|
|
| `token_file` | no | `~/.config/flynn/gmail-token.json` | Path to stored refresh token |
|
|
| `pubsub_topic` | no | — | Pub/Sub topic for push notifications (format: `projects/<project>/topics/<topic>`) |
|
|
| `pubsub_subscription_id` | no | — | Pub/Sub subscription for pull mode (format: `projects/<project>/subscriptions/<sub>`) |
|
|
| `pubsub_pull_interval` | no | `60s` | Pull interval when using subscription |
|
|
| `pubsub_max_messages` | no | `10` | Max messages per pull request (1-100) |
|
|
| `disable_push` | no | `false` | Disable watch API even if `pubsub_topic` is set |
|
|
| `watch_labels` | no | `[INBOX]` | Gmail labels to watch |
|
|
| `poll_interval` | no | `300s` | History API fallback interval |
|
|
| `history_start` | no | — | ISO date — only process emails after this date |
|
|
| `message` | no | `New email from {{from}}: {{subject}}\n\n{{snippet}}` | Template for agent message |
|
|
| `output.channel` | yes | — | Channel to route responses (e.g. `telegram`) |
|
|
| `output.peer` | yes | — | Peer/chat ID on output channel |
|
|
|
|
#### Template Variables
|
|
|
|
| Variable | Description |
|
|
|----------|-------------|
|
|
| `{{from}}` | Sender email address |
|
|
| `{{to}}` | Recipient email address |
|
|
| `{{subject}}` | Email subject line |
|
|
| `{{snippet}}` | Gmail-provided snippet (first ~200 chars) |
|
|
| `{{date}}` | Email date header |
|
|
| `{{id}}` | Gmail message ID |
|
|
| `{{labels}}` | Comma-separated label names |
|
|
|
|
#### Troubleshooting
|
|
|
|
**"Watch setup failed — Invalid topicName"**
|
|
- Ensure `pubsub_topic` format is correct: `projects/<project>/topics/<topic>`
|
|
- Verify the topic exists: `gcloud pubsub topics list`
|
|
- Check IAM permissions (Gmail API must be able to publish)
|
|
|
|
**"Permission denied" or "Forbidden"**
|
|
- Grant publish permission to Gmail API:
|
|
```bash
|
|
gcloud pubsub topics add-iam-policy-binding <topic> \
|
|
--member=serviceAccount:gmail-api-push@system.gserviceaccount.com \
|
|
--role=roles/pubsub.publisher
|
|
```
|
|
|
|
**Push notifications not arriving**
|
|
- Verify endpoint is reachable: `curl -X POST https://your-domain.com/gmail/push`
|
|
- Check subscription configuration: `gcloud pubsub subscriptions describe <sub>`
|
|
- Enable Tailscale funnel if using `tailscale_only: true`
|
|
- Fall back to pull or polling mode
|
|
|
|
**Check status**:
|
|
```bash
|
|
flynn doctor # Validates config and reports active modes (push/pull/poll)
|
|
```
|
|
```
|
|
|
|
#### 4.2 config/default.yaml
|
|
|
|
**File**: `config/default.yaml`
|
|
|
|
Update Gmail example (lines 128-136):
|
|
|
|
```yaml
|
|
# gmail:
|
|
# enabled: false
|
|
# credentials_file: ~/.config/flynn/gmail-credentials.json
|
|
# token_file: ~/.config/flynn/gmail-token.json
|
|
#
|
|
# # Push notifications (requires public endpoint or Tailscale funnel)
|
|
# pubsub_topic: projects/<your-project>/topics/gmail-push
|
|
# # pubsub_topic: gmail-push # Shorthand when project_id is in credentials
|
|
#
|
|
# # Pull subscription (works with any network setup)
|
|
# # pubsub_subscription_id: projects/<your-project>/subscriptions/gmail-pull-sub
|
|
# # pubsub_pull_interval: "60s"
|
|
# # pubsub_max_messages: 10
|
|
#
|
|
# # Disable push even if pubsub_topic is set (for debugging)
|
|
# # disable_push: false
|
|
#
|
|
# watch_labels: [INBOX]
|
|
# poll_interval: "60s" # History API fallback interval
|
|
# message: "New email from {{from}}: {{subject}}\n\n{{snippet}}"
|
|
# output:
|
|
# channel: telegram
|
|
# peer: "123456789"
|
|
```
|
|
|
|
---
|
|
|
|
### 5. Dependencies
|
|
|
|
**File**: `package.json`
|
|
|
|
Add `@google-cloud/pubsub` for pull subscription support:
|
|
|
|
```json
|
|
{
|
|
"dependencies": {
|
|
"@google-cloud/pubsub": "^4.0.0"
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Test Plan
|
|
|
|
### 5.1 Unit Tests
|
|
|
|
**File**: `src/automation/gmail.test.ts`
|
|
|
|
Add tests for new functionality:
|
|
|
|
```typescript
|
|
describe('GmailWatcher - Pub/Sub Pull', () => {
|
|
it('should set up pull subscription with full subscription path', async () => {
|
|
const config = {
|
|
...baseConfig,
|
|
pubsub_subscription_id: 'projects/test-project/subscriptions/gmail-pull',
|
|
pubsub_pull_interval: '30s',
|
|
};
|
|
const watcher = new GmailWatcher(config, channelLookup);
|
|
await watcher.connect();
|
|
|
|
// Verify subscriptionPath is set
|
|
expect(watcher['subscriptionPath']).toBe('projects/test-project/subscriptions/gmail-pull');
|
|
expect(watcher['pullSubscriptionTimer']).toBeDefined();
|
|
|
|
await watcher.disconnect();
|
|
expect(watcher['pullSubscriptionTimer']).toBeUndefined();
|
|
});
|
|
|
|
it('should auto-prefix subscription with project_id when shorthand is used', async () => {
|
|
const config = {
|
|
...baseConfig,
|
|
pubsub_subscription_id: 'gmail-pull',
|
|
};
|
|
const watcher = new GmailWatcher(config, channelLookup);
|
|
await watcher.connect();
|
|
|
|
expect(watcher['subscriptionPath']).toBe('projects/test-project/subscriptions/gmail-pull');
|
|
await watcher.disconnect();
|
|
});
|
|
|
|
it('should skip pull setup when pubsub_subscription_id is not set', async () => {
|
|
const watcher = new GmailWatcher(baseConfig, channelLookup);
|
|
await watcher.connect();
|
|
|
|
expect(watcher['subscriptionPath']).toBeUndefined();
|
|
expect(watcher['pullSubscriptionTimer']).toBeUndefined();
|
|
await watcher.disconnect();
|
|
});
|
|
|
|
it('should warn on invalid subscription path format', async () => {
|
|
const consoleSpy = vi.spyOn(console, 'warn').mockImplementation(() => {});
|
|
|
|
const config = {
|
|
...baseConfig,
|
|
pubsub_subscription_id: 'invalid-format',
|
|
credentials_file: '~/.config/flynn/gmail-credentials-no-project.json',
|
|
};
|
|
|
|
// Mock credentials without project_id
|
|
vi.mocked(readFileSync).mockImplementation((path: string) => {
|
|
if (path.includes('credentials')) {
|
|
return JSON.stringify({
|
|
installed: {
|
|
client_id: 'test-client-id',
|
|
client_secret: 'test-client-secret',
|
|
// No project_id
|
|
},
|
|
});
|
|
}
|
|
return JSON.stringify({ access_token: 'token' });
|
|
});
|
|
|
|
const watcher = new GmailWatcher(config, channelLookup);
|
|
await watcher.connect();
|
|
|
|
expect(consoleSpy).toHaveBeenCalledWith(
|
|
expect.stringContaining('pubsub_subscription_id \'invalid-format\' must be fully qualified')
|
|
);
|
|
|
|
await watcher.disconnect();
|
|
consoleSpy.mockRestore();
|
|
});
|
|
});
|
|
|
|
describe('GmailWatcher - disable_push', () => {
|
|
it('should skip watch setup when disable_push is true', async () => {
|
|
const config = {
|
|
...baseConfig,
|
|
pubsub_topic: 'projects/test-project/topics/gmail-push',
|
|
disable_push: true,
|
|
};
|
|
|
|
const watcher = new GmailWatcher(config, channelLookup);
|
|
await watcher.connect();
|
|
|
|
// Watch should not be called
|
|
const mockWatch = vi.mocked(google.gmail().users.watch);
|
|
expect(mockWatch).not.toHaveBeenCalled();
|
|
|
|
await watcher.disconnect();
|
|
});
|
|
|
|
it('should log push disabled message when disable_push is true', async () => {
|
|
const consoleSpy = vi.spyOn(console, 'log').mockImplementation(() => {});
|
|
|
|
const config = {
|
|
...baseConfig,
|
|
pubsub_topic: 'gmail-push',
|
|
disable_push: true,
|
|
};
|
|
|
|
const watcher = new GmailWatcher(config, channelLookup);
|
|
await watcher.connect();
|
|
|
|
expect(consoleSpy).toHaveBeenCalledWith(
|
|
'GmailWatcher: Push notifications disabled (disable_push: true)'
|
|
);
|
|
|
|
await watcher.disconnect();
|
|
consoleSpy.mockRestore();
|
|
});
|
|
});
|
|
|
|
describe('GmailWatcher - buildWatchErrorHint', () => {
|
|
it('should return topic hint for Invalid topicName error', () => {
|
|
const watcher = new GmailWatcher(baseConfig, channelLookup);
|
|
const hint = watcher['buildWatchErrorHint']('Invalid topicName');
|
|
|
|
expect(hint).toContain('Set automation.gmail.pubsub_topic');
|
|
expect(hint).toContain('projects/test-project/topics/gmail-push');
|
|
});
|
|
|
|
it('should return IAM hint for Permission denied error', () => {
|
|
const watcher = new GmailWatcher(baseConfig, channelLookup);
|
|
const hint = watcher['buildWatchErrorHint']('Permission denied');
|
|
|
|
expect(hint).toContain('gcloud pubsub topics add-iam-policy-binding');
|
|
expect(hint).toContain('gmail-api-push@system.gserviceaccount.com');
|
|
});
|
|
|
|
it('should return creation hint for Not found error', () => {
|
|
const watcher = new GmailWatcher(baseConfig, channelLookup);
|
|
const hint = watcher['buildWatchErrorHint']('Not found');
|
|
|
|
expect(hint).toContain('gcloud pubsub topics create');
|
|
});
|
|
});
|
|
|
|
describe('GmailWatcher - connection logging', () => {
|
|
it('should log all active modes on connect', async () => {
|
|
const consoleSpy = vi.spyOn(console, 'log').mockImplementation(() => {});
|
|
|
|
const config = {
|
|
...baseConfig,
|
|
pubsub_topic: 'gmail-push',
|
|
pubsub_subscription_id: 'gmail-pull',
|
|
};
|
|
|
|
const watcher = new GmailWatcher(config, channelLookup);
|
|
await watcher.connect();
|
|
|
|
expect(consoleSpy).toHaveBeenCalledWith(
|
|
expect.stringContaining('push=true, pull=true, poll_interval=')
|
|
);
|
|
|
|
await watcher.disconnect();
|
|
consoleSpy.mockRestore();
|
|
});
|
|
});
|
|
```
|
|
|
|
### 5.2 Doctor Check Tests
|
|
|
|
**File**: `src/cli/doctor.test.ts`
|
|
|
|
Add tests for enhanced Gmail validation:
|
|
|
|
```typescript
|
|
describe('doctor - checkGmail enhanced', () => {
|
|
it('should detect push mode', async () => {
|
|
const config = {
|
|
...baseConfig,
|
|
automation: {
|
|
gmail: {
|
|
enabled: true,
|
|
credentials_file: '/mock/credentials.json',
|
|
token_file: '/mock/token.json',
|
|
pubsub_topic: 'projects/test/topics/gmail-push',
|
|
output: { channel: 'telegram', peer: '123' },
|
|
},
|
|
},
|
|
};
|
|
|
|
const result = await checkGmail({ config });
|
|
expect(result.status).toBe('pass');
|
|
expect(result.detail).toContain('push + poll-fallback');
|
|
});
|
|
|
|
it('should detect pull mode', async () => {
|
|
const config = {
|
|
...baseConfig,
|
|
automation: {
|
|
gmail: {
|
|
enabled: true,
|
|
credentials_file: '/mock/credentials.json',
|
|
token_file: '/mock/token.json',
|
|
pubsub_subscription_id: 'projects/test/subscriptions/gmail-pull',
|
|
output: { channel: 'telegram', peer: '123' },
|
|
},
|
|
},
|
|
};
|
|
|
|
const result = await checkGmail({ config });
|
|
expect(result.status).toBe('pass');
|
|
expect(result.detail).toContain('pull + poll-fallback');
|
|
});
|
|
|
|
it('should detect hybrid mode', async () => {
|
|
const config = {
|
|
...baseConfig,
|
|
automation: {
|
|
gmail: {
|
|
enabled: true,
|
|
credentials_file: '/mock/credentials.json',
|
|
token_file: '/mock/token.json',
|
|
pubsub_topic: 'gmail-push',
|
|
pubsub_subscription_id: 'gmail-pull',
|
|
output: { channel: 'telegram', peer: '123' },
|
|
},
|
|
},
|
|
};
|
|
|
|
const result = await checkGmail({ config });
|
|
expect(result.status).toBe('pass');
|
|
expect(result.detail).toContain('push + pull + poll-fallback');
|
|
});
|
|
|
|
it('should warn when push requires public endpoint but tailscale_only is enabled', async () => {
|
|
const config = {
|
|
...baseConfig,
|
|
server: {
|
|
tailscale_only: true,
|
|
},
|
|
automation: {
|
|
gmail: {
|
|
enabled: true,
|
|
credentials_file: '/mock/credentials.json',
|
|
token_file: '/mock/token.json',
|
|
pubsub_topic: 'gmail-push',
|
|
output: { channel: 'telegram', peer: '123' },
|
|
},
|
|
},
|
|
};
|
|
|
|
const result = await checkGmail({ config });
|
|
expect(result.status).toBe('warn');
|
|
expect(result.detail).toContain('push requires public endpoint or Tailscale funnel');
|
|
});
|
|
|
|
it('should pass when tailscale funnel is enabled', async () => {
|
|
const config = {
|
|
...baseConfig,
|
|
server: {
|
|
tailscale_only: true,
|
|
tailscale: { funnel: true },
|
|
},
|
|
automation: {
|
|
gmail: {
|
|
enabled: true,
|
|
credentials_file: '/mock/credentials.json',
|
|
token_file: '/mock/token.json',
|
|
pubsub_topic: 'gmail-push',
|
|
output: { channel: 'telegram', peer: '123' },
|
|
},
|
|
},
|
|
};
|
|
|
|
const result = await checkGmail({ config });
|
|
expect(result.status).toBe('pass');
|
|
expect(result.detail).not.toContain('⚠️');
|
|
});
|
|
});
|
|
```
|
|
|
|
### 5.3 Integration Tests
|
|
|
|
Manual testing checklist:
|
|
|
|
- [ ] **Push only**: Set `pubsub_topic`, start daemon, send test email, verify POST arrives at `/gmail/push`, verify agent response
|
|
- [ ] **Pull only**: Set `pubsub_subscription_id`, start daemon, send test email, verify pull retrieves message, verify agent response
|
|
- [ ] **Hybrid**: Set both, start daemon, send test email, verify either push or pull handles it (no duplicate processing)
|
|
- [ ] **Polling only**: Omit both, start daemon, send test email, verify History API poll finds it
|
|
- [ ] **Disable push**: Set `disable_push: true` with `pubsub_topic`, verify watch is not called
|
|
- [ ] **Doctor check**: Run `flynn doctor` with each mode, verify output shows correct modes
|
|
- [ ] **Error messages**: Trigger watch failure (invalid topic), verify hint contains actionable guidance
|
|
- [ ] **Tailscale warning**: Set `tailscale_only: true` without funnel, verify doctor warns about push
|
|
|
|
---
|
|
|
|
## Files to Modify
|
|
|
|
### Core Implementation
|
|
|
|
| File | Changes |
|
|
|------|---------|
|
|
| `src/config/schema.ts` | Add `pubsub_subscription_id`, `pubsub_pull_interval`, `pubsub_max_messages`, `disable_push` fields |
|
|
| `src/automation/gmail.ts` | Add `setupPullSubscription()`, `pullSubscriptionMessages()`, `buildWatchErrorHint()`, update `connect()` / `disconnect()` |
|
|
| `src/cli/doctor.ts` | Enhance `checkGmail` with Pub/Sub mode detection and Tailscale reachability check |
|
|
| `package.json` | Add `@google-cloud/pubsub` dependency |
|
|
|
|
### Documentation
|
|
|
|
| File | Changes |
|
|
|------|---------|
|
|
| `README.md` | Rewrite Gmail section with deployment patterns, setup instructions, troubleshooting |
|
|
| `config/default.yaml` | Update Gmail example with new fields and comments |
|
|
| `docs/plans/state.json` | Mark `gmail-push-revisit` as completed |
|
|
|
|
### Tests
|
|
|
|
| File | Changes |
|
|
|------|---------|
|
|
| `src/automation/gmail.test.ts` | Add tests for pull mode, `disable_push`, error hints, connection logging |
|
|
| `src/cli/doctor.test.ts` | Add tests for enhanced Gmail validation (mode detection, Tailscale warning) |
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
This implementation plan makes Gmail notifications operationally reliable by:
|
|
|
|
1. **Supporting three deployment patterns**: Push (public endpoint), Pull (private), Hybrid (both)
|
|
2. **Clarifying config surface**: Explicit `pubsub_subscription_id`, `pubsub_pull_interval`, `disable_push` fields
|
|
3. **Improving error messages**: Actionable hints for common setup failures (invalid topic, missing IAM, etc.)
|
|
4. **Enhancing diagnostics**: `flynn doctor` reports active modes and warns about reachability issues
|
|
5. **Comprehensive documentation**: Setup guides for each pattern, GCP resource instructions, troubleshooting
|
|
|
|
**Recommended default**: Hybrid mode (push + pull + polling) for maximum reliability across all network configurations.
|
|
|
|
**Estimated effort**: 1-2 days (implementation + testing + docs)
|
|
|
|
**No breaking changes**: Existing `pubsub_topic`-only configs continue to work (push + polling)
|