Reply Pattern Tracking & DSL Filtering

1Why lifecycle events (and the shipped precedent)

Every reply path in prod already lands on "ActionableInsightsLifecycleEvent":

Because each reply is its own event row with its own actionReviewId, the multi-replies-per-action property falls out for free: reply 1 on an action can carry pattern X, reply 2 pattern Y, each independently queryable. A dedicated ReplyPatternUsage join table would duplicate what the event log already models.

2Prod scale measurements (readonly TimescaleDB)

• The table is a plain Postgres table, not a hypertable (only 9 prod tables are hypertables — snapshots, notifications, LLM usage; no action tables).
• Volume dominated by STATUS_CHANGED (1.8M), TAG_ADDED (1.3M). Reply events are tiny: 296 DRAFT_REPLY_CREATED, 706 AI_REPLY_DRAFT_ACCEPTED, 17 RUN_MACRO.
• Biggest org (Lyft) holds 1.69M events.
• Pattern-style jsonb lookup without an index = parallel seq scan, 360 ms today → ~30s+ at 100x. The partial index is a requirement, not an optimization.

3The event contract

GOAL AGENT PATH                          INTERACTIVE PATH (composer)
───────────────                          ──────────────────────────
agent drafts reply                       human clicks "generate a reply"
  │                                        │  (Mastra stream relayed via
  │                                        │   apps/web .../api/ai/agent/route.ts)
  ▼                                        ▼
DRAFT_REPLY_CREATED / RUN_MACRO          DRAFT_REPLY_CREATED           ← NEW emission
  metadata.patternIds: [...]               metadata.patternIds: [...]
  metadata.actionReviewId                  metadata.source: "composer_generate"
  │                                        │  (emitted SERVER-SIDE at
  │                                        │   generation-complete)
  ▼                                        ▼
patternIds snapshotted onto              generate response returns patternIds
ActionReview.metadata                    → client holds them with the draft
  │                                        │
  ▼                                        ▼
human approves review                    human sends (input.aiReplyDraft
  │                                        echoes patternIds back)
  ▼                                        ▼
AI_REPLY_DRAFT_ACCEPTED                  AI_REPLY_DRAFT_ACCEPTED
  metadata.patternIds  (from review)       metadata.patternIds  (from echo)
  metadata.wasEdited                       metadata.wasEdited

Rules

1. Creation events are the primary store. They are the only complete usage log: they cover rejected drafts, abandoned/regenerated drafts, and goal-agent auto-sends (no human in the loop → ACCEPTED never fires there).
2. ACCEPTED also carries patternIds so "accepted replies that used pattern X" needs no event-to-event join. It's nearly free: the review path already snapshots draft info onto ActionReview metadata (that's how aiReplyDraftSource reaches the approval handler), and the composer path already sends {source, originalAiDraft} from the client — one more field.
3. Never infer the link by recency. ACCEPTED must not "read the last DRAFT_REPLY_CREATED" — with regenerations, multiple drafters, or agent+human drafts coexisting, "last" mis-attributes silently. The correlation channel is explicit: review metadata (review path) or the UI echo (composer path). The UI is a courier for server-issued values, same as originalAiDraft today.
4. patternIds is an array — one reply can draw on several patterns.
5. Always set orgId on new events (see §6 — 28% of 2026 writes still omit it and silently vanish from org-scoped filters).

Why emit DRAFT_REPLY_CREATED on the interactive path (new write)

• symmetry One query surface: "where was pattern X used" is a single DSL filter regardless of who drafted.
• negative signal Generated-then-abandoned / regenerated drafts are the most useful data for evaluating patterns — the pattern fired and a human implicitly rejected the result. Only a creation event captures that.
• volume Composer generations ≈ thousands/month vs 650k/mo total events. Rounding error.
• Emit server-side at generation-complete in the web relay route — creation events are never client-trusted.
• Reuse the existing DRAFT_REPLY_CREATED enum value, distinguish with metadata.source: "composer_generate" (exactly how ACCEPTED distinguishes sources — avoids an enum migration).
• Exclude from the case-timeline UI so regenerate clicks aren't visible noise. Precedent exists: AI_REPLY_DRAFT_ACCEPTED is already excluded with type != … filters in lifecycle-events-search-field-planner.ts and list-actions.ts.

4The index

CREATE INDEX idx_lifecycle_pattern_usage
ON "ActionableInsightsLifecycleEvent"
USING gin ((metadata->'patternIds') jsonb_path_ops)
WHERE metadata ? 'patternIds';

• GIN + jsonb_path_ops because patternIds is an array → containment queries (metadata->'patternIds' @> '"<patternId>"').
• Partial (WHERE metadata ? 'patternIds') → contains only pattern-stamped rows. Stays kB-sized like the macro index regardless of total table growth.
• One index covers both event types (creation + ACCEPTED) since they share the metadata key.

5DSL wiring — two surfaces, both with shipped precedents

5a. pattern: filter on the actions/messages row sources

"Find all cases where a reply used pattern X" — the main filter.

1. Register the field in packages/core/filter-registry/src/dsl-field-registry.ts (canonical name, aliases, operators, rowSources).
2. Add a closure in packages/data/timescale-db/src/models/dashboard/field-sql-closures.ts, registered in FILTER_SQL_REGISTRY, emitting an EXISTS subquery against the lifecycle log correlated on the action id — exactly how tag:, theme:, and caseType: already filter via separate tables.
3. Add the filterType to the FilterClause union in dsl-query-dispatcher.ts. It's auto-picked-up by tryRegistryDispatch.
4. Boot-time validators (field-registries-validate.ts, registry-enforcement.test.ts) will fail if the field is declared without its closure wiring — run them.
5. Agent-facing docs/allowlists: packages/core/agents/skills/sift-search/references/row-sources.md, packages/core/agents/DSL.md.

-- shape the closure emits (schematic)
EXISTS (
  SELECT 1 FROM "ActionableInsightsLifecycleEvent" e
  WHERE e."actionableInsightsThreadId" = ait.id
    AND e."orgId" = $org
    AND e.metadata->'patternIds' @> $patternIdJson
    -- optional stricter variant: AND e.type = 'AI_REPLY_DRAFT_ACCEPTED'
)

5b. patternId filter on the existing lifecycle_events catalog source

Per-reply granularity — "show each reply and which pattern made it". Also what lets the agent itself search past cases by pattern (the similar-case retrieval story).

• The planner already exists: lifecycle-events-search-field-planner.ts (already supports goalId, eventType, workflowId). Adding patternId mirrors those.
• Declare it sqlStrategy: "savedSearchOnly" with no rowSources, like the other audit fields (registry lines ~567-629) so the public parser accepts it without the aggregate DSL trying to emit SQL for it.

6Hypertable verdict & the real scale work

What actually threatens the table at Uber scale, in order:

1. NULL orgId — 2.29M rows (40%) have no orgId, and it's not legacy: 28% of 2026 writes still omit it (the CREATED / STATUS_CHANGED / QUEUE_CHANGED / ASSIGNED writers). Every org-scoped index and DSL filter silently misses those rows. Fix the writers (+ backfill — orgId is derivable from the thread).
2. Index diet — 16 indexes = 16 index writes per insert at a future 70M inserts/month. Near-dead today: idx_lifecycle_event_workflow_execution (48 MB, 0 scans), idx_lifecycle_event_goal_type_created (375 MB, 124 scans), idx_lifecycle_assignee_org_created (324 MB, 135 scans), bare orgId index (56 MB, 12 scans). ~800 MB reclaimable.

Generalizing lifecycle-event filtering: this recipe (metadata key + partial index + EXISTS closure) holds for any selective event class. It is the wrong substrate for filters that would match most events (e.g. status-history filters at Uber scale) — those stay as materialized columns on the thread / ActionMetrics, like today.

7Implementation checklist

Independent follow-ups (not blockers, pre-Uber hygiene): fix NULL-orgId writers + backfill; drop the near-dead indexes.

8Open decisions

• Emit per generate click vs per materialized draft? Recommendation: per click — that's where the negative signal lives, volume is trivial. Just make it deliberate.
• Store pattern slugs alongside IDs in metadata? Slugs make events self-describing in timelines/exports even if a pattern is renamed/deleted; costs a few bytes. Recommendation: store both, filter by ID.
• Does the DSL pattern: filter default to any-usage or accepted-only? Recommendation: any-usage by default, with the accepted-only variant expressed via the lifecycle_events source (type + wasEdited) until there's demand for a dedicated operator.