Skip to content

Improvement loop

How a flagged call becomes a permanent product improvement.

The cycle

flowchart LR
  A[Call ends] --> B[Bridge logs + flags]
  B --> C{Any flags?}
  C -->|No| D[Move on]
  C -->|Yes| E[Pete reviews transcript + recording]
  E --> F[Diagnose: prompt issue / model / caller behaviour]
  F --> G[Propose base-prompt tweak]
  G --> H[Matt sign-off]
  H --> I[Patch base assistant via Vapi API]
  I --> J[Next call uses the new prompt]
  J --> A

Why patching the base assistant is enough

The provisioning executor (bridge/src/provision-executor.js) clones the base assistant at the moment of approval and bakes in {{business_name}}, [business], [name]. After that the clone is independent in name only — the system message, model, voice config, structured-data schema, and tools are all copied from the base verbatim.

So when we patch the base prompt:

  • All future clones inherit the patch automatically.
  • Existing clones are NOT patched — they still run on the old prompt body.

Currently we have zero live clones (the 4 test clones were torn down on 2026-06-21). Once we have live pilots, "patch the base" stops being enough — we'll need to also iterate over existing live clones and patch each via Vapi API.

The script that does this (future)

When we have ≥1 live pilot, add a bridge/scripts/patch-all-clones.js that:

  1. Reads /data/provisions.json for all live records
  2. For each, GET the assistant from Vapi
  3. Apply the same string replacement on the system message
  4. PATCH the assistant back

For now (zero live clones), patching the base via the Vapi dashboard or a single PATCH call is sufficient.

Frequency

  • Daily: Pete scans the daily digest. Zero flagged = move on. Flagged > 0 → review.
  • Weekly: End-of-week scan even if no flags, to look for regressions or new patterns.
  • Ad-hoc: Any time Matt or Katie reports a customer-facing issue.

What we've patched so far

Date Issue Fix
2026-06-21 Caller could give a 4-digit "phone number" and Polly accepted it Added validation prompt segment: "That doesn't sound quite right — could you give it to me again, slowly?" (+742 chars to base system message)
2026-06-21 (Pending — no live calls yet to flag) —