Creating Replay Vision scanners

A scanner is a standing LLM probe over session recordings. Once created and enabled, it runs on a Temporal schedule that sweeps every 5 minutes, applying its prompt to each new matching recording and recording the result as an observation (a queryable $recording_observed event). Each observation spends credits from a monthly org credit budget (1 credit = $0.01), and an observation's price depends on the scanner's model — so budget in credits, not in observation counts.

That schedule is exactly why creation needs a gut-check: a scanner with a permissive query and full sampling starts consuming quota automatically and can drain the whole month's budget within its first few sweeps. Creation itself does not check quota — that protection only kicks in at observation time, by which point the budget may already be gone.

First: is a scanner even the right thing?

A scanner is a standing watch over future recordings. If the user has specific sessions in front of them and a question about those sessions, they don't want a scanner at all — they want vision-scanners-inline-scan-create, which takes session_ids plus a prompt, saves nothing, and schedules nothing.

Use an inline scan when the sessions are already known: "what went wrong in these five recordings", "did any of yesterday's checkout sessions hit the coupon bug", anything you'd otherwise answer by creating a scanner and deleting it afterwards. It costs the same credits per session and reuses answers when the same question is asked twice, so re-asking is cheap.

Create a scanner only when the user wants recordings that haven't happened yet to be scanned automatically. If you find yourself planning to create a scanner, read its results once, and delete it, stop and run an inline scan instead — a throwaway scanner leaves a scheduled sweep running against every future recording that matches its query.

Core principle: size before you ship

Never create an enabled scanner blind. Estimate its monthly credit spend, check the remaining credit budget, and — when the projected spend is a meaningful fraction of what's left — show the user the numbers and get confirmation before creating. This is the heart of the skill; the rest is supporting detail.

The flow

Step 1: What should the scanner do?

Pick a scanner_type and write its scanner_config. Every type needs a prompt; the rest is type-specific:

Type	What it produces	scanner_config shape
monitor	Open-ended observation against a prompt (e.g. "flag rage clicks")	{"prompt": "..."}
classifier	Assigns tags from a fixed label set	{"prompt": "...", "tags": ["tag-a", "tag-b"]} — tags needs ≥1 entry; optional "multi_label": true, "allow_freeform_tags": false
scorer	Numeric score on a rubric	{"prompt": "...", "scale": {"min": 1, "max": 5, "label": "frustration"}} — min < max; label optional
summarizer	Free-text summary plus facet embeddings for search	{"prompt": "..."}; optional "length": "short" | "medium" | "long" (default "medium")

Summarizers always emit facet embeddings; there is no option to turn that off.

scanner_type is locked after creation — to change it you delete and recreate, so confirm the type is right up front, and get the scanner_config shape right (a wrong shape is a create error, not a silent default — unknown keys are rejected too).

If the user's intent makes the type and prompt obvious, just proceed — don't interrogate them.

Step 2: Which sessions?

The query is a RecordingsQuery shape that selects which recordings the scanner watches. date_from and date_to are ignored (the schedule controls time), so don't bother setting them. Narrow the query to the sessions that actually matter — by event, URL, person property, duration, etc. A narrow query is the single biggest lever on cost.

When the target is one experiment's exposed population, that's its own job — use the scanning-experiments-with-replay-vision skill, which derives this query from the experiment's exposure criteria instead of hand-building it.

sampling_rate (0..1, default 1.0) is a random downsample applied after the query matches. Lower it to trade coverage for budget.

Step 3: Size it — the gut-check (do not skip)

Before creating, run both checks and reason about them together:

1. Estimate spend — call vision-scanners-estimate-create with the proposed query, sampling_rate, and model. It returns matched_sessions_in_window, the window_days measured, estimated_observations_per_month, credits_per_observation (the price at that model), the resulting estimated_credits_per_month, and other_enabled_scanners_monthly_credits (what the org's other enabled scanners are already projected to spend).
2. Check budget — call vision-quota-retrieve for remaining and exhausted against the org's monthly credit_limit (credits, 1 credit = $0.01; null when uncapped).

Compare credits against credits — remaining is denominated in credits, not observations, so comparing it against estimated_observations_per_month understates the cost by the model's per-observation price.

Then decide:

• If estimated_credits_per_month plus other_enabled_scanners_monthly_credits comfortably fits within remaining, proceed.
• If it's a large fraction of (or exceeds) remaining, stop and tell the user the concrete numbers — e.g. "This scanner is projected to spend ~X credits/month (~N observations at C credits each), on top of ~Y credits from your other scanners; you have Z left this month." — and confirm before creating, or suggest tightening the query, lowering sampling_rate, or picking a cheaper model first.
• If the org is already exhausted, say so — a new enabled scanner won't produce anything until the budget resets, and its observations will be silently skipped.

Confirmation here is a conversation step, not an API capability — surface the trade-off and let the user choose. When the projected volume is clearly small relative to the budget, you don't need to ask.

Step 4: Create

Call vision-scanners-create. Minimal example:

{
  "name": "Rage click monitor",
  "scanner_type": "monitor",
  "scanner_config": { "prompt": "Flag sessions where the user repeatedly clicks the same element in frustration." },
  "query": { "kind": "RecordingsQuery", "events": [{ "id": "$rageclick", "type": "events" }] },
  "sampling_rate": 1.0,
  "model": "gemini-3.6-flash",
  "enabled": true
}

name must be unique within the team. Set enabled: false if the user wants to create it paused (no schedule, no quota consumption) and turn it on later.

After creation

• Show the scanner's PostHog URL from the response so the user can review it in the UI.
• Results take a few minutes to appear (rasterizing the recording to video + the LLM call are slow). Inspect them with vision-scanners-observations-list for one scanner over time, or vision-observations-list (requires session_id) for every scanner's findings on a single session. To dig into a recording, hand off to the investigating-replay skill.

Updating an existing scanner

vision-scanners-update is a partial update — send only changed fields. Re-run the Step 3 gut-check whenever you widen scope: a broader query or a higher sampling_rate raises the sweep volume just like a fresh broad scanner would. Toggling enabled, tweaking the prompt, or narrowing the query don't need a re-estimate. Editing config bumps scanner_version; past observations keep a snapshot of the old config.

Gotchas

• One observation per (scanner, session). Re-running a scanner on a session it already observed — even a failed or ineligible one — is a no-op and won't produce a fresh scan.
• Ineligible ≠ failed. Observations can land ineligible (e.g. too_short, no_recording) — a terminal non-error outcome. Check error_reason when triaging why a scanner produced nothing.
• Provider/model are Google/Gemini only in the current version.