Merge pull request #238 from PostHog/tom/self-driving-jira-source

feat(self-driving): connect credential inbox sources via the secure connect link

Author: Gilbert09

context/skills/self-driving/description.md

@@ -21,7 +21,7 @@ Each step file points to the next. Run them in order. **Start by reading `refere
 - **Stay off the internal surfaces.** Don't call `signals-scout-emit-signal` or any scratchpad-write tool, and don't change a scout's `emit` flag or `run_interval_minutes` — on configs, this skill only flips `enabled`. **Built-in scout bodies are never edited.** New scout skills are created in exactly one place: step 6b, and only ones the user approved there.
 - **Keep the scout troop small.** Every enabled scout is a recurring LLM spend. Step 6 enables only `signals-scout-general` plus the **one or two** specialists for the products this project uses most — never error tracking or session replay (those reach the inbox as native sources) — and step 6b adds **at most two** custom scouts. Everything else stays disabled.
 - **Batch your questions.** `wizard_ask` has a small per-run budget; one multi-select beats four yes/nos. Don't skip a step or drop a connector (e.g. Linear) or custom scouts setup to save calls.
-- **The "too many in a row / batch your questions" error is a soft nudge, not the budget running out — retry it.** `wizard_ask` raises it once, on a call it thinks should have been batched. Your genuinely sequential asks — the Linear / GitHub-Issues connector confirms in step 5, and above all the custom-scouts proposal in step 6b — can't be batched (each depends on an earlier answer or on analysis done in between), so **re-issue the exact same call once and it goes through.** Only a `cap reached (N calls)` error means the budget is actually spent. Never record a step as a follow-up — least of all the custom scouts — just because you hit the batch nudge; that silently drops real work the user wanted.
+- **The "too many in a row / batch your questions" error is a soft nudge, not the budget running out — retry it.** `wizard_ask` raises it once, on a call it thinks should have been batched. Your genuinely sequential asks — the per-source connector confirms in step 5 (GitHub Issues, Linear, and the credential connect links for Zendesk / pganalyze / Jira), and above all the custom-scouts proposal in step 6b — can't be batched (each depends on an earlier answer or on analysis done in between), so **re-issue the exact same call once and it goes through.** Only a `cap reached (N calls)` error means the budget is actually spent. Never record a step as a follow-up — least of all the custom scouts — just because you hit the batch nudge; that silently drops real work the user wanted.
 - **Decline goes first.** Every `wizard_ask` that offers choices must include a plain-language decline option (skip / none / "keep what's there"), and it must be the **first** option so it is the default highlight — an accidental `enter` then declines instead of committing the user to something. The **one exception is step 3's GitHub gate**: the run cannot proceed without GitHub, so there the affirmative ("Done — I've installed it") stays first and the decline ("I can't connect right now", which aborts) stays last.
 
 ## Live activity — `[STATUS]`

context/skills/self-driving/references/5-connected-tools.md

@@ -6,7 +6,13 @@ next_step: 6-scouts.md
 
 External tools can feed the inbox too: GitHub Issues, Linear, Zendesk, pganalyze, and Jira. Each needs a **data warehouse source** before its signal source produces anything — a source row without the warehouse connection is dormant: harmless, but silent until the source syncs. Never enable one the user hasn't confirmed.
 
-Two of these the run can connect **itself**: GitHub Issues, and Linear via a one-click OAuth link (dedicated connector files below). The other three — Zendesk, pganalyze, and Jira — need API credentials this run never collects, so the run does **not** send the user to the UI and does **not** check whether they connected: it just arms the dormant responder and records a follow-up. The user finishes those later (a downstream reminder prompts them). (Jira's warehouse source authenticates with an Atlassian API token, not the `kind=jira` OAuth integration, so there is no one-click link to hand over — treat it exactly like Zendesk.)
+The run can connect **every** one of them, each with at most one click from the user, and it never asks anyone to paste a credential into this chat:
+
+- **GitHub Issues** — reuses the GitHub App connected in step 3 (connector: `5a-github.md`).
+- **Linear** — a one-click OAuth link (connector: `5b-linear.md`).
+- **Zendesk, pganalyze, Jira** (and any other API-credential source) — a secure PostHog **connect link**. The user enters their credentials on a PostHog page in their own browser, PostHog stores them, and the run creates the live source from that stored credential — no secret ever passes through this chat (connector: `5c-credentials.md`).
+
+A tool falls back to a **dormant responder** (the row is enabled but silent until a warehouse source exists) plus a follow-up **only** when the user skips or can't finish its connect step. That used to be the default for credential sources; it is now the exception.
 
 ## Status
 
@@ -18,7 +24,7 @@ Emit:
 
 ## Tools
 
-Load `wizard_ask` via `ToolSearch select:mcp__wizard-tools__wizard_ask`. Reach `external-data-sources-list` through the PostHog `exec` tool (`info` then `call`); the source-config tools from step 4 are reached the same way.
+Load `wizard_ask` via `ToolSearch select:mcp__wizard-tools__wizard_ask`. Reach `external-data-sources-list` through the PostHog `exec` tool (`info` then `call`); the source-config tools from step 4 are reached the same way. The credential connector (`5c-credentials.md`) additionally uses `data-warehouse-source-connect-link`, `data-warehouse-stored-credentials-list`, and `external-data-sources-create` through the same `exec` tool.
 
 ## Do
 
@@ -46,7 +52,7 @@ Load `wizard_ask` via `ToolSearch select:mcp__wizard-tools__wizard_ask`. Reach `
 
    - **GitHub Issues** → read `references/5a-github.md` and follow it.
    - **Linear** → read `references/5b-linear.md` and follow it.
-   - **Zendesk / pganalyze / Jira** → this run can't create their sources (it never collects the API credentials they need — Jira wants an Atlassian subdomain, email, and API token), so **don't ask the user to connect them and don't verify**. Just enable the dormant responder (step 4 below) and record "picked but not connected" with a follow-up. A downstream reminder prompts the user to add the warehouse source later; the responder stays dormant (harmless) and starts emitting once that source syncs.
+   - **Zendesk / pganalyze / Jira** (and any other API-credential source) → read `references/5c-credentials.md` and follow it. It hands the user a secure PostHog connect link, waits for them to store their credentials in the browser, then creates the live source from that stored credential. If they skip or don't finish, it falls back to the dormant responder + follow-up (step 4 below).
 
 4. Enable the source row (step 4's write recipe) for every tool the user picked — created, verified, and picked-but-not-connected alike (a dormant row is harmless and saves a later trip):
 
@@ -58,7 +64,7 @@ Load `wizard_ask` via `ToolSearch select:mcp__wizard-tools__wizard_ask`. Reach `
 
 5. Record each picked tool's final class honestly — the report consumes these verbatim:
 
-   - **connected by this setup** — the connector flow created the source (you have its id; the first sync starts automatically)
+   - **connected by this setup** — the connector flow created the source (you have its id; the first sync starts automatically). This now includes credential sources the user connected through the `5c-credentials.md` link, not just GitHub/Linear.
    - **already connected** / **verified connected** — the source row was seen in `external-data-sources-list`
-   - **picked but not connected** — the user picked the tool but no live warehouse source exists: Zendesk / pganalyze / Jira (never connected in-run), Linear when its integration didn't land, or a GitHub Issues fallback the user skipped. **Enable the dormant responder and add a "Connect <tool>…" follow-up** — this is harmless, because a responder only emits once its warehouse source actually syncs, so a dormant row just saves the user a later trip. Record it honestly — never write that the user "confirmed connecting" and never "not used". Phrase it as "you selected <tool>, but no warehouse source was detected — the responder is enabled and stays dormant until you add the source and it starts syncing", plus the follow-up with the new-warehouse-source URL
+   - **picked but not connected** — the user picked the tool but skipped or didn't finish its connect step, so no live warehouse source exists: a connect link they didn't complete (Zendesk / pganalyze / Jira), Linear when its integration didn't land, or a GitHub Issues fallback the user skipped. **Enable the dormant responder and add a "Connect <tool>…" follow-up** — this is harmless, because a responder only emits once its warehouse source actually syncs, so a dormant row just saves the user a later trip. Record it honestly — never write that the user "confirmed connecting" and never "not used". Phrase it as "you selected <tool>, but no warehouse source was connected — the responder is enabled and stays dormant until you add the source and it starts syncing", plus the follow-up with the new-warehouse-source URL
    - **not used** — the tool was **not picked** in the connected-tools multi-select. No responder, no follow-up; record "skipped (not used)".

context/skills/self-driving/references/5c-credentials.md

@@ -0,0 +1,71 @@
+# Connector — API-credential warehouse source (Zendesk, pganalyze, Jira, …)
+
+Creates the warehouse source for a tool that authenticates with API credentials, with at most **one click** from the user and **no secret in this chat**. The only part this run can't do is the user entering their credentials — they do that on a secure PostHog **connect page** in their browser, PostHog stores them, and this run creates the live source from the stored credential. Hand over the link, wait for the user to confirm, fetch the stored credential **once**, and create the source. If they skip or nothing was stored, leave a dormant responder and move on. Never nudge or wait through retry rounds.
+
+This is the credential equivalent of `5b-linear.md` — same one-click-then-create-once posture, but the "one click" is entering credentials on the connect page instead of approving an OAuth grant.
+
+## Status
+
+Emit (substitute the tool name):
+
+```
+[STATUS] Connecting <Tool> warehouse source
+```
+
+## Tools
+
+Reach all three through the PostHog `exec` tool (`info` then `call`): `data-warehouse-source-connect-link`, `data-warehouse-stored-credentials-list`, `external-data-sources-create`.
+
+`<Type>` below is the capitalized data-warehouse source type, and `<table>` is the **one** actionable table its responder reads (sync only that one, like the Linear connector — not every table the source has):
+
+| Tool      | `<Type>`    | `<table>`  |
+| --------- | ----------- | ---------- |
+| Zendesk   | `Zendesk`   | `tickets`  |
+| pganalyze | `PgAnalyze` | `issues`   |
+| Jira      | `Jira`      | `issues`   |
+
+The connect page renders the right credential form for that kind on its own.
+
+## Do
+
+1. **Get the connect link.** Call `data-warehouse-source-connect-link` with `{ "source_type": "<Type>" }`. It returns a `connect_url` (a PostHog page in the user's project) — relay that exact URL, don't build your own.
+
+2. **Send the link.** Ask **once**, decline-first — the user enters their credentials in the browser, never here:
+
+```
+{
+  id: "<tool>-connect",
+  prompt: "One page connects <Tool>: open this link and enter your <Tool> credentials in PostHog (never paste them here) —\n\n<connect_url>\n\nThen come back.",
+  kind: "single",
+  options: [
+    { label: "Skip <Tool>", value: "skip" },
+    { label: "Done — I've entered them", value: "done" }
+  ]
+}
+```
+
+   - **skip** → record "picked but not connected" and return to step 5 (enable the dormant responder + follow-up — harmless, since it only emits once a warehouse source syncs).
+
+3. **On done, fetch the stored credential once.** Call `data-warehouse-stored-credentials-list` with `{ "source_type": "<Type>" }` and take the **newest** `credential_id`. Stored credentials are single-use and expire after 24 hours, so read it right after the user confirms.
+
+   - **Credential present** → create the source (below).
+   - **None present** (the user didn't actually store anything, or it expired) → **don't re-ask or wait** — record "picked but not connected" and return to step 5 (the dormant responder + follow-up cover it; the user can finish the connect page later). This run never nudges.
+
+4. **Create the source** with `external-data-sources-create`, passing the stored credential by reference and syncing **only** the one actionable table — never inline secrets, never every table:
+
+```json
+{
+  "source_type": "<Type>",
+  "payload": {
+    "credential_id": "<credential id>",
+    "schemas": [{ "name": "<table>", "should_sync": true, "sync_type": "full_refresh" }]
+  }
+}
+```
+
+   `full_refresh`, not incremental: inbox records get edited and closed after they're created, so an incremental append would miss the updates (the same reason the issues connectors use it). `create` validates the stored credentials, creates the source with just that one table, and consumes the credential.
+
+   - Success returns the source `id` → record "connected by this setup (source id …, first sync started)".
+   - Any failure (invalid or expired credentials, validation error, or a backend old enough that `create` doesn't yet accept `credential_id`) → don't loop the user back through the page; record "picked but not connected" and return to step 5 (dormant responder + follow-up). A failed connect never dead-ends the run.
+
+Return to step 5 (responder enabling and class recording happen there).