feat(directory-templates): market-map analyst profile + per-template request-timeout-ms cft override

A new shipped template aimed at analyst-grade market maps (Known Category and Thesis-Driven flavors), and the timeout refactor that came out of running it against a real 7-8K-word draft and watching the wall-clock AbortController truncate the tail mid-sentence. The two ship together because the template surfaced the bug — running market-map-profile on lost-in-public/market-maps/Humanoid Robots and their Input Industries.md was the first observable instance of the wall-clock pathology — and they're both groundwork for the multi-stage Claude+Perplexity+RAG pipeline the exploration doc now sketches.

Impact: a fifth shipped template auto-seeded into Content-Dev/Templates/ on first plugin load (and re-seedable from settings), running on sonar-deep-research with return-images off by design; templates can override the plugin-level wall-clock timeout per-template via a new cft-block key; the plugin-level default jumped from 10 min to 30 min because the old default was tuned for short concept-profile runs and was cutting off any deep-research template that took its time. Existing four templates are unchanged in behavior — they don't declare the override and inherit the new 30-min default, which they all finish comfortably under.

Market-map template (src/docs/templates/market-map-profile.md):
- Heading skeleton: Market Snapshot, The Question this Map Answers, Why Now, Map of the Market, Lighthouse Examples, Innovator Profiles (Link/Offering/Funding/Why/Coverage cards + per-sub-segment summary table), Media-Voices-Coverage, Market Dynamics (Sizing, Adoption, Capital Flow), Frontier and Open Questions, Adjacent Concepts.
- Dual-flavor: the system prompt has the model pick between Known Category (Humanoid Robots, Quantum Computing) and Thesis-Driven (e.g. Neural Network Hardware as Brains for Robotics) based on title and tags.
- Anti-incumbent editorial stance copied from the concept template family — cap big tech at 1 of 5-10 in every sub-bucket, attribute innovation to startups/labs/founders.
- Declares request-timeout-ms: 2400000 (40 min) with an inline comment explaining the budget — the only shipped template that overrides the plugin default today.
- Registered in templateSeederService.ts so first-load seeding and the Re-seed button both pick it up; surfaced in src/docs/templates/README.md and docs/directory-templates.md.

Timeout refactor (src/services/directoryTemplateService.ts, main.ts):
- Per-template cft-block key request-timeout-ms: accepts number or numeric string, silently falls back on non-positive / non-numeric values, otherwise wins over the plugin-level setting for that template's runs. Resolution code sits just before the streamPerplexityToFile call so the rest of the streaming primitive is untouched.
- Plugin default directoryTemplatesRequestTimeoutMs bumped 600000 → 1800000 (10 min → 30 min). Settings-pane description rewritten to call out the override and the cost framing ("$10-$50 of analyst time per good output is worth waiting for").
- Structural fix (idle-timeout discipline ported from perplexityService.ts:659-668, where the legacy modal flow already does it correctly) is filed as its own open issue, not in scope for this commit.

Context-v writing:
- context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md — three architectures weighed (zoned single-file appends, multi-doc folder per market map, per-section cft-section blocks), an include-sources YAML schema proposed covering vault paths/globs and Chroma queries, and the "Perplexity and Claude do not overwrite each other until editing" heuristic translated into three implementation strictness levels. Recommends Option A (zoned appends) for v1.
- context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md — the Humanoid Robots truncation as motivating symptom, side-by-side diagnosis of the two timeout disciplines in the codebase (wall-clock here, per-chunk idle in the legacy modal), the partial fix shipped today, why it's only partial, the structural-fix proposal, and four pre-spec open items. Cross-linked both ways to the exploration so the issue is reachable from either entry point.

Files changed:
- src/docs/templates/market-map-profile.md (new)
- src/services/templateSeederService.ts
- src/services/directoryTemplateService.ts
- main.ts
- src/docs/templates/README.md
- docs/directory-templates.md
- context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md (new)
- context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md (new)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: mpstaton

changelog/2026-05-26_01.md

@@ -0,0 +1,119 @@
+---
+title: Wall-clock timeout cuts off long deep-research streams
+lede: "The directory-template runtime caps every stream by total wall-clock duration, but the legacy modal flow already moved to per-chunk idle-timeout discipline two iterations ago — and the discrepancy is now actively truncating analyst-grade market-map drafts mid-sentence."
+date_created: 2026-05-26
+date_modified: 2026-05-26
+authors:
+  - Michael Staton
+augmented_with:
+  - Claude Opus 4.7 (1M context)
+semantic_version: 0.0.0.1
+type: issue
+status: open
+target_repo: perplexed
+tags:
+  - Issue-Resolution
+  - Perplexed
+  - Streaming-Timeouts
+  - Deep-Research
+  - Directory-Templates
+related:
+  - "[[market-map-profile]]"
+  - "[[Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG]]"
+  - "[[Partials-And-Preambles-For-Perplexed-Templates]]"
+---
+
+# Wall-clock timeout cuts off long deep-research streams
+
+## Symptom
+
+Running `market-map-profile` on `lost-in-public/market-maps/Humanoid Robots and their Input Industries.md` produced a ~7,500-word draft that terminated mid-sentence inside the *Frontier and Open Questions* section:
+
+```
+Will the evolution of robot-as-a-service (R
+```
+
+The trailing parenthesis is the last byte written. The *Adjacent Concepts and Maps* section — the final heading in the template skeleton — never appeared. The frontmatter stamps (`cf_last_run`, `cf_last_run_model`) landed correctly, so the run did complete its `processFrontMatter` post-step; what was lost was the streamed body content that hadn't yet arrived when the `AbortController` fired.
+
+The Humanoid Robots run is the first observable instance of this specific cut-off shape, but the pattern is structural — it will reproduce on any sufficiently long deep-research generation run through the directory-template flow.
+
+## Diagnosis
+
+Two different streaming primitives live in this codebase, with two different timeout disciplines:
+
+**Directory-template flow** ([`src/services/directoryTemplateService.ts`](../../src/services/directoryTemplateService.ts) `streamPerplexityToFile`, the function `market-map-profile` and the four other shipped templates run through):
+
+```ts
+const controller = new AbortController();
+const timer = activeWindow.setTimeout(() => controller.abort(), timeoutMs);
+```
+
+A **single wall-clock `setTimeout`** is armed at the moment of fetch. After `timeoutMs` elapses — regardless of whether the stream is actively producing bytes — `controller.abort()` fires and the in-flight `reader.read()` throws. The catch sets `truncated = true` and falls through to a final flush of whatever streamed so far. The plugin-level default was `600_000` ms (10 min) before today's fix; analyst-grade deep-research runs routinely run 15-25 min, so the cut-off was inevitable on long templates.
+
+**Legacy modal flow** ([`src/services/perplexityService.ts:659`](../../src/services/perplexityService.ts), `PerplexityModal`):
+
+```ts
+const STREAM_IDLE_TIMEOUT_MS = isDeepResearch ? 270_000 : 90_000;
+const readWithIdleTimeout = (): Promise<...> => {
+    let timer: number | undefined;
+    const timeout = new Promise<never>((_, reject) => {
+        timer = window.setTimeout(() => {
+            reject(new Error(`stream went idle for ${...}s ...`));
+        }, STREAM_IDLE_TIMEOUT_MS);
+    });
+    return Promise.race([reader.read(), timeout]).finally(() => {
+        if (timer !== undefined) window.clearTimeout(timer);
+    });
+};
+```
+
+**Per-chunk idle timeout.** A fresh `setTimeout` is armed and racing each `reader.read()` call. As long as bytes keep arriving, the timer keeps getting cleared and re-armed. The stream is only killed if it goes *quiet* for 270 s (deep-research) or 90 s (normal). Total wall-clock duration is unbounded.
+
+The legacy modal moved to this pattern because the same problem hit users there first — but the directory-template flow was forked from an earlier iteration of the streaming code and never received the idle-timeout backport. The legacy `PerplexityModal` and `streamPerplexityToFile` now disagree on how to time-bound a Perplexity stream, and the directory-template flow has the strictly weaker discipline.
+
+## What we shipped today (partial fix)
+
+Not a structural fix — a pressure-relief valve. Two changes:
+
+1. **Bumped the plugin-level default** ([`main.ts:333`](../../main.ts)) from `600_000` ms (10 min) to `1_800_000` ms (30 min). The settings-pane description was updated to call out the override and the cost framing (`$10-$50 of analyst time per good output is worth waiting for`).
+2. **Added a per-template override** — `request-timeout-ms:` in the cft block. Resolution code lives in [`directoryTemplateService.ts`](../../src/services/directoryTemplateService.ts) just before the `streamPerplexityToFile` call; accepts number or numeric-string, silently falls back on non-positive / non-numeric values.
+3. **`market-map-profile.md`** declares `request-timeout-ms: 2400000` (40 min) with an inline comment explaining the budget; the other four shipped templates inherit the plugin-level 30-min default.
+4. **Docs** — [`docs/directory-templates.md`](../../docs/directory-templates.md) gained a *Per-template timeout override* section with override semantics and a "when to bump" checklist; [`src/docs/templates/README.md`](../../src/docs/templates/README.md) calls out the key in the cft-block list.
+
+This buys headroom. It does not fix the structural problem: any sufficiently long deep-research run will still hit the wall eventually. The 40-min cap is a *guess* about how long the longest reasonable market-map should take, not a property derived from the stream's actual behavior.
+
+## Why this is only a partial fix
+
+The wall-clock timeout fails in two distinct shapes that the idle-timeout pattern handles correctly:
+
+**Shape 1 — slow but healthy stream.** Deep-research generations on long templates may sustain a slow trickle of tokens for 30-45 minutes. The wall-clock cap kills them at the ceiling regardless of whether they're still producing. The idle-timeout pattern lets them complete naturally as long as some byte arrives every N seconds.
+
+**Shape 2 — silently stalled stream.** Conversely, a stream may go quiet at minute 3 (Perplexity rate-limit, socket close, upstream stall) and the wall-clock cap won't notice until minute 30. The user stares at an empty file, watching the spinner spin, for 27 unnecessary minutes. The idle-timeout pattern surfaces the failure within `STREAM_IDLE_TIMEOUT_MS` seconds — fast feedback when something is genuinely wrong.
+
+Both shapes are real. The wall-clock pattern punishes the healthy-but-slow case while tolerating the stalled-but-silent case. The idle-timeout pattern inverts both — slow-but-healthy completes; stalled-but-silent fails fast.
+
+## Proposed structural fix
+
+Port the idle-timeout pattern from [`perplexityService.ts:659-668`](../../src/services/perplexityService.ts) into `streamPerplexityToFile`. Concretely:
+
+1. Replace the single wall-clock `setTimeout(controller.abort, timeoutMs)` with a per-chunk `readWithIdleTimeout()` that wraps each `reader.read()` in a `Promise.race` against a fresh timeout.
+2. Choose idle-timeout values consistent with the existing modal flow: `270_000` ms (4.5 min) for deep-research models, `90_000` ms (1.5 min) for normal models. Detect deep-research from the resolved model name string (`/deep-research/i`), the same way `perplexityService.ts` does.
+3. Retain the cft-block override key, but rename it to `stream-idle-timeout-ms:` for accuracy. Templates that have been declaring `request-timeout-ms:` need a compatibility alias for one release; document the rename in the changelog entry.
+4. Optionally retain a generous absolute wall-clock ceiling (60 min or 120 min) as a sanity backstop — but that's belt-and-suspenders, not load-bearing. The idle timeout is doing the actual safety work.
+
+The diff is moderate — `streamPerplexityToFile` is ~140 lines today; the refactor touches roughly the first 40 of those (the timer setup and the `reader.read()` call inside the loop). The post-stream cleanup pipeline (`wrapThinkBlocks`, `processContentWithImages`, `buildSourcesFooter`) is unaffected.
+
+## Open items before this becomes a spec
+
+- [ ] Decide whether the cft-block key stays named `request-timeout-ms:` (which becomes a misnomer once idle-timeout is what's actually applied) or migrates to `stream-idle-timeout-ms:`. Compatibility-alias plan if the latter.
+- [ ] Decide whether to retain a wall-clock absolute ceiling alongside the idle timeout, and if so what the value is (60 min? 120 min? infinite with a Notice that surfaces "this run has been going for X min" past a threshold?).
+- [ ] Audit other long-stream callsites in the codebase that may have the same wall-clock pathology (Gemini service, LM Studio service, Claude service streaming flows) — the idle-timeout discipline should be the house style across all of them.
+- [ ] Confirm the model-name-based deep-research detection (`/deep-research/i` regex on the resolved model) is robust against future Perplexity model name changes; consider exposing the idle-timeout values as plugin settings so they can be tuned without a code change.
+
+## Related
+
+- [[market-map-profile]] — the template that surfaced the bug
+- [[Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG]] — the broader exploration this issue feeds findings back into; the idle-timeout refactor is listed there as an open item
+- [[Partials-And-Preambles-For-Perplexed-Templates]] — the architecture-review structure of that issue is the precedent this one follows
+- [`src/services/perplexityService.ts`](../../src/services/perplexityService.ts) lines 659-668 — the idle-timeout implementation in the legacy modal flow that we're proposing to port
+- [`src/services/directoryTemplateService.ts`](../../src/services/directoryTemplateService.ts) `streamPerplexityToFile` lines 499-635 — the wall-clock implementation in the directory-template flow that the port replaces

context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md

@@ -79,7 +79,7 @@ You can keep your own notes here without polluting the prompt.
 ### The three zones
 
 1. **Frontmatter** (between `---` lines at the top) — carries `title`, `applies-to-paths` (array of globs), and an optional `description`. The runtime uses `applies-to-paths` to match a template to a target file.
-2. **`cft` block** (a code fence with language `cft`) — YAML carrying `provider`, `model`, optional `search-recency`, `return-citations`, `return-images`, and a multi-line `system:` prompt. Everything **above** the `cft` block is dropped from the request.
+2. **`cft` block** (a code fence with language `cft`) — YAML carrying `provider`, `model`, optional `search-recency`, `return-citations`, `return-images`, optional `request-timeout-ms:` (per-template wall-clock override — see *Per-template timeout override* below), and a multi-line `system:` prompt. Everything **above** the `cft` block is dropped from the request.
 3. **Heading skeleton** — the markdown structure between the `cft` block's closing fence and the first `***` divider. This becomes the user prompt. Bullets under each heading are *instructions to the model*, not literal output. Everything **below** the first `***` is excluded from the request.
 
 ### Interpolation tokens
@@ -140,14 +140,15 @@ These let you query for stale files later (`cf_last_run` older than X), audit wh
 
 ## Shipped templates
 
-Four templates ship inlined into `main.js` (via esbuild's `.md` text loader) and are seeded into the user's vault on first plugin load.
+Five templates ship inlined into `main.js` (via esbuild's `.md` text loader) and are seeded into the user's vault on first plugin load.
 
 | File | Targets | Model | Notes |
 |---|---|---|---|
 | `concept-profile.md` | `concepts/**` | `sonar-pro` | Encyclopedia-style entries on ideas, patterns, mental models. Anti-incumbent editorial stance baked in. Switched off `sonar-deep-research` because deep-research is unreliable for image return. |
 | `vocabulary-profile.md` | `Vocabulary/**` | `sonar-pro` | Term definitions with disambiguation through an innovation-consulting lens. |
 | `source-profile.md` | `Sources/**` | `sonar-pro` | Profiles of trusted sources — books, people, channels, publications, journals, reports, events. Type-aware: the system prompt enumerates seven canonical types and the model picks one from frontmatter signals (`youtube_channel_url` → channel, `aliases` → likely book, etc.). Each section has per-type bullet shapes. |
 | `toolkit-profile.md` | `Tooling/**` | `sonar-pro` | Profiles of tools, products, platforms, frameworks. |
+| `market-map-profile.md` | `lost-in-public/market-maps/**`, `market-maps/**` | `sonar-deep-research` | Analyst-grade market-map drafts. Dual flavor: Known Category (Quantum Computing, Humanoid Robots) or Thesis-Driven (e.g., Neural Network Hardware as Brains for Robotics) traversing adjacent categories. Lean v1, single-stage. Skips image return by design — deep-research's image metadata is unreliable and market-map imagery is generated separately (Ideogram → frontmatter `banner_image` / `portrait_image` / `square_image`). Multi-stage v2 (RAG pre-flight to inject canonical Lossless tools/concepts as context + Claude editorial pass to emit `[[wikilink]]`s) is deferred — see the User Notes zone of the template for the roadmap. |
 
 `source-profile` is the trickiest because `Sources/` is genuinely heterogeneous. The solution is one template, type-conditional content. Books also trigger Google Books URL handling: frontmatter `google_books_url` is used if present, otherwise the model finds it; either way the URL is harvested into frontmatter post-generation via regex, so subsequent runs skip the search.
 
@@ -199,6 +200,34 @@ All three appear in the Obsidian command palette under `Perplexed: …`.
 
 ---
 
+## Per-template timeout override
+
+The runtime applies a wall-clock timeout to every Perplexity stream via an `AbortController` (see [`directoryTemplateService.ts`](../src/services/directoryTemplateService.ts) `streamPerplexityToFile`). The plugin-level default (*Plugin settings → Directory templates → Request timeout (ms)*) is **30 min** (1,800,000 ms) — generous because most templates are well under that ceiling, and the few that aren't (the analyst-grade deep-research templates) shouldn't be cut off in the middle of a draft worth $10-$50 of analyst time.
+
+Templates can override the plugin-level default by declaring `request-timeout-ms:` inside their `cft` block:
+
+```cft
+provider: perplexity
+model: sonar-deep-research
+request-timeout-ms: 2400000   # 40 min
+system: |
+  ...
+```
+
+Override semantics:
+
+- A numeric value or a numeric string both work; non-positive / non-numeric values are silently ignored and the plugin-level default is used.
+- The override applies only to that template's runs. Other templates continue to use the plugin-level value.
+- The runtime tolerates mid-stream truncation gracefully: a stream cut off by timeout leaves a partial draft on disk with `truncated: true` flagged in the result and a Notice telling the user to re-run.
+
+When to bump above the plugin-level default:
+
+- `sonar-deep-research` on a multi-section analyst template (market maps, in-depth research reports, sector overviews). These routinely emit 6–8K-word bodies and the tail sections (Frontier, Adjacent Concepts, Open Questions) are the first to be cut.
+- Templates with deeply nested skeleton structure and many `[IMAGE N: …]` markers — Perplexity's image-result population time stretches the wall clock.
+- Templates that declare a large `include-sources:` block (per the [multi-stage exploration](../context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md)) when that feature ships — RAG context inflation adds prompt-processing time.
+
+Market-map-profile ships with `request-timeout-ms: 2400000` (40 min) for exactly the reason above. Concept, vocabulary, source, and toolkit profiles do not declare the field and inherit the plugin-level 30-min default — they all finish comfortably under that.
+
 ## Settings
 
 `Plugin settings → Directory templates`:

context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md

@@ -330,7 +330,7 @@ Structure the article as follows:
         { name: 'image-placement', when: 'return-images' },
     ],
     directoryTemplatesFrontmatterWhitelist: ['title', 'og_description', 'tags', 'og_image'],
-    directoryTemplatesRequestTimeoutMs: 600000,
+    directoryTemplatesRequestTimeoutMs: 1800000,
 
     // Find images for selection
     findImagesMaxImages: 3
@@ -2040,9 +2040,9 @@ class PerplexedSettingTab extends PluginSettingTab {
 
         new Setting(containerEl)
             .setName('Request timeout (ms)')
-            .setDesc('Maximum time to wait for the perplexity deep research response. Default 300000 (5 min).')
+            .setDesc('Maximum wall-clock time to wait for a Perplexity response. Default 1800000 (30 min) — generous because deep-research runs on long analyst-grade templates routinely take 15-25 min and the $10-$50 of value per good output is worth waiting for. Individual templates may override this per-template via `request-timeout-ms:` in their cft block.')
             .addText(text => text
-                .setPlaceholder('300000')
+                .setPlaceholder('1800000')
                 .setValue(String(this.plugin.settings.directoryTemplatesRequestTimeoutMs))
                 .onChange(async (value: string) => {
                     const n = parseInt(value, 10);

docs/directory-templates.md

@@ -17,6 +17,7 @@ The plugin seeds this folder on first run with the four templates below. You can
 | `vocabulary-profile.md` | `Vocabulary/**` | Definitions of terms with disambiguation through an innovation-consulting lens. |
 | `source-profile.md` | `Sources/**` | Profiles of trusted sources — books, people, channels, publications, journals, reports, events. Adapts emphasis to the source's type. |
 | `toolkit-profile.md` | `Tooling/**` | Profiles of tools, products, platforms, frameworks. |
+| `market-map-profile.md` | `lost-in-public/market-maps/**`, `market-maps/**` | Analyst-grade market-map drafts — both Known Category (e.g., Humanoid Robots) and Thesis-Driven (e.g., Neural Network Hardware as Brains for Robotics). Runs on `sonar-deep-research`. Single-stage v1; multi-stage RAG + Claude-edit pass is planned. |
 
 ## How a template works
 
@@ -59,7 +60,7 @@ Anything below the `***` line is excluded from the request.
 ### The three zones
 
 1. **Frontmatter** (top, between `---` lines) — carries `title`, `applies-to-paths` (array of glob patterns), and an optional `description`. The plugin uses `applies-to-paths` to match a template to a target file.
-2. **`cft` block** (a code fence with language `cft`) — YAML config: `provider`, `model`, `search-recency`, `return-citations`, `return-images`, plus a multi-line `system:` prompt. Anything above the `cft` block is treated as documentation and dropped from the request.
+2. **`cft` block** (a code fence with language `cft`) — YAML config: `provider`, `model`, `search-recency`, `return-citations`, `return-images`, optional `request-timeout-ms:` (overrides the plugin-level *Request timeout (ms)* setting for this template — useful for long deep-research runs that need 20-40 min), plus a multi-line `system:` prompt. Anything above the `cft` block is treated as documentation and dropped from the request.
 3. **Heading skeleton** (everything between the `cft` block's closing fence and the first `***`) — the user prompt. This is the markdown structure the model fills in. Bullets under each heading are *instructions to the model*, not literal output.
 
 The `***` divider terminates the user prompt. Anything below it (the User Notes zone) is for your own scratch work and never reaches the model.