lossless-group
diff --git a/‎README.md‎
Lines changed: 25 additions & 6 deletions b/‎README.md‎
Lines changed: 25 additions & 6 deletions
diff --git a/‎changelog/2026-05-26_02.md‎
Lines changed: 120 additions & 0 deletions b/‎changelog/2026-05-26_02.md‎
Lines changed: 120 additions & 0 deletions
@@ -3,6 +3,20 @@
 
 **Perplexed** is an Obsidian plugin that enables AI-powered content generation with source citations using [Perplexity](https://www.perplexity.ai/), [Anthropic Claude](https://www.anthropic.com/), [Google Gemini](https://ai.google.dev/) (with Google Search grounding), and [Perplexica / Vane](https://github.com/ItzCrazyKns/Vane) (self-hosted). This plugin brings research-grade AI capabilities directly into your Obsidian workspace, allowing you to generate well-cited content for your notes.
 
+## 💼 For Venture Capital, Private Equity, and Equities-Trading Workflows
+
+Perplexed ships a set of **analyst-grade directory templates** aimed at the research deliverables a VC analyst, PE associate, equity-research analyst, or trading-desk strategist produces daily. Drop an empty file into the matching folder, run *Apply directory template to current file*, and Perplexity Deep Research returns a 6-9K-word cited analyst draft you can curate into a memo for a partner, an IC, or a portfolio review.
+
+| Workflow | Template | What it produces |
+|---|---|---|
+| **Naming the players in a category** (incumbents vs challengers vs innovators by financial stage) | `market-category-profile.md` → `concepts/Market-Categories/` | Three-tier company landscape with explicit definitions: **Incumbents** (public / late-stage private / PE-owned), **Challengers** (Series C+ scale-ups, recently public), **Innovators** (Pre-Seed through Series B). Plus Why Now / What's Happening sections covering CAGR + category-creation momentum, and an Industry Coverage section sub-grouped into Market Reports (Gartner, IDC, Forrester, ABI) / Industry Articles / Financial News (Bloomberg, FT, Pitchbook). |
+| **Authoring a market map** (Known Category or Thesis-Driven) | `market-map-profile.md` → `lost-in-public/market-maps/` | Analyst-grade memo with 4-8 sub-segments, 20-40 named innovator cards (Offering / Funding / Why-they-matter / Coverage), Market Dynamics (Sizing / Adoption / Capital Flow), Frontier and Open Questions. Anti-incumbent editorial stance prevents big-tech over-representation. |
+| **Profiling an open spec or standard** an investment thesis depends on | `standards-and-specs-profile.md` → `Sources/Standards-and-Specs/` | Five-way authority typing (de-jure / consortium / vendor-led-open / community / de-facto), three-tier adoption framing with notable holdouts, named editors, stewardship-transition stories, named public critics with their arguments. |
+| **Catalogue an authoritative source** (book, person, channel, report, conference) | `source-profile.md` → `Sources/` | Type-aware emphasis (author / publisher / cadence / methodology), Google Books URL harvesting for books, signature-work catalog. |
+| **Encyclopedia entry on a concept, pattern, or mental model** the desk repeatedly invokes | `concept-profile.md` → `concepts/` | Definition, usage, history, examples, case studies. Mermaid + LaTeX rendering discipline baked in for diagrams and formulas. |
+
+Every analyst-grade template runs on `sonar-deep-research`, ships with idle-only timeout safety (`request-timeout-ms: 0`, per-chunk idle timer at 270s) and a 24,000-token output budget (`max-tokens: 24000`) — enough for the longest analyst drafts to land without silent mid-document truncation. See [Directory Templates](#directory-templates) below for the full set and the cft-block grammar for tuning your own.
+
 ## 🎯 Key Features
 ![Perplexed UI Modal interface](https://i.imgur.com/jaZ4UfS.png)
 - **Source-Cited AI Responses**: Get AI-generated content with proper citations and references
@@ -50,6 +64,7 @@ plugin directory.
 
 ## 📋 Table of Contents
 
+- [For Venture Capital, Private Equity, and Equities-Trading Workflows](#-for-venture-capital-private-equity-and-equities-trading-workflows)
 - [User Onboarding](#user-onboarding)
   - [Installation](#installation)
   - [Initial Setup](#initial-setup)
@@ -435,9 +450,10 @@ Two peer folders alongside `templates/` keep editorial rules vault-visible and D
 
 ```
 Content-Dev/
-├── templates/         (your four profile templates)
-├── partials/          (reusable snippets: mermaid-discipline, etc.)
-│   └── mermaid-discipline.md
+├── templates/         (your seven profile templates)
+├── partials/          (reusable snippets included via {{include: name}})
+│   ├── mermaid-discipline.md      (paired BAD/GOOD examples + 6-item self-check)
+│   └── latex-discipline.md        (Obsidian MathJax delimiters + $ escaping)
 └── preambles/         (auto-attached to every request as system / user messages)
     ├── inline-citation.md
     ├── image-placement.md
@@ -452,16 +468,19 @@ Fix the mermaid quoting rule once in `partials/mermaid-discipline.md` and every
 
 ### Shipped templates
 
-Four templates ship inlined into the plugin and seed into your vault on first plugin load:
+Seven templates ship inlined into the plugin and seed into your vault on first plugin load:
 
 | File | Targets | Use for |
 |---|---|---|
-| `concept-profile.md` | `concepts/**` | Encyclopedia-style entries on ideas, patterns, mental models. Anti-incumbent editorial stance baked in (tech giants treated as adopters/popularizers, not innovators, unless documented heyday-era origination supports otherwise). |
+| `concept-profile.md` | `concepts/**` | Encyclopedia-style entries on ideas, patterns, mental models. Anti-incumbent editorial stance baked in (tech giants treated as adopters/popularizers, not innovators, unless documented heyday-era origination supports otherwise). Includes both `mermaid-discipline` and `latex-discipline` partials. |
 | `vocabulary-profile.md` | `Vocabulary/**` | Term definitions with disambiguation through an innovation-consulting lens. |
 | `source-profile.md` | `Sources/**` | Profiles of books, people, channels, publications, journals, reports, events — type-aware, with Google Books URL harvesting for books. |
 | `toolkit-profile.md` | `Tooling/**` | Profiles of tools, products, platforms, frameworks. |
+| `market-map-profile.md` | `lost-in-public/market-maps/**`, `market-maps/**` | Analyst-grade market-map drafts — both Known Category (e.g., Humanoid Robots) and Thesis-Driven (e.g., Neural Network Hardware as Brains for Robotics). Runs on `sonar-deep-research` with idle-only timeout, `max-tokens: 24000`, and a 40-min absolute wall-clock ceiling. |
+| `standards-and-specs-profile.md` | `Sources/Standards-and-Specs/**`, `Standards-and-Specs/**` | Analyst-grade profiles of open specs and standards. Five-way authority typing (de-jure / consortium / vendor-led-open / community / de-facto). Three-tier structural adoption framing (incumbents / challengers / innovators) plus notable holdouts. Named editors, stewardship transitions, named critics. |
+| `market-category-profile.md` | `concepts/Market-Categories/**`, `Market-Categories/**` | Concept-folder reference card for a named market category. Three-tier company landscape with explicit FINANCIAL-STAGE definitions: Incumbents (public / late-stage private / PE-owned) → Challengers (Series C+ scale-ups, recently public) → Innovators (Pre-Seed through Series B). Separate Why Now / What's Happening sections covering CAGR + category-creation momentum. Industry Coverage sub-grouped into Market Reports / Industry Articles / Financial News. |
 
-All four use Perplexity's `sonar-pro` (deep-research is unreliable for image return).
+The first four templates use Perplexity's `sonar-pro`. The three deep-research templates (`market-map-profile`, `standards-and-specs-profile`, `market-category-profile`) use `sonar-deep-research` and declare per-template cft-block overrides for the wall-clock ceiling (`request-timeout-ms`), the per-chunk idle timer (`stream-idle-timeout-ms`), and the Perplexity output-token budget (`max-tokens: 24000`). See [`docs/directory-templates.md`](docs/directory-templates.md) for the full cft-block grammar and the diagnostic table for distinguishing wall-clock-timeout truncation from max_tokens truncation.
 
 ### Auto-seed behavior
 
 
@@ -0,0 +1,120 @@
+---
+title: "Long deep-research streams no longer get cut off at the ceiling — the timer now resets every time bytes arrive"
+lede: "The per-chunk idle-timeout discipline the legacy modal flow has used for two iterations now governs the directory-template flow too. A healthy slow stream completes naturally; a silently-stalled one fails in seconds instead of minutes."
+date_work_started: 2026-05-26
+date_work_completed: 2026-05-26
+date_created: 2026-05-26
+date_modified: 2026-05-26
+publish: true
+category: Changelog
+tags:
+  - Directory-Templates
+  - Streaming-Timeouts
+  - Deep-Research
+  - Idle-Timeout
+  - cft-Block
+  - Perplexity-sonar-deep-research
+  - Structural-Fix
+authors:
+  - Michael Staton
+augmented_with:
+  - Claude Code on Claude Opus 4.7 (1M context)
+files_changed:
+  - src/services/directoryTemplateService.ts
+  - docs/directory-templates.md
+  - src/docs/templates/README.md
+  - src/docs/templates/market-map-profile.md
+  - context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md
+---
+
+# The structural fix the morning's market-map run earned
+
+## Why care?
+
+If you run any directory template on Perplexity — especially the long deep-research templates like `market-map-profile` — your timeouts now behave the way you'd intuit they should. **A stream that is healthily producing bytes runs as long as it needs to run.** A stream that goes silent (Perplexity rate-limit, socket close, upstream stall) surfaces the failure inside seconds, not at the end of a 30-minute ceiling. The change is invisible when everything goes well and only noticeable when something would have gone wrong — which is when you most want it noticeable.
+
+Earlier today we shipped a market-map template, watched it produce a beautiful 7,500-word draft on Humanoid Robots, and then watched the last sentence cut off mid-word because the wall-clock timer fired before the stream finished. We patched the immediate pain by raising the ceiling. **This entry is the structural fix underneath it.**
+
+## What's new?
+
+A single concept: the timer that decides "is this stream still alive?" is now armed *per chunk*, not once per request.
+
+- **Old behavior.** One `setTimeout(controller.abort, timeoutMs)` at fetch time. If the stream takes longer than `timeoutMs`, even by one byte, abort. If the stream goes silent for 27 of the 30 allotted minutes, wait the full 30 anyway before noticing.
+- **New behavior.** Each `reader.read()` is raced against a fresh timer. As long as bytes keep arriving, the timer is cleared and re-armed. The stream is only killed if it goes **quiet** for `idleMs` — by default 270 seconds for deep-research models, 90 seconds for everything else.
+
+Two cft-block keys now govern timeouts (both optional):
+
+```cft
+provider: perplexity
+model: sonar-deep-research
+stream-idle-timeout-ms: 270000   # per-chunk idle (default already 270s for deep-research)
+request-timeout-ms: 2400000      # absolute wall-clock ceiling (40 min)
+system: |
+  ...
+```
+
+`stream-idle-timeout-ms:` is the new primary safety. `request-timeout-ms:` is the legacy key, now repurposed as an opt-in absolute ceiling — set to `0` to disable and rely on idle-only.
+
+## How it works
+
+Two different streaming primitives have lived in this codebase since the directory-template flow forked from the legacy `PerplexityModal` flow. The legacy modal moved to per-chunk idle-timeout discipline two iterations ago after the same problem hit users there first. The directory-template flow never received the backport — until today.
+
+The pattern, ported from `perplexityService.ts:659-668` into `streamPerplexityToFile`:
+
+```ts
+const readWithIdleTimeout = (): Promise<ReadableStreamReadResult<Uint8Array>> => {
+    let timer: number | undefined;
+    const timeout = new Promise<never>((_, reject) => {
+        timer = activeWindow.setTimeout(() => {
+            reject(new Error(`stream went idle for ${idleMs / 1000}s (likely API stall, rate limit, or socket close)`));
+        }, idleMs);
+    });
+    return Promise.race([reader.read(), timeout]).finally(() => {
+        if (timer !== undefined) activeWindow.clearTimeout(timer);
+    });
+};
+
+// inside the read loop:
+({ value, done } = await readWithIdleTimeout());
+```
+
+`Promise.race([reader.read(), timeout])` resolves with whichever wins. If a byte arrives, the `.finally()` clears the timer before it can fire and the loop continues. If the timer fires first, it rejects, the surrounding catch sets `truncated = true`, the existing cleanup pipeline flushes whatever already arrived, and the run returns with a partial-but-cited draft on disk and a Notice telling the user to re-run.
+
+The `AbortController` stays — both as the cancel mechanism for the user-initiated *Cancel* command, and as the abort target for the optional wall-clock ceiling. The two timers are complementary: idle handles "is this stream alive right now?" and ceiling handles "regardless, do not let this run forever."
+
+### The two pathologies the old design handled poorly
+
+**Shape 1 — slow but healthy stream.** Deep-research generations on long templates sustain a slow trickle of tokens for tens of minutes. The old wall-clock cap killed them at the ceiling regardless of whether they were still producing. The idle timer lets them complete as long as bytes keep arriving inside the idle window. (The ceiling is still there if you want a hard cap — it's just not what's making the moment-to-moment safety decision anymore.)
+
+**Shape 2 — silently stalled stream.** Conversely, a stream may go quiet at minute 3 (Perplexity rate-limit, socket close, upstream stall) and the old wall-clock cap wouldn't notice until minute 30. The user stared at an empty file for 27 unnecessary minutes. The idle timer surfaces the failure within `idleMs` seconds — fast feedback when something is genuinely wrong.
+
+Both shapes are real. The wall-clock pattern punished healthy-but-slow while tolerating stalled-but-silent. The idle-timeout pattern inverts both — slow-but-healthy completes; stalled-but-silent fails fast.
+
+## Migration notes
+
+**Nothing breaks for existing templates.** The `request-timeout-ms:` key is still honored where declared; it just means "absolute wall-clock ceiling" now instead of "the only timer at all." Templates that didn't declare it still inherit `settings.requestTimeoutMs` as their ceiling. `market-map-profile.md` keeps its `2400000` (40 min) value — under the new semantics it's the ceiling on top of the 270s idle timer, which is exactly what an analyst-grade deep-research budget should look like.
+
+To get **truly unbounded healthy streams** (idle-only safety, no ceiling), set `request-timeout-ms: 0` in the cft block, or set the plugin-level *Request timeout (ms)* setting to `0`. The idle timer will catch silent stalls fast either way.
+
+To override **the idle timer itself**, declare `stream-idle-timeout-ms: <number>` in the cft block. The defaults (270s deep, 90s normal) match the legacy modal flow and have been load-bearing there for two iterations, so most templates won't need to touch this.
+
+## What's deferred
+
+The [open issue](../context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md) listed four follow-ups. This entry closes items 1 and 2 (the port itself, and the dual-key naming decision). Two remain explicitly deferred:
+
+- **Cross-service audit.** The same wall-clock pathology may live in the Gemini service, the LM Studio service, and the Claude streaming flows. The idle-timeout discipline should be the house style across all of them; the audit is its own change.
+- **Settings-pane exposure of idle defaults.** Today the 270s/90s values are hardcoded in `streamPerplexityToFile` (matching `perplexityService.ts`). Exposing them as plugin settings is a follow-up if we ever need to tune them without a code change.
+
+The deep-research detection is by model-name regex (`/deep-research/i`) — the same shape `perplexityService.ts` uses. Robust enough for today's Perplexity model lineup; revisit if Perplexity ever ships a long-running model under a different naming convention.
+
+## Why this matters for the multi-stage exploration
+
+The [Multi-Stage Cooperative Claude + Perplexity with RAG exploration](../context-v/explorations/Multi-Stage-Cooperative-Claude-and-Perplexity-with-RAG.md) explicitly listed this fix as upstream of the multi-stage spec. Reason: the eventual Claude editorial pass on a 7-8K-word draft is itself a long generation that would hit the same wall-clock cliff. With per-chunk idle-timeout discipline now load-bearing in the directory-template flow, the editorial pass can reuse the same `readWithIdleTimeout` primitive when it lands. The multi-stage spec is now unblocked on this dimension.
+
+## Files touched
+
+- `src/services/directoryTemplateService.ts` — `streamPerplexityToFile` signature changed from `timeoutMs: number` to `timeouts: { idleMs: number; ceilingMs: number }`. `readWithIdleTimeout()` helper added. Callsite in `applyTemplate` resolves both keys from cft config with sensible fallbacks (idle defaults from deep-research detection, ceiling defaults from `settings.requestTimeoutMs`, explicit `0` disables ceiling).
+- `docs/directory-templates.md` — *Per-template timeout override* section rewritten as *Per-template timeout overrides* (plural), now documents both keys, the two pathologies the idle timer fixes, and the migration semantics.
+- `src/docs/templates/README.md` — cft-key list updated to mention both keys.
+- `src/docs/templates/market-map-profile.md` — inline comment on the `request-timeout-ms` declaration rewritten to explain it as the absolute ceiling on top of the 270s idle timer.
+- `context-v/issues/Wall-Clock-Timeout-Cuts-Off-Long-Deep-Research-Streams.md` — referenced as the canonical issue this closes.