Skip to content

Commit b26ad46

Browse files
claudeclaude
committed
plan: Q2 Foundry-equivalent integration v1 — 4 phases, 28 deliverables
Q2 = user interface (Gotham/Workshop/Vertex equivalent). SMB = first tenant testbed (Steuerberater data reality check). Phase 1: demo-able (Cypher console, graph viz, property scroll, neural-debug) Phase 2: operational (actions, streaming, decision capture, orchestration DAG) Phase 3: intelligent (Blackboard NARS, FreeEnergy auto-commit, scenarios) Phase 4: fly (Arrow Flight, Ballista, GEL, Dragonfly, data lineage) Firefly stack: Ballista (distributed DataFusion) + Dragonfly (CPU lane) + GEL (graph execution language, ArenaIR → JIT → native) Session capstone epiphany logged: GEL + Firefly + Pearl 2³ = what Foundry can't do. CausalEdge64 with 8 causal masks per edge is irreducible — Vertex would need fundamental redesign to match. https://claude.ai/code/session_01SbYsmmbPf9YQuYbHZN52Zh
1 parent e5a3222 commit b26ad46

3 files changed

Lines changed: 221 additions & 0 deletions

File tree

‎.claude/board/EPIPHANIES.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2840,3 +2840,18 @@ Option C: Add a second dispatch mode (content-mode vs edge-mode) that uses Hammi
28402840
instead of PaletteSemiring for the distance function.
28412841

28422842
Cross-ref: driver.rs:75-212, Jirak calibration (this session), I-NOISE-FLOOR-JIRAK.
2843+
2844+
## 2026-04-24 — Session capstone: GEL + Firefly + Pearl 2³ = what Foundry can't do
2845+
2846+
**Status:** FINDING
2847+
**Owner scope:** @truth-architect, @integration-lead
2848+
2849+
Three-layer epiphany from the Palantir FfB Technical Overview read:
2850+
2851+
**1. Code IS Graph IS Executable.** Foundry says "treat data like code" (versioning, branching). Our 4096-row BindSpace goes further: the surface IS executable. GQL (query) → GEL (graph execution language, any program AS a graph) → ArenaIR (OOP → graph-executable transform) → JIT (Cranelift native). A class = node + typed edges. A method call = graph traversal. An if/else = conditional edge predicate. Code and data share one address space: 0x000..0xFFF.
2852+
2853+
**2. Firefly Repository = Ballista + Dragonfly + GEL.** Foundry bundles Spark + Flink. We'd bundle Ballista (distributed DataFusion) + Dragonfly (fast-path CPU lane for BindSpace sweep / Hamming / palette cascade) + GEL (the ArenaIR the 16 strategies already produce). Lance versioned dataset with CausalEdge64-annotated SPO = the Firefly Repository.
2854+
2855+
**3. NARS SPO × Pearl 2³ × CausalEdge64 — what Vertex can't do.** Foundry Vertex explores graphs but has NO causal typing on edges. Our CausalEdge64 packs Pearl 2³ = 8 causal masks (correlation / direct cause / confounder / mediator / collider / instrument / front-door / counterfactual) + NARS truth (frequency, confidence) + inference type + plasticity + temporal position into 64 bits per edge. Every SPO triple carries its own causal ontology and epistemology. This is irreducible — Vertex would need a fundamental redesign to match.
2856+
2857+
Cross-ref: FfB_Technical_Overview_v4.pdf (Palantir), CausalEdge64 (causal-edge crate), I-SUBSTRATE-MARKOV, driver.rs content Hamming cascade (PR #259), CypherBridge (PR #258).

‎.claude/board/INTEGRATION_PLANS.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,18 @@
3636

3737
---
3838

39+
## v1 — Q2 Foundry-Equivalent Integration (authored 2026-04-24)
40+
41+
**Author:** main thread (Opus 4.7 1M), session 2026-04-24
42+
**Status:** Proposed
43+
**Scope:** Q2 = user interface (Gotham/Workshop/Vertex equivalent) + SMB = first tenant testbed. 4 phases: demo-able → operational → intelligent → fly. Firefly stack: Ballista + Dragonfly + GEL.
44+
**Path:** `.claude/plans/q2-foundry-integration-v1.md`
45+
**Deliverables:** Q2-1.1..1.7 (Phase 1 MVP), Q2-2.1..2.7 (Phase 2 workflows), Q2-3.1..3.7 (Phase 3 reasoning), Q2-4.1..4.7 (Phase 4 scale). 28 deliverables total.
46+
**Foundation:** 8 PRs merged this session (#253-#260) provide substrate.
47+
**Key differentiators vs Palantir Foundry:** active inference as dispatch, NARS truth as primary data, CausalEdge64 Pearl 2³ masks, JIT-compiled lenses, zero-dep contract crate.
48+
49+
---
50+
3951
## v1 — Supabase Subscriber Wire-up (authored 2026-04-24)
4052

4153
**Author:** sonnet agent, session 2026-04-24 (branch claude/supabase-subscriber-wire-up)

‎.claude/plans/q2-foundry-integration-v1.md‎

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
# Q2 Foundry-Equivalent Integration Plan — v1
2+
3+
> **Status:** Proposed (2026-04-24)
4+
> **Owner:** @integration-lead, @truth-architect
5+
> **Scope:** Q2 = user interface for the cognitive stack; SMB = first tenant / testbed
6+
> **Depends on:** 8 PRs merged this session (#253-#260)
7+
8+
---
9+
10+
## Vision
11+
12+
Q2 is the Palantir Gotham/Workshop/Vertex equivalent: graph exploration,
13+
property scrolling, Cypher queries, live neural-debug overlay, action
14+
triggers, scenario branching. SMB (small business — Steuerberater data)
15+
is the first tenant consuming Q2, providing the reality-check testbed
16+
for the entire cognitive stack.
17+
18+
Foundry's frame: "end-to-end data operating system."
19+
Our frame: **"end-to-end cognitive operating system"** — same layers but
20+
with active inference, NARS truth, and CausalEdge64 as substrate.
21+
22+
---
23+
24+
## Architecture: Firefly Stack
25+
26+
```
27+
┌─────────────────────────────────────────────────────────────────┐
28+
│ Q2 UI (TypeScript / React / Solid) │
29+
│ Object Explorer — property scrolling (PrefetchDepth L0→L3) │
30+
│ Cypher Console — polyglot query input (GQL/Cypher/SPARQL) │
31+
│ Neural Debug Overlay — live BindSpace health + NeuronState │
32+
│ Action Panel — trigger ActionSpec (Manual/Auto/Suggested) │
33+
│ Scenario Viewer — World::fork() branches + diff │
34+
├─────────────────────────────────────────────────────────────────┤
35+
│ Transport Layer │
36+
│ JSON REST (:3001) — queries, encode, dispatch, admin │
37+
│ Arrow Flight SQL (:3002) — bulk graph traversals, streaming │
38+
│ SSE/WebSocket (:3001/subscribe) — realtime push from watcher │
39+
├─────────────────────────────────────────────────────────────────┤
40+
│ Wire Schema (serde DTOs — protocol-agnostic) │
41+
│ WireUnifiedStep → OrchestrationBridge::route() │
42+
│ WireDispatch → ShaderDriver::dispatch() │
43+
│ WireEncode → DeepNSM → BindSpace content row │
44+
│ WireBlackboard → A2A Blackboard + NARS reasoning rounds │
45+
├─────────────────────────────────────────────────────────────────┤
46+
│ Cognitive Engine (Rust) │
47+
│ BindSpace 4096 (0x000..0xFFF) — content/cycle/topic/angle │
48+
│ ShaderDriver — meta prefilter + Hamming cascade + palette │
49+
│ CypherBridge — lg.cypher → parse → SPO commit/query │
50+
│ OrchestrationBridge — nd.*/lg.* domain routing │
51+
│ RBAC Policy — PermissionSpec × PrefetchDepth × ActionSpec │
52+
├─────────────────────────────────────────────────────────────────┤
53+
│ Ontology Layer (contract crate, zero-dep) │
54+
│ PropertySchema (Required/Optional/Free × CodecRoute) │
55+
│ LinkSpec (typed edges, Cardinality) │
56+
│ ActionSpec (Manual/Auto/Suggested triggers) │
57+
│ ModelBinding + ModelHealth (NARS-based monitoring) │
58+
│ SimulationSpec (World::fork() what-if parameters) │
59+
├─────────────────────────────────────────────────────────────────┤
60+
│ Storage (Lance + AriGraph) │
61+
│ SPO triple store — CausalEdge64 per edge (Pearl 2³ masks) │
62+
│ Lance versioned datasets — World::fork() = branch │
63+
│ CAM-PQ compressed search — 6-byte fingerprints, O(1) │
64+
│ Episodic memory — ±5 Markov window, VSA trajectory │
65+
├─────────────────────────────────────────────────────────────────┤
66+
│ Firefly Repository │
67+
│ Ballista — distributed DataFusion execution │
68+
│ Dragonfly — fast-path CPU lane (BindSpace sweep, JIT kernels) │
69+
│ GEL — Graph Execution Language (ArenaIR → JIT → native) │
70+
└─────────────────────────────────────────────────────────────────┘
71+
```
72+
73+
---
74+
75+
## Deliverables — 4 Phases
76+
77+
### Phase 1: Make the stack demo-able (Q2 MVP)
78+
79+
Target: browser-accessible Cypher console + graph viz + property scroll
80+
against real SMB data. Minimum viable Gotham.
81+
82+
| D-id | Deliverable | Effort | Blocked by |
83+
|---|---|---|---|
84+
| Q2-1.1 | **Cypher Console UI** — textarea → POST /v1/shader/route → render result | small | nothing |
85+
| Q2-1.2 | **Object Explorer** — node click → GET properties at PrefetchDepth L0→L3 | small | Q2-1.1 |
86+
| Q2-1.3 | **Neural Debug Overlay** — wire neural-debug into /v1/shader/health, render NeuronState per row | medium | nothing |
87+
| Q2-1.4 | **Graph Viz** — force-directed graph from LinkSpec edges, node = entity, edge = predicate | medium | Q2-1.1 |
88+
| Q2-1.5 | **SMB seed data** — load Customer/Invoice/TaxDeclaration schemas + sample data via /v1/shader/encode | small | nothing |
89+
| Q2-1.6 | **RBAC gate on endpoints** — Policy.evaluate() middleware on Axum routes | small | nothing |
90+
| Q2-1.7 | **Cypher Phase 2** — wire real parser (add CREATE to core), SPO commit on CREATE, BindSpace search on MATCH | medium | CypherBridge PR #258 |
91+
92+
### Phase 2: Make the stack operational (Q2 workflows)
93+
94+
Target: actions, decision capture, writeback, streaming. Gotham → Workshop equivalent.
95+
96+
| D-id | Deliverable | Effort | Blocked by |
97+
|---|---|---|---|
98+
| Q2-2.1 | **Action Panel** — trigger ActionSpec from UI, POST /v1/shader/route step_type="lg.action" | medium | Q2-1.2 |
99+
| Q2-2.2 | **Decision Capture** — user corrections flow back as NARS revisions on affected triples | medium | Q2-2.1 |
100+
| Q2-2.3 | **SSE/WebSocket subscribe** — /v1/shader/subscribe endpoint wired to LanceVersionWatcher (DM-4) | medium | PR #255 (merged) |
101+
| Q2-2.4 | **Streaming property updates** — Q2 UI subscribes to entity changes, updates in-place | medium | Q2-2.3 |
102+
| Q2-2.5 | **DrainTask runtime** — DM-6 scaffold → real drain of steering_intent via OrchestrationBridge | medium | PR #255 (merged) |
103+
| Q2-2.6 | **FunctionSpec** — callable logic bound to entity types (Foundry Functions equivalent) | small | nothing |
104+
| Q2-2.7 | **Orchestration DAG** — UnifiedStep.depends_on + topological scheduler | medium | nothing |
105+
106+
### Phase 3: Make the stack intelligent (Q2 reasoning)
107+
108+
Target: Blackboard endpoint, NARS multi-expert reasoning, FreeEnergy-gated
109+
auto-commit, scenario branching. The AGI glove fully wired.
110+
111+
| D-id | Deliverable | Effort | Blocked by |
112+
|---|---|---|---|
113+
| Q2-3.1 | **Blackboard REST endpoint** — POST/GET /v1/shader/blackboard/{post,read,advance} | medium | nothing |
114+
| Q2-3.2 | **NARS reasoning rounds** — blackboard/advance triggers deduction/revision/abduction across entries | large | Q2-3.1 |
115+
| Q2-3.3 | **FreeEnergy-gated auto-commit** — dispatch + F < 0.2 → auto-commit SPO triple to AriGraph | medium | PR #259 (merged, resonance works) |
116+
| Q2-3.4 | **Scenario branching UI** — World::fork() from Q2, show base vs fork diff, SimulationSpec params | medium | Q2-1.4 |
117+
| Q2-3.5 | **Content-to-SPO pipeline** — /v1/shader/encode → dispatch → FreeEnergy → Resolution → AriGraph commit (end-to-end) | large | Q2-3.3 |
118+
| Q2-3.6 | **Scenario-aware ontology** — scenario_id column on BindSpace/SPO so rows know which branch they're in | medium | Q2-3.4 |
119+
| Q2-3.7 | **CausalEdge64 explorer** — Q2 UI shows Pearl 2³ mask, NARS truth, inference type per edge | small | Q2-1.4 |
120+
121+
### Phase 4: Make the stack fly (Q2 at scale)
122+
123+
Target: Arrow Flight for bulk queries, Ballista for distributed execution,
124+
GEL for graph-executable JIT. The Firefly Repository.
125+
126+
| D-id | Deliverable | Effort | Blocked by |
127+
|---|---|---|---|
128+
| Q2-4.1 | **Arrow Flight SQL server** — :3002, FlightSqlService impl wrapping OrchestrationBridge | large | nothing |
129+
| Q2-4.2 | **Ballista integration** — distributed DataFusion queries across nodes | large | Q2-4.1 |
130+
| Q2-4.3 | **GEL IR** — formal graph-executable language spec, ArenaIR → GEL → JIT | large | nothing |
131+
| Q2-4.4 | **Dragonfly CPU lane** — fast-path BindSpace sweep optimized for single-node hot queries | medium | nothing |
132+
| Q2-4.5 | **Data lineage** — LineageEdge tying output rows to input rows + transform step_ids | medium | Q2-2.7 |
133+
| Q2-4.6 | **Data connectors** — EntityStore impls: S3, Postgres, Mongo, DATEV (SMB-specific) | N × small | nothing |
134+
| Q2-4.7 | **Marketplace** — signed ontology bundles shareable across tenants | large | Q2-4.6 |
135+
136+
---
137+
138+
## SMB as Testbed
139+
140+
SMB (small business / Steuerberater) exercises every layer:
141+
142+
| Q2 feature | SMB reality check |
143+
|---|---|
144+
| Object Explorer | Scroll customer → see Required tax_id, Optional address, Free notes |
145+
| Cypher Console | `MATCH (c:Customer)-[:issued]->(i:Invoice) WHERE i.total > 10000` |
146+
| Actions | "Approve invoice" → ActionSpec(Manual) → NARS revise status triple |
147+
| Decision Capture | Accountant corrects OCR'd tax_id → NARS frequency 0.4 → 0.95 |
148+
| Neural Debug | "Why is this customer flagged?" → NeuronState shows which rows resonate |
149+
| RBAC | Accountant sees Detail (L1), Auditor sees Full (L3), Admin sees + writes |
150+
| Streaming | New invoice arrives → LanceVersionWatcher bumps → Q2 updates in-place |
151+
| Scenarios | "What if this customer switches to quarterly invoicing?" → World::fork() |
152+
| FreeEnergy | Missing Required property → F > 0.8 → FailureTicket → UI shows alert |
153+
| CausalEdge64 | Invoice → payment → customer satisfaction: direct cause (001) with NARS 0.7/0.9 |
154+
155+
---
156+
157+
## What we have ON TOP of Foundry (differentiators to protect)
158+
159+
1. Active inference as dispatch mechanism (not bolt-on)
160+
2. NARS truth values as primary data (not metadata)
161+
3. CausalEdge64 with Pearl 2³ masks (8 causal types per edge)
162+
4. VSA algebra on role-indexed identities (coreference via unbind)
163+
5. CAM-PQ O(1) compressed similarity
164+
6. JIT-compiled sensor lenses (10μs/sentence, no ONNX runtime)
165+
7. Zero-dep contract crate (~500 LOC, any consumer)
166+
8. Polyglot query routing (Cypher/GQL/Gremlin/SPARQL → same IR)
167+
9. 12 thinking styles as dispatch parameters
168+
10. Three-layer agent coordination (teleport / file / branch pub-sub)
169+
11. Content Hamming cascade (PR #259) — style-threshold-gated similarity
170+
12. Chapman-Kolmogorov Markov by construction (VSA bundling)
171+
172+
---
173+
174+
## Non-goals (explicit)
175+
176+
- NOT replicating Foundry's 200+ connectors. Start with 3: S3, Postgres, DATEV.
177+
- NOT building a full Workshop low-code builder. Start with Cypher console + graph viz.
178+
- NOT multi-tenant SaaS in Phase 1. Single tenant (SMB testbed). Multi-tenant in Phase 4.
179+
- NOT replacing Neo4j. Augmenting with causal typing + NARS + active inference that Neo4j can't do.
180+
181+
---
182+
183+
## Session 2026-04-24 — what shipped (foundation for this plan)
184+
185+
| PR | What | Phase it enables |
186+
|---|---|---|
187+
| #253 | Vsa16kF32 carrier + algebra | All (substrate) |
188+
| #254 | Archetype scaffold | Phase 2 (scenarios) |
189+
| #255 | Supabase subscriber (DM-4/6) | Phase 2 (streaming) |
190+
| #256 | Cycle f32 migration | All (substrate) |
191+
| #257 | SMB traits + Ontology + RBAC + encode | Phase 1 (demo) |
192+
| #258 | CypherBridge | Phase 1 (queries) |
193+
| #259 | Hamming content cascade | Phase 3 (reasoning) |
194+
| #260 | AGENT_LOG split/gitignore | Governance |

0 commit comments

Comments
 (0)