Skip to content

Commit 0250033

Browse files
devlux76devlux76
committed
feat: Implement model profile management and numeric constant refactoring
- Added ModelProfile and ModelProfileResolver for managing model metadata. - Introduced ModelDefaults for default policies related to model derivation. - Refactored numeric constants into a dedicated NumericConstants module. - Updated various vector backends (Wasm, WebGL, WebGPU) to utilize new constants. - Enhanced MemoryVectorStore and OPFSVectorStore to read vectors using FLOAT32_BYTES. - Implemented a guard script to prevent hardcoded model-derived numeric literals. - Added tests for model defaults and profile resolution to ensure correctness. - Documented execution plan for upcoming development sessions.
1 parent 7417d27 commit 0250033

22 files changed

Lines changed: 861 additions & 56 deletions

‎CORTEX-DESIGN-PLAN-TODO.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,20 @@ This plan is synthesized from:
88
- `Cortex-sketch-errata.md`
99
- Existing proof-of-concept backend files in this repo
1010

11+
## 0. Execution Synchronization (2026-03-11)
12+
13+
Canonical execution plan: `PROJECT-EXECUTION-PLAN.md`.
14+
15+
Next session highest priority (P0):
16+
1. Perform a full code pass before feature expansion.
17+
2. Remove hardcoded model-dependent numbers from implementation paths.
18+
3. Route model-derived values through a resolved `ModelProfile` contract.
19+
4. Keep strict TDD and required runtime lanes (`unit-node`, `runtime-browser`, `runtime-electron`).
20+
21+
Interpretation rule for this document:
22+
1. Architectural intent remains here.
23+
2. Execution order, command contract, and test matrix are governed by `PROJECT-EXECUTION-PLAN.md`.
24+
1125
## 1. Design
1226

1327
### 1.1 Product contract

‎Cortex-sketch-errata.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,15 @@
1+
## Execution Note (2026-03-11)
2+
3+
Canonical execution plan: `PROJECT-EXECUTION-PLAN.md`.
4+
5+
Next session highest priority (P0):
6+
1. Perform a full code pass before new feature work.
7+
2. Replace hardcoded model-dependent constants with resolved model metadata values.
8+
3. Keep strict TDD and runtime-lane validation as merge gates.
9+
10+
Errata note:
11+
1. Numeric literals in this addendum are illustrative unless explicitly sourced from model metadata.
12+
113
Errata / Addendum: Metroid NN Radius Graph
(added 11 March 2026 – integrates directly after the existing CORTEX query routing section)
214
Why this exists
315
The hierarchical routing already returns strong Metroids (medoids) + top pages.
We now add a sparse, cosine-radius nearest-neighbor graph in IndexedDB so Cortex.query() can instantly jumpstart a connected component and hand it to an Open TSP Path solver (dummy-node trick) for a coherent linear ordering instead of a random top-k list.

‎Cortex-sketch.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,16 @@
1+
## Execution Note (2026-03-11)
2+
3+
Canonical execution plan: `PROJECT-EXECUTION-PLAN.md`.
4+
5+
Next session highest priority (P0):
6+
1. Run a full code pass across implementation files.
7+
2. Remove hardcoded model-dependent constants.
8+
3. Replace them with values derived from model metadata via a `ModelProfile` layer.
9+
10+
Sketch note:
11+
1. Numeric values in this sketch are examples for architectural discussion.
12+
2. Implementation values must be model-derived or explicit runtime policy values.
13+
114
A concrete TypeScript-oriented architecture that maps each conceptual element (Page → Shelf, HIPPOCAMPUS, CORTEX, backends, etc.) into interfaces, classes, and modules, staying framework-agnostic but browser-friendly.
215

316
## Data model types

‎PROJECT-EXECUTION-PLAN.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Project Execution Plan (2026-03-11)
2+
3+
This file is the canonical carry-over plan for implementation sequencing, test gates, and next-session priority. Keep this file updated as work progresses.
4+
5+
## Next Session Highest Priority (P0)
6+
7+
Run a full code pass across the repository before new feature coding.
8+
9+
Instruction:
10+
1. Traverse all TypeScript source and tests.
11+
2. Remove hardcoded model-dependent assumptions.
12+
3. Classify every numeric constant as one of:
13+
- `model-derived`: must come from resolved model metadata.
14+
- `runtime-policy`: must come from explicit policy/config objects.
15+
4. Add or update tests first (Red), then implement (Green), then refactor.
16+
17+
Definition of done for this pass:
18+
1. No model-dependent domain constants remain in feature code.
19+
2. A `ModelProfile` contract is the single source of truth for model-derived values.
20+
3. A guard command fails CI when disallowed hardcoded literals are introduced.
21+
22+
## Non-Negotiable Rules
23+
24+
1. Strict TDD: Red -> Green -> Refactor for every slice.
25+
2. Runtime realism: browser and Electron lanes are required merge gates.
26+
3. Provider fallback policy: `webnn -> webgpu -> webgl -> wasm`.
27+
4. Numeric ownership: model-derived values from profile; policy values from declared policy objects.
28+
29+
## Execution Sequence
30+
31+
1. Lock contracts and TDD workflow.
32+
2. Lock command contract and CI lanes.
33+
3. Implement model-profile layer first:
34+
- `core/ModelProfile.ts`
35+
- `core/ModelProfileResolver.ts`
36+
- `core/ModelDefaults.ts`
37+
4. Replace hardcoded model-dependent values with `ModelProfile` lookups.
38+
5. Implement embedding runner with fallback chain and telemetry:
39+
- `embeddings/EmbeddingRunner.ts`
40+
- `embeddings/ProviderResolver.ts`
41+
- `embeddings/TransformersEmbeddingBackend.ts`
42+
- `embeddings/OrtWebglEmbeddingBackend.ts`
43+
- `embeddings/OnnxEmbeddingRunner.ts`
44+
6. Build Hippocampus ingest using profile-derived chunking/dimensions.
45+
7. Build Cortex retrieval using profile-derived routing/truncation policies.
46+
8. Add parity, integration, runtime, and benchmark gates.
47+
48+
## Required Test Matrix
49+
50+
1. `unit-node` (required on each PR):
51+
- Model profile derivation and defaults
52+
- Fallback resolver semantics
53+
- Deterministic logic helpers
54+
2. `runtime-browser` (required on each PR):
55+
- Real IndexedDB + OPFS
56+
- Real ONNX runtime provider selection
57+
3. `runtime-electron` (required on each PR):
58+
- Desktop parity for embedding/storage contracts
59+
4. `integration-vertical` (required on each PR):
60+
- ingest -> persist -> query coherence with deterministic fixtures
61+
5. `benchmark-nightly` (release gate):
62+
- latency and throughput trend tracking
63+
64+
## Command Contract
65+
66+
1. `npm run test:unit`
67+
2. `npm run test:unit -- tests/model/ModelProfileResolver.test.ts`
68+
3. `npm run test:unit -- tests/model/ModelDefaults.test.ts`
69+
4. `npm run test:unit -- tests/embeddings/ProviderResolver.test.ts`
70+
5. `npm run test:unit -- tests/embeddings/OnnxEmbeddingRunner.test.ts`
71+
6. `npm run guard:model-derived`
72+
7. `npm run test:browser`
73+
8. `npm run test:electron`
74+
9. `npm run build && npm run lint`
75+
10. `npm run test:all`
76+
11. `npm run benchmark`
77+
78+
## Known Hardcoded Hotspots To Clean First
79+
80+
1. `core/types.ts` comments with sample token/dimension values.
81+
2. `Policy.ts` sample projection dimensions.
82+
3. `Cortex-sketch.md` and `Cortex-sketch-errata.md` illustrative constants (`2048`, `768`, `128`, `0.68`, `40`, etc.).
83+
4. Any ingest/query defaults currently assumed without model metadata backing.
84+
85+
## Scope Notes
86+
87+
1. ONNX runner scope for this milestone is embeddings-first.
88+
2. `webgl` support is preserved through explicit ORT fallback adapter even when Transformers.js device mapping does not expose it directly.
89+
3. Daydreamer depth, peer exchange, and advanced merge policies remain post-vertical work.

‎Policy.ts‎

Lines changed: 108 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,114 @@
1-
export type QueryScope = "broad" | "default" | "narrow";
1+
import type { ModelProfile } from "./core/ModelProfile";
2+
3+
export type QueryScope = "broad" | "normal" | "narrow" | "default";
24

35
export interface ProjectionHead {
4-
dimIn: number; // D
5-
dimOut: number; // e.g. 64, 128
6-
bits?: number; // if you also want binary codes, e.g. 128
7-
offset: number; // where P_m starts in WASM memory
6+
dimIn: number;
7+
dimOut: number;
8+
bits?: number;
9+
// Byte offset for the projection head in a flattened projection buffer.
10+
offset: number;
811
}
912

1013
export interface RoutingPolicy {
11-
broad: ProjectionHead; // e.g. 64-d + 128-bit codes
12-
normal: ProjectionHead; // e.g. 128-d
13-
narrow: ProjectionHead; // e.g. 256-d or full D
14+
broad: ProjectionHead;
15+
normal: ProjectionHead;
16+
narrow: ProjectionHead;
17+
}
18+
19+
export interface RoutingPolicyDerivation {
20+
broadDimRatio: number;
21+
normalDimRatio: number;
22+
narrowDimRatio: number;
23+
broadHashBits: number;
24+
dimAlignment: number;
25+
minProjectionDim: number;
26+
}
27+
28+
export const DEFAULT_ROUTING_POLICY_DERIVATION: RoutingPolicyDerivation =
29+
Object.freeze({
30+
broadDimRatio: 1 / 8,
31+
normalDimRatio: 1 / 4,
32+
narrowDimRatio: 1 / 2,
33+
broadHashBits: 128,
34+
dimAlignment: 8,
35+
minProjectionDim: 8,
36+
});
37+
38+
function assertPositiveInteger(name: string, value: number): void {
39+
if (!Number.isInteger(value) || value <= 0) {
40+
throw new Error(`${name} must be a positive integer`);
41+
}
42+
}
43+
44+
function assertPositiveFinite(name: string, value: number): void {
45+
if (!Number.isFinite(value) || value <= 0) {
46+
throw new Error(`${name} must be positive and finite`);
47+
}
48+
}
49+
50+
function alignDown(value: number, alignment: number): number {
51+
return Math.floor(value / alignment) * alignment;
52+
}
53+
54+
function deriveProjectionDim(
55+
dimIn: number,
56+
ratio: number,
57+
derivation: RoutingPolicyDerivation,
58+
): number {
59+
const raw = Math.floor(dimIn * ratio);
60+
const aligned = alignDown(raw, derivation.dimAlignment);
61+
const bounded = Math.max(derivation.minProjectionDim, aligned);
62+
return Math.min(dimIn, bounded);
63+
}
64+
65+
function validateDerivation(derivation: RoutingPolicyDerivation): void {
66+
assertPositiveFinite("broadDimRatio", derivation.broadDimRatio);
67+
assertPositiveFinite("normalDimRatio", derivation.normalDimRatio);
68+
assertPositiveFinite("narrowDimRatio", derivation.narrowDimRatio);
69+
assertPositiveInteger("broadHashBits", derivation.broadHashBits);
70+
assertPositiveInteger("dimAlignment", derivation.dimAlignment);
71+
assertPositiveInteger("minProjectionDim", derivation.minProjectionDim);
72+
}
73+
74+
export function createRoutingPolicy(
75+
modelProfile: Pick<ModelProfile, "embeddingDimension">,
76+
overrides: Partial<RoutingPolicyDerivation> = {},
77+
): RoutingPolicy {
78+
assertPositiveInteger("embeddingDimension", modelProfile.embeddingDimension);
79+
80+
const derivation: RoutingPolicyDerivation = {
81+
...DEFAULT_ROUTING_POLICY_DERIVATION,
82+
...overrides,
83+
};
84+
85+
validateDerivation(derivation);
86+
87+
const dimIn = modelProfile.embeddingDimension;
88+
const broadDim = deriveProjectionDim(dimIn, derivation.broadDimRatio, derivation);
89+
const normalDim = deriveProjectionDim(dimIn, derivation.normalDimRatio, derivation);
90+
const narrowDim = deriveProjectionDim(dimIn, derivation.narrowDimRatio, derivation);
91+
92+
const broadOffset = 0;
93+
const normalOffset = broadOffset + broadDim * dimIn;
94+
const narrowOffset = normalOffset + normalDim * dimIn;
95+
96+
return {
97+
broad: {
98+
dimIn,
99+
dimOut: broadDim,
100+
bits: derivation.broadHashBits,
101+
offset: broadOffset,
102+
},
103+
normal: {
104+
dimIn,
105+
dimOut: normalDim,
106+
offset: normalOffset,
107+
},
108+
narrow: {
109+
dimIn,
110+
dimOut: narrowDim,
111+
offset: narrowOffset,
112+
},
113+
};
14114
}

‎README.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,19 @@ A neurobiologically inspired, fully on-device epistemic memory engine for autono
66

77
> "The library of dreams is not a collection of words but of things that words have touched — and how those things feel from the inside."
88
9+
## Execution Update (2026-03-11)
10+
11+
Canonical plan file: `PROJECT-EXECUTION-PLAN.md`.
12+
13+
Next session highest priority (P0):
14+
1. Run a full code pass before new feature coding.
15+
2. Remove hardcoded model-dependent assumptions.
16+
3. Ensure model-derived values come from resolved model metadata (`ModelProfile`).
17+
4. Keep strict TDD (Red -> Green -> Refactor) across all changes.
18+
19+
Docs note:
20+
1. Numeric examples in design docs are illustrative unless explicitly sourced from model metadata.
21+
922
## What is CORTEX?
1023

1124
Modern agents are great at *retrieving* facts.

0 commit comments

Comments
 (0)