Commit 6958d00
Add typed, plugin-declared indexes to the ECS database (#107)
* Add typed, plugin-declared indexes to the ECS database
Plugins now declare `indexes` alongside `components` / `archetypes`. The
runtime maintains them eagerly per mutation at the Store layer, so
`db.indexes.<name>` and `t.indexes.<name>` lookups are always fresh —
including inside the same transaction that just mutated a row. Unique
conflicts pre-check before any column mutation and surface from the
offending insert/update call, so store and index stay consistent under
rollback.
Public API additions:
- `Database.Index<C, Keys, Unique>` and `Database.Index.Handle<C, I>`
(compile-time-checked component keys; `get` exposed only on unique).
- 9th generic `IX` on `Database<...>`; 4th generic `IX` on
`Store<...>` / `ReadonlyStore<...>` / `TransactionContext<...>` /
`TransactionDeclaration<...>`. All defaults are `{}` so existing call
sites keep type-checking unchanged.
- `Store.Schema.indexes` (optional) and plugin descriptor `indexes`,
slotted between `archetypes` and `computed` in the property order.
Auto-routing:
- `db.select(include, { where })` and `db.observe.select(...)` route
through a matching index when `where` is pure equality on the index's
full key tuple and no order is given. Other shapes fall back to the
archetype scan unchanged.
Registration strictness (matches the `===` rule `combinePlugins`
already enforces for components / transactions, plus one new rule):
- Same name + different decl object → throws.
- Different name + same shape → throws (almost always an unintentional
duplicate; error names both indexes so the author can pick one).
Architecture:
- `Core` stays untouched; the Store layer owns the `IndexRegistry`
and wraps `archetype.insert` / `store.update` / `store.delete` to
maintain indexes. TransactionalStore / Observed / Reconciling only
forward the new `IX` generic; their runtime is unchanged because
mutations flow through Store methods.
- `db.indexes` is `store.indexes` by reference. Rollback in
`applyWriteOperations` goes back through the same Store methods,
so index undo is free.
Tests: 38 new index tests (66 total in the index suite); full data
package suite 2276 passing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Seed each index from its own archetypes, not every entity
The previous rebuild path walked every live entity in the database via
a generator over `queryArchetypes([])`, then called `store.read` per
entity. That ignored what archetypes are for: an index over `["email"]`
can only ever be populated by archetypes whose component set contains
`email`. Every other archetype's rows are skipped at `idx.add` time
anyway — but only after we already walked them and did a redundant
`locate` from `read`.
Now:
- `createIndexRegistry` no longer takes an `EntityIterator`. `register`
returns the new `RuntimeIndex` (or null on a benign identity-equal
re-register) so the caller can seed it. `rebuild()` is replaced by
`clear()` — the registry zeros its buckets, seeding is the store's
job.
- `createStore` adds `seedIndexFromArchetypes(idx)`. It calls
`core.queryArchetypes(idx.components)` to narrow to the archetypes
that could contribute, then reads values directly from each
archetype's dense column buffers per row. No `read`, no `locate`.
- `reset()` and `fromData()` call `indexRegistry.clear()` and then
seed each index via the same targeted walk. Each index sees only
the archetypes containing its components.
Also: dropped one gratuitous cast in `create-database.ts`
(`partialDatabase` is already `any`; the `as { indexes: unknown }`
narrowed for an assignment that didn't need narrowing).
All 2276 tests still pass; tsc -b clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Add computed, multi-value, and sorted indexes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Redesign index declaration API around a unified `key` union
Replaces the V1 `components`/`compute` split with a single `key` field
that accepts four shapes — bare column, column tuple, function, or slot
map — and an `order: { by, compare? }` for within-bucket sorting.
`get` now returns `Entity | null` (known absent), never `undefined`.
- `IndexKey<C>` constrains the `key` declaration to columns of C; the
inlined `IndexDeclarations` shape makes typo-keys a compile error.
- `findRange` accepts comparison-operator filters (`<,<=,>,>=,==,!=`)
on the bucket key, scalar or per-field.
- Auto-router dispatches a raw-equality `where` through the matching
index's `handle.find` (spies + instrumentation see the call).
- README catalogues all 12 patterns with their handle signatures.
- 92 runtime tests + 2 type-only proof files exercise every shape.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Defer within-bucket sorting to first read after writes
Sorted indexes now append on insert and mark the bucket dirty; the first
find/findRange that touches a dirty bucket pays one O(n log n) sort,
subsequent reads are free until the next write. Optimizes the batched-
write-then-read workload from O(k·n) to O(k + n log n) per batch of `k`
inserts. Public interface unchanged.
Bucket-key Map stays eager so the uniqueness pre-check fires before any
column mutation and t.indexes.x.get(...) inside a transaction still sees
just-written rows.
- Strengthen unique-conflict tests to snapshot store + index pre-throw and
assert no drift after rollback.
- New database.index.performance.test.ts: nine inserts-are-linear-in-N
tests across every catalogue shape, plus a "second read is free" check.
- README: per-insert and read-after-write big-O tables.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Make single-entity index writes O(1) in bucket size
Non-unique bucket payload is now Set<Entity> instead of Entity[]. Removes
the arr.indexOf scan from remove/update-with-bucket-change, so a single-
row write touches O(b) buckets at O(1) each — never proportional to
bucket size or total entities in the index.
For sorted indexes, the materialized + sorted array is cached per bucket
and dropped on any write touching that bucket. First read after writes
pays the O(n log n) sort once; subsequent reads slice the cached array.
Red/green: three new perf tests in database.index.performance.test.ts
under "Single-entity write is O(1) in bucket size" — non-unique delete,
update-with-bucket-change, sorted-index delete. Each compares 4k vs 40k
bucket and asserts ratio < 3×. All three failed with array-based storage,
pass after the Set refactor. They stay in the suite to catch regressions
(any reintroduction of arr.indexOf in the bucket hot path makes them red
again).
Insertion tests get warmThenMeasure to absorb JIT cold-start variance
when running alongside the full repo suite; QUADRATIC_FLOOR lifted to 5×
to still catch real O(n²) but tolerate noise on size-doubling tests.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>1 parent 851f7cf commit 6958d00
18 files changed
Lines changed: 3753 additions & 57 deletions
File tree
- packages/data/src
- ecs
- database
- index-registry
- public
- transactional-store
- store
- public
- table
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
135 | 135 | | |
136 | 136 | | |
137 | 137 | | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
138 | 335 | | |
139 | 336 | | |
140 | 337 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
| 29 | + | |
| 30 | + | |
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
35 | | - | |
| 36 | + | |
36 | 37 | | |
37 | | - | |
| 38 | + | |
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
| |||
48 | 49 | | |
49 | 50 | | |
50 | 51 | | |
51 | | - | |
| 52 | + | |
| 53 | + | |
52 | 54 | | |
53 | | - | |
| 55 | + | |
54 | 56 | | |
55 | 57 | | |
56 | 58 | | |
| |||
70 | 72 | | |
71 | 73 | | |
72 | 74 | | |
73 | | - | |
| 75 | + | |
74 | 76 | | |
75 | 77 | | |
76 | 78 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| 14 | + | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
| |||
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
23 | | - | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| |||
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
32 | | - | |
| 33 | + | |
| 34 | + | |
33 | 35 | | |
34 | 36 | | |
35 | 37 | | |
| |||
49 | 51 | | |
50 | 52 | | |
51 | 53 | | |
52 | | - | |
| 54 | + | |
53 | 55 | | |
54 | 56 | | |
55 | 57 | | |
| |||
84 | 86 | | |
85 | 87 | | |
86 | 88 | | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
92 | 95 | | |
93 | 96 | | |
94 | 97 | | |
| |||
99 | 102 | | |
100 | 103 | | |
101 | 104 | | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
106 | 110 | | |
107 | 111 | | |
108 | 112 | | |
| |||
131 | 135 | | |
132 | 136 | | |
133 | 137 | | |
134 | | - | |
| 138 | + | |
135 | 139 | | |
136 | 140 | | |
137 | 141 | | |
138 | | - | |
| 142 | + | |
| 143 | + | |
139 | 144 | | |
140 | 145 | | |
141 | 146 | | |
| |||
149 | 154 | | |
150 | 155 | | |
151 | 156 | | |
| 157 | + | |
152 | 158 | | |
153 | 159 | | |
154 | 160 | | |
| |||
160 | 166 | | |
161 | 167 | | |
162 | 168 | | |
163 | | - | |
| 169 | + | |
| 170 | + | |
164 | 171 | | |
165 | 172 | | |
166 | 173 | | |
| |||
172 | 179 | | |
173 | 180 | | |
174 | 181 | | |
175 | | - | |
| 182 | + | |
| 183 | + | |
176 | 184 | | |
177 | 185 | | |
178 | 186 | | |
| |||
189 | 197 | | |
190 | 198 | | |
191 | 199 | | |
192 | | - | |
| 200 | + | |
193 | 201 | | |
194 | 202 | | |
195 | 203 | | |
| |||
198 | 206 | | |
199 | 207 | | |
200 | 208 | | |
| 209 | + | |
201 | 210 | | |
202 | 211 | | |
203 | 212 | | |
| |||
0 commit comments