Add typed, plugin-declared indexes to the ECS database#107
Merged
Conversation
Plugins now declare `indexes` alongside `components` / `archetypes`. The
runtime maintains them eagerly per mutation at the Store layer, so
`db.indexes.<name>` and `t.indexes.<name>` lookups are always fresh —
including inside the same transaction that just mutated a row. Unique
conflicts pre-check before any column mutation and surface from the
offending insert/update call, so store and index stay consistent under
rollback.
Public API additions:
- `Database.Index<C, Keys, Unique>` and `Database.Index.Handle<C, I>`
(compile-time-checked component keys; `get` exposed only on unique).
- 9th generic `IX` on `Database<...>`; 4th generic `IX` on
`Store<...>` / `ReadonlyStore<...>` / `TransactionContext<...>` /
`TransactionDeclaration<...>`. All defaults are `{}` so existing call
sites keep type-checking unchanged.
- `Store.Schema.indexes` (optional) and plugin descriptor `indexes`,
slotted between `archetypes` and `computed` in the property order.
Auto-routing:
- `db.select(include, { where })` and `db.observe.select(...)` route
through a matching index when `where` is pure equality on the index's
full key tuple and no order is given. Other shapes fall back to the
archetype scan unchanged.
Registration strictness (matches the `===` rule `combinePlugins`
already enforces for components / transactions, plus one new rule):
- Same name + different decl object → throws.
- Different name + same shape → throws (almost always an unintentional
duplicate; error names both indexes so the author can pick one).
Architecture:
- `Core` stays untouched; the Store layer owns the `IndexRegistry`
and wraps `archetype.insert` / `store.update` / `store.delete` to
maintain indexes. TransactionalStore / Observed / Reconciling only
forward the new `IX` generic; their runtime is unchanged because
mutations flow through Store methods.
- `db.indexes` is `store.indexes` by reference. Rollback in
`applyWriteOperations` goes back through the same Store methods,
so index undo is free.
Tests: 38 new index tests (66 total in the index suite); full data
package suite 2276 passing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous rebuild path walked every live entity in the database via
a generator over `queryArchetypes([])`, then called `store.read` per
entity. That ignored what archetypes are for: an index over `["email"]`
can only ever be populated by archetypes whose component set contains
`email`. Every other archetype's rows are skipped at `idx.add` time
anyway — but only after we already walked them and did a redundant
`locate` from `read`.
Now:
- `createIndexRegistry` no longer takes an `EntityIterator`. `register`
returns the new `RuntimeIndex` (or null on a benign identity-equal
re-register) so the caller can seed it. `rebuild()` is replaced by
`clear()` — the registry zeros its buckets, seeding is the store's
job.
- `createStore` adds `seedIndexFromArchetypes(idx)`. It calls
`core.queryArchetypes(idx.components)` to narrow to the archetypes
that could contribute, then reads values directly from each
archetype's dense column buffers per row. No `read`, no `locate`.
- `reset()` and `fromData()` call `indexRegistry.clear()` and then
seed each index via the same targeted walk. Each index sees only
the archetypes containing its components.
Also: dropped one gratuitous cast in `create-database.ts`
(`partialDatabase` is already `any`; the `as { indexes: unknown }`
narrowed for an assignment that didn't need narrowing).
All 2276 tests still pass; tsc -b clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the V1 `components`/`compute` split with a single `key` field
that accepts four shapes — bare column, column tuple, function, or slot
map — and an `order: { by, compare? }` for within-bucket sorting.
`get` now returns `Entity | null` (known absent), never `undefined`.
- `IndexKey<C>` constrains the `key` declaration to columns of C; the
inlined `IndexDeclarations` shape makes typo-keys a compile error.
- `findRange` accepts comparison-operator filters (`<,<=,>,>=,==,!=`)
on the bucket key, scalar or per-field.
- Auto-router dispatches a raw-equality `where` through the matching
index's `handle.find` (spies + instrumentation see the call).
- README catalogues all 12 patterns with their handle signatures.
- 92 runtime tests + 2 type-only proof files exercise every shape.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sorted indexes now append on insert and mark the bucket dirty; the first find/findRange that touches a dirty bucket pays one O(n log n) sort, subsequent reads are free until the next write. Optimizes the batched- write-then-read workload from O(k·n) to O(k + n log n) per batch of `k` inserts. Public interface unchanged. Bucket-key Map stays eager so the uniqueness pre-check fires before any column mutation and t.indexes.x.get(...) inside a transaction still sees just-written rows. - Strengthen unique-conflict tests to snapshot store + index pre-throw and assert no drift after rollback. - New database.index.performance.test.ts: nine inserts-are-linear-in-N tests across every catalogue shape, plus a "second read is free" check. - README: per-insert and read-after-write big-O tables. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Non-unique bucket payload is now Set<Entity> instead of Entity[]. Removes the arr.indexOf scan from remove/update-with-bucket-change, so a single- row write touches O(b) buckets at O(1) each — never proportional to bucket size or total entities in the index. For sorted indexes, the materialized + sorted array is cached per bucket and dropped on any write touching that bucket. First read after writes pays the O(n log n) sort once; subsequent reads slice the cached array. Red/green: three new perf tests in database.index.performance.test.ts under "Single-entity write is O(1) in bucket size" — non-unique delete, update-with-bucket-change, sorted-index delete. Each compares 4k vs 40k bucket and asserts ratio < 3×. All three failed with array-based storage, pass after the Set refactor. They stay in the suite to catch regressions (any reintroduction of arr.indexOf in the bucket hot path makes them red again). Insertion tests get warmThenMeasure to absorb JIT cold-start variance when running alongside the full repo suite; QUADRATIC_FLOOR lifted to 5× to still catch real O(n²) but tolerate noise on size-doubling tests. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
indexes(typed against components), accessible viadb.indexes.<name>andt.indexes.<name>withfind/findRange/get(the last only onunique: trueindexes).db.selectanddb.observe.selecttransparently use a matching index whenwhereis pure equality on the index's full key tuple.Architecture
The Store layer owns the
IndexRegistry:Core(store/core/) is untouched — raw column-storage data structure.Storewrapsarchetype.insert/store.update/store.deleteto maintain indexes; pre-checks unique constraints before mutating.TransactionalStore/ObservedDatabase/ReconcilingDatabaseonly forward the newIXgeneric through their types — runtime unchanged, because mutations all flow through Store methods.Databasederivesdb.indexesfromstore.indexes(same reference);extend(plugin)propagatesplugin.indexestostore.extend.Rollback works for free:
applyWriteOperationsgoes back through the same Store methods that maintain indexes, so index undo follows store undo automatically.API additions
Database.Index<C, Keys, Unique>andDatabase.Index.Handle<C, I>(getonly on unique).IXonDatabase<...>; 4th genericIXonStore<...>/ReadonlyStore<...>/TransactionContext<...>/TransactionDeclaration<...>. All defaults are{}— existing call sites keep type-checking unchanged.Store.Schema.indexes(optional) and the plugin descriptorindexesfield, slotted betweenarchetypesandcomputedin the required property order.Registration strictness
Matches the
===rulecombinePluginsalready enforces forcomponents/transactions, plus one new rule for cross-name duplicates:Test plan
tsc -bclean onpackages/datadatabase.index.test.tscovering find / findRange / get, compound + unique behavior, observe.select routing, auto-routing fallbacks, strict registration red/green, in-transaction freshness, atomic unique-conflict rollback for both insert and update)database.index.type-test.tsverifies that declaring an index with an unknown component, callinggeton a non-unique handle, passing the wrong scalar type tofind/findRange/get, or callingt.indexes.bogusare all compile-time errorsdatabase.test.ts,create-database.test.ts,create-store.test.ts,create-observed-database.test.ts, undo/redo, reconciling, and deep-extends-chain tests all still passdb.indexes.byEmail.get(...)on a sync-mode database still does the right thing under transient/committed envelope replay🤖 Generated with Claude Code