Skip to content

Commit 3407966

Browse files
authored
feat(ontology): runtime reader — OntologyRuntime + EntityResolver + concept-index lookup (#58 reader) (#154)
* feat(ontology): runtime reader — OntologyRuntime + EntityResolver + concept-index lookup (#58 reader) Reader follow-on to PR #92's concept-index emission. Closes the last feature dependency for issue #107's four-guarantee notebook (the resolve beat). Public surface in ``bigquery_agent_analytics``: * OntologyRuntime — frozen dataclass; loads ontology + binding (from YAML files via from_files or in-memory via from_models) plus an optional ConceptIndexLookup wired to the emitted BigQuery table. Read-only accessors over entities / relationships / synonyms / annotations / SKOS schemes / notations / labels; provenance properties (compile_fingerprint / compile_id) computed locally via the same bigquery_ontology._fingerprint helpers PR #92's emission uses. * EntityResolver Protocol + two reference impls: - ExactEntityResolver: in-memory match on entity_name, no BigQuery roundtrip; at most one candidate per query. - LabelSynonymResolver: BQ-backed match against concept-index label / synonym / notation rows; re-ranks candidates by label-kind priority (name > pref > alt > hidden > synonym > notation). NO embedding / LLM / fuzzy in this slice — explicit non-goals; future PRs can implement the Protocol without touching the runtime surface. * ConceptIndexLookup — BigQuery-backed accessor that is fingerprint-strict. Three trust points: 1. Eager verify() at construction. The runtime computes expected_fp = compile_fingerprint(fingerprint_model( ontology), fingerprint_model(binding), compiler_version) locally and compares against the table's __meta row. Mismatch raises FingerprintMismatchError before from_files / from_models returns. 2. Public verify() method for explicit re-checks before long batches. 3. Per-query "WHERE compile_fingerprint = @expected_fp" defense in depth. Even if the table is swapped or partially corrupted between verify and query, stale rows can't surface in the result. Three lookup methods: lookup_by_label (with label_kind / language / case_insensitive filters), lookup_by_entity_name, lookup_by_notation. Returns ConceptIndexRowView dataclass matching PR #92's main-table schema 1:1. Stable failure codes — all subclass ConceptIndexError: * FingerprintMismatchError — __meta row's compile_fingerprint differs from locally-computed value. * MetaTableMissingError — __meta sibling table doesn't exist / query failed. * MetaTableEmptyError — __meta exists but zero rows (PR #92 always writes exactly one). Same trust discipline as Phase C compiled extractors — stale provenance never produces a confident match. Tests: * tests/test_ontology_runtime.py: 39 CI cases using in-memory fake BQ clients. Layered coverage: TestOntologyRuntimeConstruction (5), TestOntologyRuntimeAccessors (10), TestConceptIndexLookupVerify (4), TestConceptIndexLookupQueries (8), TestExactEntityResolver (5), TestLabelSynonymResolver (5), TestEntityResolverProtocol (2). * tests/test_ontology_runtime_live.py: gated behind BQAA_RUN_LIVE_TESTS=1 + BQAA_RUN_LIVE_ONTOLOGY_RUNTIME_TESTS=1 + PROJECT_ID + DATASET_ID. Compiles a tiny ontology to concept-index SQL via PR #92's emission path, executes the DDL to create real BQ tables, attaches the runtime, runs LabelSynonymResolver + lookup_by_notation, asserts every candidate carries the runtime's compile_fingerprint, drops the tables on the way out. Docs: docs/ontology_runtime_reader.md describes the contract, trust gates, failure codes, public API. CHANGELOG + docs/README.md index entries added. * fix(ontology-runtime): 5 reviewer findings + live BQ validation Addresses PR #154 round-1 reviewer findings. P1 #1 — table_id injection guard. ConceptIndexLookup.__init__ used to interpolate table_id directly into backtick-quoted SQL in verify() and _run_lookup() without shape validation. New _TABLE_ID_PATTERN regex (same discipline as BigQueryBundleStore from Phase C): exactly three ASCII segments, [A-Za-z0-9_-]+, fullmatch (not match — trailing newline rejected). Backtick, semicolon, whitespace, comment markers, wrong dot count, and non-string all raise ValueError at construction. Validation flows through OntologyRuntime.from_models so callers who never construct ConceptIndexLookup directly still get the protection. P1 #2 — drop unsafe singular relationship(name) accessor; add traversal helpers. Per #58's traversal-first contract (entity_resolution_primitives.md:118), relationship names are NOT unique once #62's relaxed (name, from, to) uniqueness ships — traversal-style names like skos_broader legally repeat across endpoint pairs. The singular relationship(name) -> Relationship | None accessor would silently return the first match and hide duplicates; dropped entirely. New relationships_by_name(name) -> tuple[ Relationship, ...] returns every match. Plus four SKOS traversal helpers the user asked for: * in_scheme(scheme) -> tuple of entity names (forward map). * broader(entity_name) -> direct skos:broader parents. * narrower(entity_name) -> inverse (children that declare this as broader). Not transitive. * related(entity_name) -> skos:related values (not auto-symmetrized). Companion notations_for(entity_name) returns every notation value (scalar or list), companion to notation_for which returns only the first. P2 #1 — verify() always re-queries. Dropped the _verified cache flag. The docstring promised "re-check before long batches" but the cached fast path made re-runs no-ops. Now every verify() call hits BigQuery again so a table swap or fingerprint update mid-flight is caught. Construction-time eager verify still runs once; subsequent verify() calls all re-query. P2 #2 — reject multiple __meta rows. PR #92's emission writes exactly one meta row per table; multiple rows indicate manual tampering. verify() SQL now uses LIMIT 2 (not LIMIT 1) so a multi-row case is detectable without a full scan; len(rows) != 1 fails closed with new MetaTableMultipleRowsError (subclasses ConceptIndexError). The empty case still uses MetaTableEmptyError; multi is a distinct error code so callers can switch on the failure mode. P2 #3 — labels_for() now emits skos:notation as label_kind='notation', matching PR #92's six-kind vocabulary (name / pref / alt / hidden / synonym / notation). The previous omission meant a caller comparing in-memory labels against emitted concept-index rows would see five kinds locally vs six remotely. Tests: 39 -> 50 cases. New TestRoundOneFindings (11 cases) locks each finding with explicit reproducers: * test_lookup_rejects_malformed_table_id — every injection pattern + valid form acceptance. * test_runtime_propagates_table_id_validation — validation flows through OntologyRuntime factories. * test_relationships_by_name_returns_every_match — builds duplicate-name relationships directly via the Pydantic models (load_ontology enforces pre-#62 uniqueness, so the test bypasses the YAML path). * test_singular_relationship_accessor_is_dropped — hasattr(runtime, "relationship") is False. * test_in_scheme_lists_member_entities, test_broader_narrower_related — traversal helpers. * test_verify_always_re_queries — swappable fake client returning matching→mismatched fingerprints across calls; construction + two re-checks all hit BigQuery (3 calls total). * test_multiple_meta_rows_fails_closed, test_verify_uses_limit_2_to_detect_multi_row — multi-row __meta path + LIMIT 2 SQL lock. * test_labels_for_includes_notation, test_notations_for_returns_all_values — notation surfaces in labels_for; CA appears as BOTH 'synonym' (declared in synonyms:) AND 'notation' (declared in skos:notation), matching PR #92's multiplicity contract. Live BQ validation: ran tests/test_ontology_runtime_live.py against real BigQuery (test-project-0728-467323 / adk_e2e_test) — the full emit-DDL → create-tables → attach-runtime → verify-fingerprint → run-resolver → notation-lookup → drop-tables round-trip passes in ~15s. All 50 CI tests pass; pyink + isort clean. * fix(ontology-runtime): lookup_by_notation queries label-kind row + stale doc summaries Addresses PR #154 round-2 reviewer findings. P2 - lookup_by_notation() was querying the per-row notation column (``WHERE notation = @notation``), but PR #92's concept-index emission contract says: * every declared notation value gets one ``label_kind='notation'`` row where ``label`` IS the notation, and * the per-row ``notation`` column is only the per-entity *display token* — for multi-notation entities, ``_entity_notation()`` keeps only the lexicographically smallest value. Result: ``lookup_by_notation("B")`` on an entity declaring ``skos:notation: ["A", "B"]`` used to miss the entity entirely because the ``notation`` column held "A". Fix: query ``WHERE label_kind = 'notation' AND label = @notation`` so all declared notations are reachable. Tests: 50 -> 52. * test_lookup_by_notation rewritten to match the new predicate (rows now carry label='CA', label_kind='notation'). * test_lookup_by_notation_finds_secondary_notations is the reviewer's reproducer: notation row with label='B' but the per-row notation column carrying 'A' — the entity is found via the new predicate. * test_lookup_by_notation_sql_pins_label_predicate captures the SQL and asserts ``label = @notation`` is in the WHERE clause AND ``WHERE notation = @notation`` is not — prevents regression. Live BQ validation: bumped the live test fixture to declare two notations (skos:notation: ["CA", "ZZ-LATE"]) and added a lookup for the secondary value. Locks the round-1 P2 fix against real BigQuery, not just the in-memory fake: * lookup_by_notation("CA") finds the entity (primary, lex-min). * lookup_by_notation("ZZ-LATE") ALSO finds the entity even though the per-row notation column is "CA" — the new predicate path is the only correct one. * Round-trip passes end-to-end against test-project-0728-467323 in ~30s. P3 - stale doc summaries: * docs/README.md index entry now lists MetaTableMultipleRowsError alongside the other three stable failure codes, and calls out the round-1 fixes (table_id validation, always-requery verify, traversal helpers). * CHANGELOG entry updated with MetaTableMultipleRowsError in the failure-codes list and bumped from "39 cases" to "52 cases." * docs/ontology_runtime_reader.md test-section bumped from 50 -> 52, TestConceptIndexLookupQueries count from 8 -> 10 with the notation-predicate semantics spelled out, live test description updated to mention the multi-notation fixture. * fix(ontology-runtime): notation_for returns lex-min display token, not first-authored Addresses PR #154 round-3 P2 reviewer finding. P2 - notation_for() returned the first authored skos:notation list value, but PR #92's emitter uses min(values) for the per-row notation display column (see bigquery_ontology.concept_index._entity_notation). The two views disagreed when notations were declared in non-sorted order: an entity with skos:notation: ["B", "A"] would have ExactEntityResolver report "B" while concept-index-backed rows reported "A". The reader runtime and the concept-index table must agree on the same display token. Fix: notation_for(entity_name) now returns min(notations_for( entity_name)) so the runtime view matches the emission's lex-min rule. notations_for() still preserves declaration order — callers needing every authored value (e.g. to feed ConceptIndexLookup.lookup_by_notation) keep their option. Tests: 52 -> 54. * test_notation_for_returns_lex_min_display_token: entity declaring skos:notation: ["B", "A", "C"] returns "A" from notation_for() (lex-min, not first-authored "B"). * test_exact_resolver_uses_lex_min_notation: end-to-end lock via ExactEntityResolver — candidate.notation == lex-min so the two resolver paths agree on the same entity's notation. Live BQ validation: bumped the live fixture from skos:notation: ["CA", "ZZ-LATE"] (already sorted, where lex- min == first-authored "CA") to ["ZZ-LATE", "CA"] (first- authored "ZZ-LATE", lex-min "CA"). The live test now asserts runtime.notation_for("CaliforniaRegion") == "CA" against real BigQuery — locking both the round-2 multi-notation lookup AND the round-3 lex-min display-token rule. Round-trip passes against test-project-0728-467323 in ~16s. Docs: * ontology_runtime_reader.md: notation_for table row describes the lex-min rule, test list bumped to 54 with the new regression spelled out. * CHANGELOG: CI case count bumped 52 -> 54. * fix(ontology-runtime): ExactEntityResolver honors limit=0 + refresh stale live-fixture doc Addresses PR #154 round-4 P3 reviewer findings. P3 #1 - ExactEntityResolver.resolve() accepted ``limit`` for Protocol symmetry but ignored it — always returned one candidate for a match regardless of cap. LabelSynonymResolver honored ``limit=0`` via the BQ slice. Now both Protocol implementations agree: ``limit <= 0`` returns ``[]`` so callers can disable a resolver branch by passing ``limit=0`` regardless of which implementation they hold. * test_limit_zero_returns_empty: limit=0 + limit=-1 both return []; limit=1 still returns the one match. P3 #2 - docs/ontology_runtime_reader.md live-test description still said ``["CA", "ZZ-LATE"]``, but the actual round-3 live test reversed the list to ``["ZZ-LATE", "CA"]`` so first-authored ≠ lex-min and the lex-min display-token rule is provably load-bearing. Doc now spells out the non-sorted order, lists all three live assertions (primary + secondary notation lookups plus the lex-min notation_for check), and the proof shape stays obvious to readers. CI suite bumped 54 -> 55 in both docs/ontology_runtime_reader.md and CHANGELOG.md. TestExactEntityResolver entry bumped 5 -> 6 with the limit=0 case documented.
1 parent 128bc78 commit 3407966

7 files changed

Lines changed: 3178 additions & 0 deletions

File tree

CHANGELOG.md

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,53 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
99

1010
### Added
1111

12+
- **Ontology runtime reader** in
13+
``bigquery_agent_analytics.ontology_runtime`` and
14+
[`docs/ontology_runtime_reader.md`](docs/ontology_runtime_reader.md).
15+
Issue [#58](https://github.com/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/issues/58)
16+
reader follow-on to PR #92's concept-index emission.
17+
Public surface:
18+
* ``OntologyRuntime`` — façade that loads
19+
``Ontology + Binding`` (from YAML files or in-memory
20+
models) plus an optional :class:`ConceptIndexLookup`
21+
wired to the emitted BigQuery table. Read-only
22+
accessors over entities / relationships / synonyms /
23+
annotations / SKOS schemes / notations / labels;
24+
provenance properties (``compile_fingerprint`` /
25+
``compile_id``) computed locally.
26+
* ``EntityResolver`` Protocol + two reference
27+
implementations: ``ExactEntityResolver`` (in-memory
28+
match on ``entity_name``, no BQ roundtrip) and
29+
``LabelSynonymResolver`` (BQ-backed match against the
30+
concept-index ``label`` / ``synonym`` / ``notation``
31+
rows, re-ranked by label-kind priority
32+
``name > pref > alt > hidden > synonym > notation``).
33+
**No embedding / LLM / fuzzy in this slice** — explicit
34+
non-goals; future PRs can implement the Protocol
35+
without touching the runtime surface.
36+
* ``ConceptIndexLookup`` — BigQuery-backed accessor that
37+
is **fingerprint-strict**. Three trust points: eager
38+
``verify()`` at construction (compares the table's
39+
``__meta`` row against the locally-computed
40+
``compile_fingerprint(ontology_fp, binding_fp,
41+
compiler_version)``); explicit ``verify()`` method for
42+
re-checks before long batches; per-query
43+
``WHERE compile_fingerprint = @expected_fp`` as defense
44+
in depth so stale rows can't surface even mid-flight.
45+
Three lookup methods: ``lookup_by_label`` (with
46+
label-kind / language / case-insensitive filters),
47+
``lookup_by_entity_name``, ``lookup_by_notation``.
48+
Stable failure codes: ``FingerprintMismatchError``,
49+
``MetaTableMissingError``, ``MetaTableEmptyError``,
50+
``MetaTableMultipleRowsError`` — all subclass
51+
``ConceptIndexError``.
52+
CI suite (55 cases) uses in-memory fake BQ clients;
53+
live test (gated behind ``BQAA_RUN_LIVE_ONTOLOGY_RUNTIME_TESTS=1``)
54+
emits a real concept-index via PR #92's path, attaches
55+
the runtime, runs resolver queries against the live
56+
table, asserts provenance, drops the tables on the way
57+
out. Closes the last feature dependency for #107's
58+
four-guarantee notebook (the resolve beat).
1259
- **Compiled-extractor rollout guide** at
1360
[`docs/extractor_compilation_rollout_guide.md`](docs/extractor_compilation_rollout_guide.md).
1461
Operational playbook for the Phase C pipeline (issue

docs/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ architecture, rationale, and implementation plans behind key SDK features.
2626
| [ontology_graph_v5_design.md](ontology_graph_v5_design.md) | V5: TTL import, mixed extraction, temporal lineage |
2727
| [learning_ontology_and_context_graph.md](learning_ontology_and_context_graph.md) | Learning guide for ontology and context graph |
2828
| [implementation_plan_concept_index_runtime.md](implementation_plan_concept_index_runtime.md) | Phased implementation plan for concept index + runtime entity resolution (issue #58) |
29+
| [ontology_runtime_reader.md](ontology_runtime_reader.md) | Ontology runtime reader (issue #58 reader follow-on to PR #92). `OntologyRuntime` loads ontology + binding + optional concept-index lookup. `EntityResolver` Protocol + two reference impls: `ExactEntityResolver` (in-memory) + `LabelSynonymResolver` (BQ-backed). `ConceptIndexLookup` is fingerprint-strict: eager `verify()` at construction + every `lookup_*` query includes `WHERE compile_fingerprint = @expected_fp` as defense in depth. Stable failure codes: `FingerprintMismatchError`, `MetaTableMissingError`, `MetaTableEmptyError`, `MetaTableMultipleRowsError`. `table_id` validated at construction (same regex discipline as Phase C's bundle mirror); `verify()` always re-queries (no cache); SKOS traversal helpers (`in_scheme`, `broader`, `narrower`, `related`) + `relationships_by_name` (tuple, never singular) reflect #58's traversal-first contract. NO embedding / LLM / fuzzy in this slice — those are explicit non-goals; future PRs can implement the Protocol without changing the runtime surface. |
2930

3031
## Ontology Reference
3132

docs/ontology_runtime_reader.md

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
# Ontology Runtime Reader (issue #58 reader)
2+
3+
**Status:** Implemented (issue #58 reader follow-on to PR #92)
4+
**Parent epic:** [issue #58](https://github.com/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/issues/58)
5+
**Builds on:** [PR #92 concept-index emission](https://github.com/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/pull/92), [`docs/entity_resolution_primitives.md`](entity_resolution_primitives.md)
6+
7+
---
8+
9+
## What this is
10+
11+
PR #92 ships **emission**: `gm compile --emit-concept-index` writes a deterministic concept-index table plus an `__meta` sibling carrying `compile_fingerprint` / `compile_id` provenance. This module ships the **reader**: a public Python surface in `bigquery_agent_analytics` that loads ontology + binding, attaches a fingerprint-strict BigQuery-backed concept-index lookup, and exposes two reference entity resolvers.
12+
13+
Reader is read-only by design. The emission side is the writer.
14+
15+
## Public surface
16+
17+
```python
18+
from bigquery_agent_analytics import (
19+
OntologyRuntime,
20+
EntityResolver,
21+
ExactEntityResolver,
22+
LabelSynonymResolver,
23+
ConceptIndexLookup,
24+
ConceptIndexRowView,
25+
ResolverCandidate,
26+
ConceptIndexError,
27+
FingerprintMismatchError,
28+
MetaTableMissingError,
29+
MetaTableEmptyError,
30+
)
31+
```
32+
33+
## Usage
34+
35+
### In-memory only (no concept index)
36+
37+
```python
38+
from bigquery_agent_analytics import OntologyRuntime, ExactEntityResolver
39+
40+
runtime = OntologyRuntime.from_files(
41+
ontology_path="ont.yaml",
42+
binding_path="bnd.yaml",
43+
compiler_version="bigquery_ontology 0.2.3",
44+
)
45+
46+
# Walk the loaded models
47+
print(runtime.entity("CaliforniaRegion").abstract)
48+
print(runtime.synonyms_for("Region")) # ('Area', 'Zone')
49+
print(runtime.schemes_for("CaliforniaRegion")) # ('GeoScheme',)
50+
print(runtime.notation_for("CaliforniaRegion")) # 'CA'
51+
52+
# Exact-name resolution without any BigQuery roundtrip
53+
candidates = ExactEntityResolver(runtime).resolve("CaliforniaRegion")
54+
```
55+
56+
### With concept-index lookup
57+
58+
```python
59+
from google.cloud import bigquery
60+
from bigquery_agent_analytics import OntologyRuntime, LabelSynonymResolver
61+
62+
runtime = OntologyRuntime.from_files(
63+
ontology_path="ont.yaml",
64+
binding_path="bnd.yaml",
65+
compiler_version="bigquery_ontology 0.2.3",
66+
concept_index_table="my-project.my_dataset.concept_index",
67+
bq_client=bigquery.Client(project="my-project", location="US"),
68+
)
69+
70+
# Eager fingerprint verification ran inside from_files;
71+
# if it had failed the constructor would have raised
72+
# FingerprintMismatchError before returning.
73+
74+
resolver = LabelSynonymResolver(runtime)
75+
candidates = resolver.resolve("California")
76+
# Returns ResolverCandidate(entity_name=..., matched_label=...,
77+
# matched_label_kind='name'|'pref'|'alt'|...,
78+
# compile_fingerprint=...)
79+
```
80+
81+
## `OntologyRuntime` accessors
82+
83+
| Method | Returns | Notes |
84+
|--------|---------|-------|
85+
| `entity(name, *, case_insensitive=False)` | `Entity \| None` | Single-entity lookup. Entity names are unique per ontology. |
86+
| `entities()` | `tuple[Entity, ...]` | Declared order. |
87+
| `relationships()` | `tuple[Relationship, ...]` | Declared order. |
88+
| `relationships_by_name(name)` | `tuple[Relationship, ...]` | **Always a tuple, never None / singular.** Relationship names are NOT unique per the #58 contract: traversal-style names like `skos_broader` legally repeat across distinct `(from, to)` endpoint pairs (see [entity_resolution_primitives.md §3](entity_resolution_primitives.md)). A singular `relationship(name)` accessor would silently hide duplicates; callers must handle the tuple shape explicitly. |
89+
| `synonyms_for(entity_name)` | `tuple[str, ...]` | The `synonyms:` YAML field. |
90+
| `schemes_for(entity_name)` | `tuple[str, ...]` | `skos:inScheme` annotation values (scalar OR list). |
91+
| `in_scheme(scheme)` | `tuple[str, ...]` | Forward map: entity names that are members of this scheme. |
92+
| `notation_for(entity_name)` | `str \| None` | **Lex-min display token** when multiple `skos:notation` values are declared (matches PR #92's `_entity_notation()` rule so the runtime and concept-index rows agree). Use `notations_for(...)` for every authored value. |
93+
| `notations_for(entity_name)` | `tuple[str, ...]` | Every `skos:notation` value (scalar or list normalized). |
94+
| `broader(entity_name)` | `tuple[str, ...]` | Direct parents from `skos:broader` (not transitive). |
95+
| `narrower(entity_name)` | `tuple[str, ...]` | Direct children — inverse of `broader`. |
96+
| `related(entity_name)` | `tuple[str, ...]` | `skos:related` values (not auto-symmetrized). |
97+
| `labels_for(entity_name)` | `tuple[(label, kind), ...]` | All six kinds the emission produces: name + synonyms + `skos:prefLabel` / `skos:altLabel` / `skos:hiddenLabel` (with or without `@<lang>`) + `skos:notation`. Same vocabulary as the concept-index emission. |
98+
| `annotations_for(entity_name)` | `dict[str, AnnotationValue]` | Raw annotations. |
99+
| `compile_fingerprint` (property) | `str` | Locally-computed full 64-hex sha256. |
100+
| `compile_id` (property) | `str` | 12-hex display token. |
101+
102+
## `EntityResolver` Protocol
103+
104+
Single method: `resolve(query, *, limit=10) -> list[ResolverCandidate]`.
105+
106+
Reference implementations:
107+
108+
* **`ExactEntityResolver(runtime, *, case_insensitive=False)`** — in-memory match on `entity_name`. Returns at most one candidate (entity_name is unique). No BigQuery roundtrip.
109+
* **`LabelSynonymResolver(runtime)`** — BQ-backed match against the concept-index `label` / `synonym` / `notation` rows. Requires `runtime.concept_index`. Re-ranks results by label-kind priority (`name > pref > alt > hidden > synonym > notation`); within a kind, the emission's stable sort order is preserved.
110+
111+
**Out of scope for this slice** (explicit non-goals): embedding-backed resolvers, LLM-driven matching, fuzzy / Levenshtein matching, cross-language fallback. The Protocol surface stays small enough that fuzzier resolvers can be added in future PRs without touching `OntologyRuntime`.
112+
113+
## Trust contract — fingerprint-strict reads
114+
115+
Same discipline as Phase C compiled extractors: stale provenance must never produce a confident match. The fingerprint check runs at **three points**:
116+
117+
1. **Construction-time, eager.** `OntologyRuntime.from_files(...)` / `from_models(...)` calls `ConceptIndexLookup.verify()` when a `concept_index_table` is supplied. The runtime computes the expected fingerprint locally via `compile_fingerprint(fingerprint_model(ontology), fingerprint_model(binding), compiler_version)` and compares against the `__meta` sibling table's row. Mismatch → `FingerprintMismatchError` raised before the constructor returns.
118+
2. **Explicit re-check.** `runtime.concept_index.verify()` is exposed as a public method so callers can re-check before a long batch.
119+
3. **Per-query defense in depth.** Every `lookup_*` SQL query includes `WHERE compile_fingerprint = @expected_compile_fingerprint`. Even if the table is swapped or partially corrupted between verify and query, rows with a stale fingerprint can't surface in the result.
120+
121+
### Stable failure codes
122+
123+
| Exception | Trigger |
124+
|-----------|---------|
125+
| `FingerprintMismatchError` | `__meta` row's `compile_fingerprint` differs from the locally-computed value. The table was compiled from a different ontology + binding (or different compiler version). |
126+
| `MetaTableMissingError` | The `__meta` sibling doesn't exist or the query failed. Without it, the reader has no fingerprint to compare and must fail-closed. |
127+
| `MetaTableEmptyError` | `__meta` exists but contains zero rows. PR #92 emits exactly one meta row; an empty table indicates manual tampering. |
128+
| `MetaTableMultipleRowsError` | `__meta` has more than one row. PR #92 emits exactly one; multiple rows indicate manual tampering and the runtime can't pick a "winning" fingerprint without ambiguity. `verify()` uses `LIMIT 2` so this is detected without scanning the whole table. |
129+
130+
All four subclass `ConceptIndexError` for blanket-catch.
131+
132+
### `verify()` always re-queries
133+
134+
The constructor calls `verify()` eagerly so fingerprint mismatches surface at startup. Subsequent calls to `runtime.concept_index.verify()` always re-query BigQuery — there is no cached "already verified" fast path. The intent is operational: before a long batch, call `verify()` to catch a table swap or fingerprint update mid-flight.
135+
136+
### `table_id` is validated at construction
137+
138+
`ConceptIndexLookup.__init__` and `OntologyRuntime.from_models(concept_index_table=...)` both reject malformed `project.dataset.table` identifiers at construction. Same regex discipline as `BigQueryBundleStore` (Phase C): exactly three ASCII segments, each `[A-Za-z0-9_-]+`. Backticks, semicolons, whitespace, comment markers (`--`, `/*`), trailing newlines, and wrong dot counts all raise `ValueError` before any SQL is built — injection can't reach the SQL.
139+
140+
## Concept-index lookup API
141+
142+
| Method | Use case |
143+
|--------|----------|
144+
| `lookup_by_label(label, *, case_insensitive=True, label_kinds=None, language=None, limit=100)` | "Find concepts matching this label." Backs `LabelSynonymResolver`. |
145+
| `lookup_by_entity_name(entity_name, *, label_kinds=None, limit=100)` | "Show me every label for this concept." Inverse direction. |
146+
| `lookup_by_notation(notation, *, limit=100)` | "Find concepts by notation code." Exact match (no case folding — notations are display tokens like `"ACME-7"`). |
147+
148+
Every method returns `list[ConceptIndexRowView]` carrying the full emission schema (entity_name, label, label_kind, notation, scheme, language, is_abstract, compile_id, compile_fingerprint).
149+
150+
## Tests
151+
152+
CI suite — `tests/test_ontology_runtime.py` (55 cases) using in-memory fake BigQuery clients:
153+
154+
- **`TestOntologyRuntimeConstruction`** (5) — in-memory + from-files factories; `concept_index_table` requires `bq_client`; eager fingerprint verification at construction; matching-fingerprint happy path.
155+
- **`TestOntologyRuntimeAccessors`** (10) — entity / relationships lookup, declared-order, case-sensitivity, synonyms / annotations / schemes / notation / labels traversal (covers SKOS `inScheme` list + scalar normalization, language-suffixed annotations), provenance properties (compile_fingerprint / compile_id).
156+
- **`TestConceptIndexLookupVerify`** (4) — happy path, mismatch, missing meta table, empty meta table.
157+
- **`TestConceptIndexLookupQueries`** (10) — label / entity_name / notation lookups (notation queries `label_kind='notation' AND label=@notation` so secondary notations on multi-notation entities are caught — the per-row `notation` column carries only the lex-min display token); SQL-shape lock prevents regression to the old `WHERE notation = @notation` path; `WHERE compile_fingerprint = @expected_fp` defense-in-depth lock; label-kind / language / case-insensitive filters; empty result not an error.
158+
- **`TestExactEntityResolver`** (6) — known entity, missing entity, case-sensitivity (default + opt-in), empty query, `limit=0` / negative limit returns empty (matches `LabelSynonymResolver`'s `limit=0` behavior so callers can disable a resolver branch by passing `limit=0` regardless of which Protocol implementation they hold).
159+
- **`TestLabelSynonymResolver`** (5) — requires concept index; happy path; label-kind priority re-ranking (`name > pref > alt > hidden > synonym > notation`); limit cap; empty query.
160+
- **`TestEntityResolverProtocol`** (2) — both reference resolvers satisfy `isinstance(resolver, EntityResolver)`.
161+
- **`TestRoundOneFindings`** (11) — round-1 reviewer-finding reproducers:
162+
- `table_id` rejected at construction: backtick / semicolon / whitespace / `--` / wrong dot count / trailing newline / non-string. Validation flows through `OntologyRuntime.from_models`.
163+
- `relationships_by_name` returns every matching `Relationship` (locked via direct-model construction since `load_ontology` enforces #62's pre-relaxation uniqueness). The unsafe singular `relationship(name)` accessor is dropped — `hasattr(runtime, "relationship") is False`.
164+
- SKOS traversal helpers: `in_scheme(scheme)`, `broader(entity)`, `narrower(entity)` (inverse direction), `related(entity)` (non-auto-symmetric).
165+
- `verify()` always re-queries — locked with a swappable fake client that returns matching → mismatched fingerprints across calls; verifies the construction call + two subsequent re-checks all hit BigQuery (3 calls total).
166+
- Multiple `__meta` rows → `MetaTableMultipleRowsError`; `verify()` SQL uses `LIMIT 2` so the multi-row case is detected without scanning the whole table.
167+
- `labels_for()` emits notation as `label_kind='notation'` (matching PR #92's six-kind vocabulary).
168+
- `notation_for()` returns the **lex-min display token** to match PR #92's `_entity_notation()` rule. Round-3 regression: an entity declaring `skos:notation: ["B", "A", "C"]` returns `"A"` from `notation_for()`, not the first-authored `"B"`; `ExactEntityResolver` candidates carry the same lex-min so both resolver paths agree on the same entity.
169+
170+
Live BQ suite — `tests/test_ontology_runtime_live.py` (1 case), gated behind `BQAA_RUN_LIVE_TESTS=1` + `BQAA_RUN_LIVE_ONTOLOGY_RUNTIME_TESTS=1` + `PROJECT_ID` + `DATASET_ID`. **Validated against real BigQuery.** Compiles a tiny ontology with **two notations declared in NON-sorted order** (`skos:notation: ["ZZ-LATE", "CA"]` — first-authored is `"ZZ-LATE"` but lex-min is `"CA"`) to concept-index SQL via PR #92's emission path, executes the DDL to create real BQ tables (main + `__meta`), attaches the runtime, and asserts:
171+
- `lookup_by_notation("CA")` finds the entity (primary, lex-min notation).
172+
- `lookup_by_notation("ZZ-LATE")` ALSO finds the entity (round-2 secondary-notation fix — locks against regressing to the per-row-`notation`-column predicate that would miss it).
173+
- `runtime.notation_for("CaliforniaRegion") == "CA"` (round-3 lex-min display-token rule — first-authored value would be `"ZZ-LATE"` but the runtime must match PR #92's emission rule).
174+
- Per-row `notation` column carries the lex-min display token while `label` is the queried notation value.
175+
- Every candidate carries the runtime's `compile_fingerprint`.
176+
177+
Drops the tables on the way out. Round-trip passes end-to-end against `test-project-0728-467323` in ~16s.
178+
179+
## Out of scope (deferred)
180+
181+
- **Embedding / LLM-backed resolvers** — future PRs can layer fuzzier matching on top of the `EntityResolver` Protocol without changing `OntologyRuntime`'s surface.
182+
- **Cross-language fallback**`lookup_by_label` filters by language when asked; no automatic "if French missed, try English."
183+
- **Mutation** — read-only by design. The emission side (`gm compile --emit-concept-index`) is the writer.
184+
- **Result ranking by user signals** — candidates come back in the emission's stable sort + label-kind priority. Ranking by usage / recency / context belongs in the consumer.
185+
186+
## Related
187+
188+
- [PR #92 concept-index emission](https://github.com/GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK/pull/92) — the writer side. The reader verifies against the meta rows that emission produces.
189+
- [`docs/entity_resolution_primitives.md`](entity_resolution_primitives.md) — the broader entity-resolution RFC `EntityResolver` slots into.
190+
- [`docs/implementation_plan_concept_index_runtime.md`](implementation_plan_concept_index_runtime.md) — A-series (emission) shipped; this PR ships the B-series reader scope.

0 commit comments

Comments
 (0)