Commit 3407966
authored
* feat(ontology): runtime reader — OntologyRuntime + EntityResolver + concept-index lookup (#58 reader)
Reader follow-on to PR #92's concept-index emission. Closes
the last feature dependency for issue #107's
four-guarantee notebook (the resolve beat).
Public surface in ``bigquery_agent_analytics``:
* OntologyRuntime — frozen dataclass; loads ontology +
binding (from YAML files via from_files or in-memory via
from_models) plus an optional ConceptIndexLookup wired to
the emitted BigQuery table. Read-only accessors over
entities / relationships / synonyms / annotations / SKOS
schemes / notations / labels; provenance properties
(compile_fingerprint / compile_id) computed locally via
the same bigquery_ontology._fingerprint helpers PR #92's
emission uses.
* EntityResolver Protocol + two reference impls:
- ExactEntityResolver: in-memory match on entity_name,
no BigQuery roundtrip; at most one candidate per query.
- LabelSynonymResolver: BQ-backed match against
concept-index label / synonym / notation rows; re-ranks
candidates by label-kind priority
(name > pref > alt > hidden > synonym > notation).
NO embedding / LLM / fuzzy in this slice — explicit
non-goals; future PRs can implement the Protocol without
touching the runtime surface.
* ConceptIndexLookup — BigQuery-backed accessor that is
fingerprint-strict. Three trust points:
1. Eager verify() at construction. The runtime computes
expected_fp = compile_fingerprint(fingerprint_model(
ontology), fingerprint_model(binding), compiler_version)
locally and compares against the table's __meta row.
Mismatch raises FingerprintMismatchError before
from_files / from_models returns.
2. Public verify() method for explicit re-checks before
long batches.
3. Per-query "WHERE compile_fingerprint = @expected_fp"
defense in depth. Even if the table is swapped or
partially corrupted between verify and query, stale
rows can't surface in the result.
Three lookup methods: lookup_by_label (with label_kind /
language / case_insensitive filters), lookup_by_entity_name,
lookup_by_notation. Returns ConceptIndexRowView dataclass
matching PR #92's main-table schema 1:1.
Stable failure codes — all subclass ConceptIndexError:
* FingerprintMismatchError — __meta row's compile_fingerprint
differs from locally-computed value.
* MetaTableMissingError — __meta sibling table doesn't
exist / query failed.
* MetaTableEmptyError — __meta exists but zero rows
(PR #92 always writes exactly one).
Same trust discipline as Phase C compiled extractors —
stale provenance never produces a confident match.
Tests:
* tests/test_ontology_runtime.py: 39 CI cases using
in-memory fake BQ clients. Layered coverage:
TestOntologyRuntimeConstruction (5), TestOntologyRuntimeAccessors
(10), TestConceptIndexLookupVerify (4), TestConceptIndexLookupQueries
(8), TestExactEntityResolver (5), TestLabelSynonymResolver
(5), TestEntityResolverProtocol (2).
* tests/test_ontology_runtime_live.py: gated behind
BQAA_RUN_LIVE_TESTS=1 + BQAA_RUN_LIVE_ONTOLOGY_RUNTIME_TESTS=1
+ PROJECT_ID + DATASET_ID. Compiles a tiny ontology to
concept-index SQL via PR #92's emission path, executes
the DDL to create real BQ tables, attaches the runtime,
runs LabelSynonymResolver + lookup_by_notation, asserts
every candidate carries the runtime's compile_fingerprint,
drops the tables on the way out.
Docs: docs/ontology_runtime_reader.md describes the
contract, trust gates, failure codes, public API. CHANGELOG
+ docs/README.md index entries added.
* fix(ontology-runtime): 5 reviewer findings + live BQ validation
Addresses PR #154 round-1 reviewer findings.
P1 #1 — table_id injection guard. ConceptIndexLookup.__init__
used to interpolate table_id directly into backtick-quoted SQL
in verify() and _run_lookup() without shape validation. New
_TABLE_ID_PATTERN regex (same discipline as
BigQueryBundleStore from Phase C): exactly three ASCII
segments, [A-Za-z0-9_-]+, fullmatch (not match — trailing
newline rejected). Backtick, semicolon, whitespace, comment
markers, wrong dot count, and non-string all raise ValueError
at construction. Validation flows through
OntologyRuntime.from_models so callers who never construct
ConceptIndexLookup directly still get the protection.
P1 #2 — drop unsafe singular relationship(name) accessor; add
traversal helpers. Per #58's traversal-first contract
(entity_resolution_primitives.md:118), relationship names are
NOT unique once #62's relaxed (name, from, to) uniqueness
ships — traversal-style names like skos_broader legally
repeat across endpoint pairs. The singular
relationship(name) -> Relationship | None accessor would
silently return the first match and hide duplicates; dropped
entirely. New relationships_by_name(name) -> tuple[
Relationship, ...] returns every match. Plus four SKOS
traversal helpers the user asked for:
* in_scheme(scheme) -> tuple of entity names (forward map).
* broader(entity_name) -> direct skos:broader parents.
* narrower(entity_name) -> inverse (children that declare
this as broader). Not transitive.
* related(entity_name) -> skos:related values (not
auto-symmetrized).
Companion notations_for(entity_name) returns every notation
value (scalar or list), companion to notation_for which
returns only the first.
P2 #1 — verify() always re-queries. Dropped the _verified
cache flag. The docstring promised "re-check before long
batches" but the cached fast path made re-runs no-ops. Now
every verify() call hits BigQuery again so a table swap or
fingerprint update mid-flight is caught. Construction-time
eager verify still runs once; subsequent verify() calls all
re-query.
P2 #2 — reject multiple __meta rows. PR #92's emission writes
exactly one meta row per table; multiple rows indicate manual
tampering. verify() SQL now uses LIMIT 2 (not LIMIT 1) so a
multi-row case is detectable without a full scan; len(rows)
!= 1 fails closed with new MetaTableMultipleRowsError
(subclasses ConceptIndexError). The empty case still uses
MetaTableEmptyError; multi is a distinct error code so
callers can switch on the failure mode.
P2 #3 — labels_for() now emits skos:notation as
label_kind='notation', matching PR #92's six-kind vocabulary
(name / pref / alt / hidden / synonym / notation). The
previous omission meant a caller comparing in-memory labels
against emitted concept-index rows would see five kinds
locally vs six remotely.
Tests: 39 -> 50 cases. New TestRoundOneFindings (11 cases)
locks each finding with explicit reproducers:
* test_lookup_rejects_malformed_table_id — every injection
pattern + valid form acceptance.
* test_runtime_propagates_table_id_validation — validation
flows through OntologyRuntime factories.
* test_relationships_by_name_returns_every_match — builds
duplicate-name relationships directly via the Pydantic
models (load_ontology enforces pre-#62 uniqueness, so the
test bypasses the YAML path).
* test_singular_relationship_accessor_is_dropped —
hasattr(runtime, "relationship") is False.
* test_in_scheme_lists_member_entities,
test_broader_narrower_related — traversal helpers.
* test_verify_always_re_queries — swappable fake client
returning matching→mismatched fingerprints across calls;
construction + two re-checks all hit BigQuery (3 calls
total).
* test_multiple_meta_rows_fails_closed,
test_verify_uses_limit_2_to_detect_multi_row — multi-row
__meta path + LIMIT 2 SQL lock.
* test_labels_for_includes_notation,
test_notations_for_returns_all_values — notation surfaces
in labels_for; CA appears as BOTH 'synonym' (declared in
synonyms:) AND 'notation' (declared in skos:notation),
matching PR #92's multiplicity contract.
Live BQ validation: ran
tests/test_ontology_runtime_live.py against real BigQuery
(test-project-0728-467323 / adk_e2e_test) — the full
emit-DDL → create-tables → attach-runtime →
verify-fingerprint → run-resolver → notation-lookup →
drop-tables round-trip passes in ~15s. All 50 CI tests
pass; pyink + isort clean.
* fix(ontology-runtime): lookup_by_notation queries label-kind row + stale doc summaries
Addresses PR #154 round-2 reviewer findings.
P2 - lookup_by_notation() was querying the per-row notation
column (``WHERE notation = @notation``), but PR #92's
concept-index emission contract says:
* every declared notation value gets one
``label_kind='notation'`` row where ``label`` IS the
notation, and
* the per-row ``notation`` column is only the per-entity
*display token* — for multi-notation entities,
``_entity_notation()`` keeps only the lexicographically
smallest value.
Result: ``lookup_by_notation("B")`` on an entity declaring
``skos:notation: ["A", "B"]`` used to miss the entity
entirely because the ``notation`` column held "A". Fix:
query ``WHERE label_kind = 'notation' AND label = @notation``
so all declared notations are reachable.
Tests: 50 -> 52.
* test_lookup_by_notation rewritten to match the new
predicate (rows now carry label='CA', label_kind='notation').
* test_lookup_by_notation_finds_secondary_notations is the
reviewer's reproducer: notation row with label='B' but
the per-row notation column carrying 'A' — the entity is
found via the new predicate.
* test_lookup_by_notation_sql_pins_label_predicate captures
the SQL and asserts ``label = @notation`` is in the
WHERE clause AND ``WHERE notation = @notation`` is not —
prevents regression.
Live BQ validation: bumped the live test fixture to declare
two notations (skos:notation: ["CA", "ZZ-LATE"]) and added a
lookup for the secondary value. Locks the round-1 P2 fix
against real BigQuery, not just the in-memory fake:
* lookup_by_notation("CA") finds the entity (primary,
lex-min).
* lookup_by_notation("ZZ-LATE") ALSO finds the entity even
though the per-row notation column is "CA" — the new
predicate path is the only correct one.
* Round-trip passes end-to-end against
test-project-0728-467323 in ~30s.
P3 - stale doc summaries:
* docs/README.md index entry now lists
MetaTableMultipleRowsError alongside the other three
stable failure codes, and calls out the round-1 fixes
(table_id validation, always-requery verify, traversal
helpers).
* CHANGELOG entry updated with MetaTableMultipleRowsError
in the failure-codes list and bumped from "39 cases" to
"52 cases."
* docs/ontology_runtime_reader.md test-section bumped from
50 -> 52, TestConceptIndexLookupQueries count from 8 -> 10
with the notation-predicate semantics spelled out, live
test description updated to mention the multi-notation
fixture.
* fix(ontology-runtime): notation_for returns lex-min display token, not first-authored
Addresses PR #154 round-3 P2 reviewer finding.
P2 - notation_for() returned the first authored skos:notation
list value, but PR #92's emitter uses min(values) for the
per-row notation display column (see
bigquery_ontology.concept_index._entity_notation). The two
views disagreed when notations were declared in non-sorted
order: an entity with skos:notation: ["B", "A"] would have
ExactEntityResolver report "B" while concept-index-backed
rows reported "A". The reader runtime and the concept-index
table must agree on the same display token.
Fix: notation_for(entity_name) now returns min(notations_for(
entity_name)) so the runtime view matches the emission's
lex-min rule. notations_for() still preserves declaration
order — callers needing every authored value (e.g. to feed
ConceptIndexLookup.lookup_by_notation) keep their option.
Tests: 52 -> 54.
* test_notation_for_returns_lex_min_display_token: entity
declaring skos:notation: ["B", "A", "C"] returns "A" from
notation_for() (lex-min, not first-authored "B").
* test_exact_resolver_uses_lex_min_notation: end-to-end lock
via ExactEntityResolver — candidate.notation == lex-min
so the two resolver paths agree on the same entity's
notation.
Live BQ validation: bumped the live fixture from
skos:notation: ["CA", "ZZ-LATE"] (already sorted, where lex-
min == first-authored "CA") to ["ZZ-LATE", "CA"] (first-
authored "ZZ-LATE", lex-min "CA"). The live test now asserts
runtime.notation_for("CaliforniaRegion") == "CA" against
real BigQuery — locking both the round-2 multi-notation
lookup AND the round-3 lex-min display-token rule.
Round-trip passes against test-project-0728-467323 in ~16s.
Docs:
* ontology_runtime_reader.md: notation_for table row
describes the lex-min rule, test list bumped to 54 with
the new regression spelled out.
* CHANGELOG: CI case count bumped 52 -> 54.
* fix(ontology-runtime): ExactEntityResolver honors limit=0 + refresh stale live-fixture doc
Addresses PR #154 round-4 P3 reviewer findings.
P3 #1 - ExactEntityResolver.resolve() accepted ``limit`` for
Protocol symmetry but ignored it — always returned one
candidate for a match regardless of cap. LabelSynonymResolver
honored ``limit=0`` via the BQ slice. Now both Protocol
implementations agree: ``limit <= 0`` returns ``[]`` so
callers can disable a resolver branch by passing ``limit=0``
regardless of which implementation they hold.
* test_limit_zero_returns_empty: limit=0 + limit=-1 both
return []; limit=1 still returns the one match.
P3 #2 - docs/ontology_runtime_reader.md live-test description
still said ``["CA", "ZZ-LATE"]``, but the actual round-3 live
test reversed the list to ``["ZZ-LATE", "CA"]`` so
first-authored ≠ lex-min and the lex-min display-token rule
is provably load-bearing. Doc now spells out the non-sorted
order, lists all three live assertions (primary + secondary
notation lookups plus the lex-min notation_for check), and
the proof shape stays obvious to readers.
CI suite bumped 54 -> 55 in both docs/ontology_runtime_reader.md
and CHANGELOG.md. TestExactEntityResolver entry bumped 5 -> 6
with the limit=0 case documented.
1 parent 128bc78 commit 3407966
7 files changed
Lines changed: 3178 additions & 0 deletions
File tree
- docs
- src/bigquery_agent_analytics
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
12 | 59 | | |
13 | 60 | | |
14 | 61 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
0 commit comments