Skip to content

Commit 0f40ab6

Browse files
committed
Align Python native contract and SQL correctness
1 parent 326838a commit 0f40ab6

95 files changed

Lines changed: 5742 additions & 3402 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

docs/native-fixtures.md

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,8 @@ The suite currently covers:
7979
- Parameter interpolation in query filters.
8080
- Pre-aggregation routing shape and DuckDB execution against seeded rollup tables.
8181
- Semantic SQL rewrite cases for single-model and relationship queries.
82-
- Query-local table calculations on the Rust SQL compiler path, including Rust-only DuckDB result coverage.
82+
- Query-local table calculations for the shared Python/Rust subset. Python applies these after fetching rows;
83+
Rust compiles them into SQL window expressions.
8384
- Native `.sql` definition files.
8485
- Native SQL frontmatter model definitions.
8586
- YAML `sql_metrics` and `sql_segments` blocks.
@@ -112,6 +113,15 @@ The default Rust runner loads every manifest fixture, asserts `expected/validati
112113

113114
The `adbc-exec` Rust runner executes every query with `expected_result` or `rust_expected_result` through DuckDB ADBC, using the fixture seed SQL and result columns from the manifest. Any Rust-only expected output must include `rust_only_reason`. It is enabled in CI after installing the DuckDB ADBC driver.
114115

116+
Table-calculation fixture contract:
117+
118+
- Shared table calculations may use `percent_of_total`, `percent_of_previous`, `running_total`, `rank`, `row_number`, or `moving_average`.
119+
- Shared calculations should include deterministic query `order_by` when row order affects the result.
120+
- Python evaluates shared calculations with `TableCalculationProcessor` after query execution.
121+
- Rust evaluates shared calculations by compiling them into SQL expressions.
122+
- Rust-only table calculation types (`dense_rank`, `difference`, `lead`, `lag`) must use `rust_expected_result` and `rust_only_reason`.
123+
- Python-only post-query table calculation types (`percent_of_column_total`, `percentile`) stay out of shared native fixtures until Rust supports them.
124+
115125
## Adding Fixtures
116126

117127
Add the narrowest fixture that proves one semantic behavior. Avoid kitchen-sink fixtures unless the behavior itself is cross-feature interaction.

docs/native-format.md

Lines changed: 46 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,20 @@ The native format has two source forms:
99

1010
The native format is the runtime contract. External formats such as LookML, MetricFlow, Hex, Rill, Malloy, Omni, Superset, GoodData, Snowflake Cortex, ThoughtSpot, Holistics, Tableau, AtScale SML, BSL, and Yardstick should be converted into this format by Python importers before they are expected to run through the Rust native runtime.
1111

12+
## Rust Loader Scope
13+
14+
The Rust runtime and Rust CLI directory loader intentionally have a smaller direct
15+
input surface than Python:
16+
17+
- `.yml` / `.yaml`: native Sidemantic YAML or Cube YAML.
18+
- `.sql`: native Sidemantic SQL definition files.
19+
20+
They do not auto-detect LookML, MetricFlow/dbt manifests, Hex, Rill, Malloy,
21+
Omni, Superset, GoodData, Snowflake Cortex, ThoughtSpot, Holistics, Tableau,
22+
AtScale SML, BSL, Yardstick, or other external source formats. Convert those
23+
formats through the Python CLI/API first, then load the exported native YAML/SQL
24+
with the Rust runtime.
25+
1226
## Versioning
1327

1428
Current native format version: `1`.
@@ -61,6 +75,18 @@ Top-level sections:
6175
| `metrics` | No | Graph-level metrics. Rust assigns these to exactly one owning model when possible. |
6276
| `parameters` | No | Graph-level parameters for templates and query-time substitution. |
6377

78+
Top-level metrics are graph-scoped in the Python runtime. The Rust runtime does not
79+
store a separate graph-metric namespace at execution time; it assigns each top-level
80+
metric to one owning model by resolving explicit model references, metric dependencies,
81+
entity dimensions, or a single-model project fallback. If Rust cannot infer exactly
82+
one owner, loading fails. Portable native files should therefore make top-level metric
83+
dependencies explicit, for example `orders.total_revenue` rather than `total_revenue`
84+
when multiple models define the same local metric name. Dotted top-level metric names
85+
are allowed and are resolved by exact metric name before `model.metric` parsing.
86+
87+
Top-level parameters remain graph-scoped in both runtimes. Query APIs interpolate
88+
parameter values before SQL compilation.
89+
6490
## Models
6591

6692
Models describe physical or logical query sources.
@@ -94,6 +120,12 @@ At least one of `table`, `sql`, or `source_uri` should be present unless the mod
94120
| `pre_aggregations` | No | List of pre-aggregation definitions. |
95121
| `default_time_dimension` | No | Time dimension to add by default when the query needs time grouping. |
96122
| `default_grain` | No | Default time grain for the default time dimension. |
123+
| `auto_dimensions` | No | Python auto-discovery flag. Rust accepts `false` for compatibility and rejects `true` because it does not perform schema discovery. |
124+
125+
Canonical CLI-authored files should use `metrics` and `sql`. The native loaders
126+
also accept compatibility input aliases: model-level `measures` for `metrics`,
127+
dimension/metric `expr` for `sql`, and metric `measure` for `sql`. Exports use
128+
canonical field names.
97129

98130
Single-column primary key:
99131

@@ -332,8 +364,21 @@ relationships:
332364
| `primary_key_columns` | Conditional | Explicit target-column list. |
333365
| `through` | For many-to-many | Junction model. |
334366
| `through_foreign_key` | For many-to-many | Source-to-through key. |
367+
| `through_foreign_key_columns` | For many-to-many | Explicit source-to-through key columns. |
335368
| `related_foreign_key` | For many-to-many | Through-to-target key. |
336-
| `sql` | No | Custom join SQL using runtime placeholders where supported. |
369+
| `related_foreign_key_columns` | For many-to-many | Explicit through-to-target key columns. |
370+
| `sql` | No | Custom join SQL using `{from}` and `{to}` runtime placeholders. |
371+
372+
For CLI-authored native files, prefer explicit `foreign_key` and `primary_key`
373+
fields. Omitted keys are still supported for compatibility: `many_to_one`
374+
defaults the source key to `{name}_id`, while `one_to_many` and `one_to_one`
375+
default the related-side key to `id`; omitted `primary_key` resolves to the
376+
target model's declared primary key when building graph joins.
377+
378+
When `sql` is present, Python and Rust use it instead of the FK/PK-generated
379+
predicate. `{from}` is replaced with the source model's runtime alias and `{to}`
380+
with the target model's runtime alias. Reverse graph traversal swaps the
381+
placeholders automatically.
337382

338383
Relationship types:
339384

0 commit comments

Comments
 (0)