Align native contract and SQL correctness

Author: nicosuave

docs/native-fixtures.md

@@ -79,7 +79,8 @@ The suite currently covers:
 - Parameter interpolation in query filters.
 - Pre-aggregation routing shape and DuckDB execution against seeded rollup tables.
 - Semantic SQL rewrite cases for single-model and relationship queries.
-- Query-local table calculations on the Rust SQL compiler path, including Rust-only DuckDB result coverage.
+- Query-local table calculations for the shared Python/Rust subset. Python applies these after fetching rows;
+  Rust compiles them into SQL window expressions.
 - Native `.sql` definition files.
 - Native SQL frontmatter model definitions.
 - YAML `sql_metrics` and `sql_segments` blocks.
@@ -112,6 +113,15 @@ The default Rust runner loads every manifest fixture, asserts `expected/validati
 
 The `adbc-exec` Rust runner executes every query with `expected_result` or `rust_expected_result` through DuckDB ADBC, using the fixture seed SQL and result columns from the manifest. Any Rust-only expected output must include `rust_only_reason`. It is enabled in CI after installing the DuckDB ADBC driver.
 
+Table-calculation fixture contract:
+
+- Shared table calculations may use `percent_of_total`, `percent_of_previous`, `running_total`, `rank`, `row_number`, or `moving_average`.
+- Shared calculations should include deterministic query `order_by` when row order affects the result.
+- Python evaluates shared calculations with `TableCalculationProcessor` after query execution.
+- Rust evaluates shared calculations by compiling them into SQL expressions.
+- Rust-only table calculation types (`dense_rank`, `difference`, `lead`, `lag`) must use `rust_expected_result` and `rust_only_reason`.
+- Python-only post-query table calculation types (`percent_of_column_total`, `percentile`) stay out of shared native fixtures until Rust supports them.
+
 ## Adding Fixtures
 
 Add the narrowest fixture that proves one semantic behavior. Avoid kitchen-sink fixtures unless the behavior itself is cross-feature interaction.

docs/native-format.md

@@ -9,6 +9,20 @@ The native format has two source forms:
 
 The native format is the runtime contract. External formats such as LookML, MetricFlow, Hex, Rill, Malloy, Omni, Superset, GoodData, Snowflake Cortex, ThoughtSpot, Holistics, Tableau, AtScale SML, BSL, and Yardstick should be converted into this format by Python importers before they are expected to run through the Rust native runtime.
 
+## Rust Loader Scope
+
+The Rust runtime and Rust CLI directory loader intentionally have a smaller direct
+input surface than Python:
+
+- `.yml` / `.yaml`: native Sidemantic YAML or Cube YAML.
+- `.sql`: native Sidemantic SQL definition files.
+
+They do not auto-detect LookML, MetricFlow/dbt manifests, Hex, Rill, Malloy,
+Omni, Superset, GoodData, Snowflake Cortex, ThoughtSpot, Holistics, Tableau,
+AtScale SML, BSL, Yardstick, or other external source formats. Convert those
+formats through the Python CLI/API first, then load the exported native YAML/SQL
+with the Rust runtime.
+
 ## Versioning
 
 Current native format version: `1`.
@@ -61,6 +75,18 @@ Top-level sections:
 | `metrics` | No | Graph-level metrics. Rust assigns these to exactly one owning model when possible. |
 | `parameters` | No | Graph-level parameters for templates and query-time substitution. |
 
+Top-level metrics are graph-scoped in the Python runtime. The Rust runtime does not
+store a separate graph-metric namespace at execution time; it assigns each top-level
+metric to one owning model by resolving explicit model references, metric dependencies,
+entity dimensions, or a single-model project fallback. If Rust cannot infer exactly
+one owner, loading fails. Portable native files should therefore make top-level metric
+dependencies explicit, for example `orders.total_revenue` rather than `total_revenue`
+when multiple models define the same local metric name. Dotted top-level metric names
+are allowed and are resolved by exact metric name before `model.metric` parsing.
+
+Top-level parameters remain graph-scoped in both runtimes. Query APIs interpolate
+parameter values before SQL compilation.
+
 ## Models
 
 Models describe physical or logical query sources.
@@ -94,6 +120,12 @@ At least one of `table`, `sql`, or `source_uri` should be present unless the mod
 | `pre_aggregations` | No | List of pre-aggregation definitions. |
 | `default_time_dimension` | No | Time dimension to add by default when the query needs time grouping. |
 | `default_grain` | No | Default time grain for the default time dimension. |
+| `auto_dimensions` | No | Python auto-discovery flag. Rust accepts `false` for compatibility and rejects `true` because it does not perform schema discovery. |
+
+Canonical CLI-authored files should use `metrics` and `sql`. The native loaders
+also accept compatibility input aliases: model-level `measures` for `metrics`,
+dimension/metric `expr` for `sql`, and metric `measure` for `sql`. Exports use
+canonical field names.
 
 Single-column primary key:
 
@@ -332,8 +364,21 @@ relationships:
 | `primary_key_columns` | Conditional | Explicit target-column list. |
 | `through` | For many-to-many | Junction model. |
 | `through_foreign_key` | For many-to-many | Source-to-through key. |
+| `through_foreign_key_columns` | For many-to-many | Explicit source-to-through key columns. |
 | `related_foreign_key` | For many-to-many | Through-to-target key. |
-| `sql` | No | Custom join SQL using runtime placeholders where supported. |
+| `related_foreign_key_columns` | For many-to-many | Explicit through-to-target key columns. |
+| `sql` | No | Custom join SQL using `{from}` and `{to}` runtime placeholders. |
+
+For CLI-authored native files, prefer explicit `foreign_key` and `primary_key`
+fields. Omitted keys are still supported for compatibility: `many_to_one`
+defaults the source key to `{name}_id`, while `one_to_many` and `one_to_one`
+default the related-side key to `id`; omitted `primary_key` resolves to the
+target model's declared primary key when building graph joins.
+
+When `sql` is present, Python and Rust use it instead of the FK/PK-generated
+predicate. `{from}` is replaced with the source model's runtime alias and `{to}`
+with the target model's runtime alias. Reverse graph traversal swaps the
+placeholders automatically.
 
 Relationship types:
 

docs/runtime-feature-matrix.md

@@ -27,7 +27,7 @@ This matrix documents current product support for native Sidemantic projects. It
 | Conversion metrics | Yes | Yes, fixture-covered compile | No dedicated fixture yet | No dedicated fixture yet |
 | Retention metrics | Yes | Yes, fixture-covered compile | No dedicated fixture yet | No dedicated fixture yet |
 | Cohort metrics | Yes | Yes, fixture-covered compile | No dedicated fixture yet | No dedicated fixture yet |
-| Table calculations | Post-query processing | Yes, Rust fixture-covered compile and Rust-only result coverage | No dedicated fixture yet | No dedicated fixture yet |
+| Table calculations | Yes, shared fixture post-query result parity | Yes, shared fixture SQL/window result parity | No dedicated fixture yet | No dedicated fixture yet |
 | Pre-aggregation routing | Yes | Yes, fixture-covered compile | No dedicated fixture yet | No dedicated fixture yet |
 | Semantic SQL rewrite | Yes | Native subset, fixture-covered | Native subset target | Narrow subset |
 | DuckDB execution | Yes | Via ADBC, fixture result parity in CI | Native DuckDB process | No |

docs/rust-runtime.md

@@ -105,4 +105,4 @@ cd sidemantic-rs && cargo test --test native_fixtures
 
 CI runs these in the `Native Compatibility` job.
 
-The shared fixture suite currently includes executable coverage for basic models, joins, fanout-safe symmetric aggregation, many-to-many joins, parameters in filters, embedded SQL definitions, SQL frontmatter definitions, default time dimensions, segments, derived/ratio metrics, and pre-aggregation routing. Table calculations have Rust-only DuckDB result coverage because Python does not accept `table_calculations` in the native query API yet. `source_uri` is covered as a validation-only load fixture and query compilation rejects it until a concrete table or SQL source is provided.
+The shared fixture suite currently includes executable coverage for basic models, joins, fanout-safe symmetric aggregation, many-to-many joins, parameters in filters, embedded SQL definitions, SQL frontmatter definitions, default time dimensions, segments, derived/ratio metrics, table calculations, and pre-aggregation routing. `source_uri` is covered as a validation-only load fixture and query compilation rejects it until a concrete table or SQL source is provided.

scripts/generate_schema.py

@@ -2,11 +2,54 @@
 """Generate JSON Schema from Pydantic models for YAML editor support."""
 
 import json
+from copy import deepcopy
 from pathlib import Path
 
 from sidemantic import Dimension, Metric, Model, Parameter, Relationship, Segment
 
 
+def add_native_relationship_aliases(schema: dict) -> dict:
+    """Expose native YAML relationship aliases that map to Python API fields."""
+    properties = schema.setdefault("properties", {})
+
+    if "foreign_key" in properties and "foreign_key_columns" not in properties:
+        foreign_key_columns = deepcopy(properties["foreign_key"])
+        foreign_key_columns["title"] = "Foreign Key Columns"
+        foreign_key_columns["description"] = "Explicit source-column list (alias for foreign_key)"
+        properties["foreign_key_columns"] = foreign_key_columns
+
+    if "primary_key" in properties and "primary_key_columns" not in properties:
+        primary_key_columns = deepcopy(properties["primary_key"])
+        primary_key_columns["title"] = "Primary Key Columns"
+        primary_key_columns["description"] = "Explicit target-column list (alias for primary_key)"
+        properties["primary_key_columns"] = primary_key_columns
+
+    if "sql" not in properties:
+        properties["sql"] = {
+            "anyOf": [{"type": "string"}, {"type": "null"}],
+            "default": None,
+            "description": "Custom join SQL using {from} and {to} runtime placeholders",
+            "title": "Sql",
+        }
+
+    return schema
+
+
+def patch_relationship_schemas(schema: dict) -> None:
+    """Patch every embedded Relationship schema emitted by Pydantic."""
+    if not isinstance(schema, dict):
+        return
+    if schema.get("title") == "Relationship":
+        add_native_relationship_aliases(schema)
+    for value in schema.values():
+        if isinstance(value, dict):
+            patch_relationship_schemas(value)
+        elif isinstance(value, list):
+            for item in value:
+                if isinstance(item, dict):
+                    patch_relationship_schemas(item)
+
+
 def generate_schema() -> dict:
     """Generate JSON Schema for sidemantic YAML files."""
     # Get schemas from pydantic models
@@ -51,6 +94,8 @@ def generate_schema() -> dict:
         },
     }
 
+    patch_relationship_schemas(schema)
+
     return schema
 
 

sidemantic-rs/src/config/loader.rs

@@ -15,7 +15,7 @@ use crate::error::{Result, SidemanticError};
 
 use super::schema::{CubeConfig, SidemanticConfig, NATIVE_FORMAT_VERSION};
 use super::sql_parser::{
-    parse_sql_definitions, parse_sql_graph_definitions_extended, parse_sql_model,
+    parse_sql_definitions, parse_sql_graph_definitions_extended, parse_sql_models,
 };
 
 #[derive(Debug)]
@@ -226,52 +226,72 @@ fn model_from_sql_frontmatter(frontmatter: serde_yaml::Mapping) -> Result<Model>
 }
 
 fn parse_sql_content(content: &str) -> Result<ParsedConfig> {
-    let has_model_statement = {
-        let upper = content.to_ascii_uppercase();
-        upper.contains("MODEL") && upper.contains("MODEL (")
-    };
-
     let mut models: Vec<Model> = Vec::new();
     let mut top_level_metrics: Vec<Metric> = Vec::new();
     let mut top_level_parameters: Vec<Parameter> = Vec::new();
 
-    if has_model_statement {
-        let model = parse_sql_model(content).map_err(|e| {
-            SidemanticError::Validation(format!("failed to parse SQL model statement: {e}"))
-        })?;
-        let model_metric_names: HashSet<String> = model
-            .metrics
-            .iter()
-            .map(|metric| metric.name.clone())
-            .collect();
-        models.push(model);
-
-        let (sql_metrics, _, sql_parameters, _) = parse_sql_graph_definitions_extended(content)
-            .map_err(|e| {
-                SidemanticError::Validation(format!("failed to parse SQL graph definitions: {e}"))
-            })?;
-        for metric in sql_metrics {
-            if !model_metric_names.contains(&metric.name) {
-                top_level_metrics.push(metric);
+    match parse_sql_models(content) {
+        Ok(parsed_models) => {
+            let model_metric_names: HashSet<String> = parsed_models
+                .iter()
+                .flat_map(|model| model.metrics.iter().map(|metric| metric.name.clone()))
+                .collect();
+            models.extend(parsed_models);
+
+            let graph_definitions = parse_sql_graph_definitions_extended(content);
+            let (sql_metrics, _, sql_parameters, _) = match graph_definitions {
+                Ok(definitions) => definitions,
+                Err(_)
+                    if content
+                        .trim_start()
+                        .to_ascii_lowercase()
+                        .starts_with("model ")
+                        && content.to_ascii_lowercase().contains(" from ") =>
+                {
+                    (Vec::new(), Vec::new(), Vec::new(), Vec::new())
+                }
+                Err(err) => {
+                    return Err(SidemanticError::Validation(format!(
+                        "failed to parse SQL graph definitions: {err}"
+                    )));
+                }
+            };
+            for metric in sql_metrics {
+                if !model_metric_names.contains(&metric.name) {
+                    top_level_metrics.push(metric);
+                }
             }
+            top_level_parameters.extend(sql_parameters);
         }
-        top_level_parameters.extend(sql_parameters);
-    } else {
-        let (frontmatter, sql_body) = parse_sql_frontmatter_and_body(content)?;
-        let (sql_metrics, sql_segments, sql_parameters, sql_preaggs) =
-            parse_sql_graph_definitions_extended(&sql_body).map_err(|e| {
-                SidemanticError::Validation(format!("failed to parse SQL graph definitions: {e}"))
-            })?;
-        top_level_parameters.extend(sql_parameters);
-
-        if let Some(frontmatter) = frontmatter {
-            let mut model = model_from_sql_frontmatter(frontmatter)?;
-            model.metrics.extend(sql_metrics);
-            model.segments.extend(sql_segments);
-            model.pre_aggregations.extend(sql_preaggs);
-            models.push(model);
-        } else {
-            top_level_metrics.extend(sql_metrics);
+        Err(model_err)
+            if content
+                .trim_start()
+                .to_ascii_lowercase()
+                .starts_with("model ") =>
+        {
+            return Err(SidemanticError::Validation(format!(
+                "failed to parse SQL model statement: {model_err}"
+            )));
+        }
+        Err(_) => {
+            let (frontmatter, sql_body) = parse_sql_frontmatter_and_body(content)?;
+            let (sql_metrics, sql_segments, sql_parameters, sql_preaggs) =
+                parse_sql_graph_definitions_extended(&sql_body).map_err(|e| {
+                    SidemanticError::Validation(format!(
+                        "failed to parse SQL graph definitions: {e}"
+                    ))
+                })?;
+            top_level_parameters.extend(sql_parameters);
+
+            if let Some(frontmatter) = frontmatter {
+                let mut model = model_from_sql_frontmatter(frontmatter)?;
+                model.metrics.extend(sql_metrics);
+                model.segments.extend(sql_segments);
+                model.pre_aggregations.extend(sql_preaggs);
+                models.push(model);
+            } else {
+                top_level_metrics.extend(sql_metrics);
+            }
         }
     }
 
@@ -345,10 +365,14 @@ pub fn load_from_sql_string_with_metadata(content: &str) -> Result<LoadedGraphMe
 ///
 /// This function:
 /// 1. Recursively finds all `.yml`/`.yaml`/`.sql` files
-/// 2. Auto-detects format (Sidemantic vs Cube.js)
+/// 2. Auto-detects only native Sidemantic YAML vs Cube YAML for YAML files
 /// 3. Parses and collects all models
 /// 4. Infers relationships from FK naming conventions
 /// 5. Returns a unified SemanticGraph
+///
+/// External formats supported by the Python package (LookML, MetricFlow, Hex,
+/// Rill, Malloy, and similar) must be converted to native YAML/SQL before using
+/// the Rust runtime loader.
 pub fn load_from_directory(dir: impl AsRef<Path>) -> Result<SemanticGraph> {
     Ok(load_from_directory_with_metadata(dir)?.graph)
 }
@@ -1001,7 +1025,9 @@ fn infer_relationships(models: &mut HashMap<String, Model>) {
                             primary_key_columns: Some(target_primary_keys.clone()),
                             through: None,
                             through_foreign_key: None,
+                            through_foreign_key_columns: None,
                             related_foreign_key: None,
+                            related_foreign_key_columns: None,
                             sql: None,
                             metadata: None,
                         },
@@ -1019,7 +1045,9 @@ fn infer_relationships(models: &mut HashMap<String, Model>) {
                             primary_key_columns: Some(target_primary_keys),
                             through: None,
                             through_foreign_key: None,
+                            through_foreign_key_columns: None,
                             related_foreign_key: None,
+                            related_foreign_key_columns: None,
                             sql: None,
                             metadata: None,
                         },
@@ -1218,6 +1246,47 @@ METRIC (
             .contains("Unsupported native Sidemantic format version 2; supported version is 1"));
     }
 
+    #[test]
+    fn test_load_from_sql_string_supports_compact_model_syntax() {
+        let sql = r#"
+model orders from orders (
+  primary key (order_id)
+  status
+  sum(amount) as revenue
+)
+"#;
+
+        let loaded = load_from_sql_string_with_metadata(sql).unwrap();
+        let orders = loaded.graph.get_model("orders").unwrap();
+        assert!(orders.get_dimension("status").is_some());
+        assert!(orders.get_metric("revenue").is_some());
+    }
+
+    #[test]
+    fn test_load_from_sql_string_keeps_multiple_legacy_models_separate() {
+        let sql = r#"
+MODEL (name orders, table orders, primary_key order_id);
+METRIC order_count AS COUNT(*);
+
+MODEL (name customers, table customers, primary_key customer_id);
+METRIC customer_count AS COUNT(*);
+"#;
+
+        let loaded = load_from_sql_string_with_metadata(sql).unwrap();
+
+        let orders = loaded.graph.get_model("orders").unwrap();
+        assert!(orders.get_metric("order_count").is_some());
+        assert!(orders.get_metric("customer_count").is_none());
+
+        let customers = loaded.graph.get_model("customers").unwrap();
+        assert!(customers.get_metric("customer_count").is_some());
+        assert!(customers.get_metric("order_count").is_none());
+        assert_eq!(
+            loaded.model_order,
+            vec!["orders".to_string(), "customers".to_string()]
+        );
+    }
+
     #[test]
     fn test_sql_frontmatter_version_is_not_model_metadata() {
         let sql = r#"

sidemantic-rs/src/config/mod.rs

@@ -15,5 +15,5 @@ pub use loader::{
 pub use schema::{CubeConfig, ModelConfig, SidemanticConfig};
 pub use sql_parser::{
     parse_sql_definitions, parse_sql_graph_definitions, parse_sql_graph_definitions_extended,
-    parse_sql_model, parse_sql_statement_blocks,
+    parse_sql_model, parse_sql_models, parse_sql_statement_blocks,
 };