Skip to content

fix: add Trino and Spark adapter support for create_table_as and full_name_split#948

Merged
haritamar merged 2 commits intomasterfrom
devin/1772379764-trino-dremio-spark-fixes
Mar 2, 2026
Merged

fix: add Trino and Spark adapter support for create_table_as and full_name_split#948
haritamar merged 2 commits intomasterfrom
devin/1772379764-trino-dremio-spark-fixes

Conversation

@devin-ai-integration
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot commented Mar 1, 2026

Summary

Adds Trino- and Spark-specific macro dispatches to fix two categories of runtime errors when elementary runs against these adapters:

  1. trino__full_name_split: The default full_name_split macro uses 0-based array indexing on the result of split(). Trino arrays are 1-based, causing TrinoUserError: SQL array indices start at 1 in schema_changes_from_baseline tests. The new macro uses indices 1/2/3.

  2. trino__edr_get_create_table_as_sql: The default dispatch calls dbt.get_create_table_as_sql() → dbt-trino's trino__create_table_as, which accesses model.config. When called from edr_create_table_as (outside model materialization), there is no model — only a relation dict — causing 'dict object' has no attribute 'config'. The fix uses a simplified CREATE TABLE ... AS statement directly.

  3. spark__edr_get_create_table_as_sql: Adds Spark-specific table creation using CREATE OR REPLACE TEMPORARY VIEW for temp tables and CREATE TABLE for regular tables.

Review & Testing Checklist for Human

  • trim() syntax in trino__full_name_split: The macro uses trim(value, '"') (two-argument form). Trino may only support trim(BOTH '"' FROM value). Verify this works against an actual Trino instance — the ClickHouse implementation explicitly uses the BOTH ... FROM syntax for this reason.
  • Trino ignores temporary flag: trino__edr_get_create_table_as_sql always creates permanent tables. Verify this is acceptable given that trino__has_temp_table_support() returns false.
  • Spark table format: spark__edr_get_create_table_as_sql uses CREATE TABLE ... AS SELECT without a USING clause. Verify Spark uses the correct default format (e.g., Hive metastore format or parquet).
  • Run integration tests: Test against actual Trino and Spark instances to ensure schema_changes_from_baseline and other elementary tests pass.

Test Plan

  1. Run elementary CI against Trino to verify schema_changes_from_baseline tests pass (should fix SQL array indices start at 1 errors)
  2. Run elementary CI against Trino to verify edr monitor report passes (should fix 'dict object' has no attribute 'config' errors)
  3. Run elementary CI against Spark to verify table creation works correctly

Notes

Summary by CodeRabbit

Release Notes

  • New Features
    • Added Trino database adapter support for table name parsing and CREATE TABLE AS operations
    • Introduced Spark database adapter for streamlined table creation operations
    • Implemented context-independent SQL generation capabilities for both adapters to improve database compatibility
    • Enhanced table management utilities for Trino and Spark platforms

Open with Devin

…_name_split

- Add trino__full_name_split: Trino arrays are 1-based, not 0-based like the default
- Add trino__edr_get_create_table_as_sql: bypass dbt-trino's create_table_as which
  accesses model.config (fails outside model materialization context)
- Add spark__edr_get_create_table_as_sql: use temporary view for temp tables

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
@devin-ai-integration
Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 1, 2026

👋 @devin-ai-integration[bot]
Thank you for raising your pull request.
Please make sure to add tests and document all user-facing changes.
You can do this by editing the docs files in the elementary repository.

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Mar 1, 2026

📝 Walkthrough

Walkthrough

This pull request adds database adapter support for Trino and Spark by introducing three new public macros. A Trino-specific name-splitting macro handles database/schema/table name extraction, while Trino and Spark-specific table creation macros provide simplified SQL generation paths that bypass existing model context dependencies.

Changes

Cohort / File(s) Summary
Name-Splitting Adapter
macros/edr/system/system_utils/full_names.sql
Added trino__full_name_split(part_name) macro that maps part names to 1-based indices and handles quoted character trimming for full table name splitting.
Table Creation Adapters
macros/utils/table_operations/create_table_as.sql
Added trino__edr_get_create_table_as_sql for simplified Trino create-table-as SQL and spark__edr_get_create_table_as_sql for Spark temporary/permanent table switching logic, both bypassing model context dependencies.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 Hop, hop! Trino hops into view,
Spark ignites with SQL anew,
Names split clean, tables take their place,
Context freed from its embrace,
Adapters bloom across the race! ✨

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically describes the main change: adding adapter support for Trino and Spark to two key macros (create_table_as and full_name_split).
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch devin/1772379764-trino-dremio-spark-fixes

Comment @coderabbitai help to get the list of available commands and usage tips.

Trino requires TRIM(BOTH '"' FROM value) instead of TRIM(value, '"')

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
macros/utils/table_operations/create_table_as.sql (1)

94-99: Consider documenting behavior when temporary=true is requested.

The temporary parameter is silently ignored, always creating a permanent table. Other adapters with similar constraints (like Redshift with dbt-fusion) include a comment explaining cleanup behavior. Consider adding similar documentation to clarify:

  1. That temporary table requests create permanent tables
  2. How/when these tables get cleaned up
📝 Suggested documentation improvement
 {% macro trino__edr_get_create_table_as_sql(temporary, relation, sql_query) %}
   {# dbt-trino's create_table_as accesses model.config which fails when called
-     outside a model context (e.g. from edr_create_table_as). Use simplified SQL. #}
+     outside a model context (e.g. from edr_create_table_as). Use simplified SQL.
+     Note: Trino temporary tables are not supported here - creates regular tables
+     that are cleaned up by Elementary's normal cleanup logic. #}
   create table {{ relation }}
   as {{ sql_query }}
 {% endmacro %}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@macros/utils/table_operations/create_table_as.sql` around lines 94 - 99,
Update the trino__edr_get_create_table_as_sql macro to document that the
temporary parameter is ignored: inside the macro
(trino__edr_get_create_table_as_sql) add a brief comment stating that when
temporary=true a permanent table is created, explain why (dbt-trino
model.context limitation), and describe the expected cleanup behavior or
how/when callers should remove these tables (e.g., manual cleanup or external
job). Ensure the comment references the temporary parameter by name so future
readers know it is intentionally not implemented.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@macros/utils/table_operations/create_table_as.sql`:
- Around line 94-99: Update the trino__edr_get_create_table_as_sql macro to
document that the temporary parameter is ignored: inside the macro
(trino__edr_get_create_table_as_sql) add a brief comment stating that when
temporary=true a permanent table is created, explain why (dbt-trino
model.context limitation), and describe the expected cleanup behavior or
how/when callers should remove these tables (e.g., manual cleanup or external
job). Ensure the comment references the temporary parameter by name so future
readers know it is intentionally not implemented.

ℹ️ Review info

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Disabled knowledge base sources:

  • Linear integration is disabled

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 06a56ee and 4d4329a.

📒 Files selected for processing (2)
  • macros/edr/system/system_utils/full_names.sql
  • macros/utils/table_operations/create_table_as.sql

@haritamar haritamar merged commit f1b70aa into master Mar 2, 2026
24 checks passed
@haritamar haritamar deleted the devin/1772379764-trino-dremio-spark-fixes branch March 2, 2026 11:32
Copy link
Copy Markdown
Contributor Author

@devin-ai-integration devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 3 additional findings.

Open in Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant