feat: add dbt-fabricspark adapter support (ELE-5295) by devin-ai-integration[bot] · Pull Request #964 · elementary-data/dbt-data-reliability

devin-ai-integration · 2026-03-12T12:37:29Z

feat: add dbt-fabricspark adapter support (ELE-5295)

Summary

Adds fabricspark__ prefixed macros to support the dbt-fabricspark adapter (Microsoft Fabric Lakehouse via Spark/Livy API). The dbt-fabricspark adapter dispatches as fabricspark but does not declare dependencies=["spark"] in its AdapterPlugin, so the dispatch chain is fabricspark__ → default__ instead of fabricspark__ → spark__ → default__. Since fabricspark speaks Spark SQL (not T-SQL), the existing fabric__ macros are wrong for it and the default__ macros often produce invalid SQL.

Each new fabricspark__ macro is a trivial one-line delegation to the corresponding spark__ implementation, following the same pattern used for sqlserver__ → fabric__. Only macros where spark__ differs from default__ needed overrides — 21 files across cross-db utils, data types, table operations, system utils, metadata collection, and test utils.

Updates since last revision

Removed the defensive fallback in get_elementary_relation.sql per reviewer feedback. Multiple callers rely on None returns to detect missing relations, so the api.Relation.create() fallback was dropped entirely. The PR now contains only the fabricspark__ macro delegations with no behavioral changes to existing code paths.

Review & Testing Checklist for Human

Completeness check: Verify no additional macros are needed where fabricspark would fall through to default__ and produce invalid Spark SQL. The ticket notes the list is "non-exhaustive" — macros without a spark__ override were intentionally skipped since Spark also uses default__ for those.
fabricspark__edr_type_bool / fabricspark__edr_type_timestamp: These delegate to default__ (not spark__) since Spark has no override for them. Confirm this is correct for the fabricspark dialect.
fabricspark__generate_elementary_profile_args: Delegates to spark__ which produces Databricks-oriented connection params (http_path, token). These may not match Fabric Spark's actual connection model — consider whether a dedicated implementation is needed (or accept as a known limitation for now).
Test plan: Since there's no fabricspark adapter in CI, the macro delegations can't be directly tested. Existing Spark adapter tests validate the underlying spark__ implementations. Recommend a manual smoke test with an actual Fabric Lakehouse environment if available, or at minimum confirming existing CI (Spark, Fabric) still passes to ensure no regressions.

Notes

Requested by: @haritamar
Link to Devin Session
Consider filing an upstream PR on microsoft/dbt-fabricspark to add dependencies=["spark"] to their AdapterPlugin, which would make most of these overrides unnecessary long-term.

Summary by CodeRabbit

New Features
- Added Fabric Spark support across many utilities: metadata, time/timestamp helpers, date calculations, data type normalization/mappings, and data-type lists.
- Expanded table operations for Fabric Spark: create/replace, create-table-as, delete/insert, replace/insert flows, temp relation helpers, and name-length handling.
- Added test-table cleanup, escape helpers, safe-cast, default incremental strategy, target-database, and Fabric Spark profile argument helpers.
- Introduced Fabric Spark aliases for improved Spark compatibility.

Add fabricspark__ prefixed macros that delegate to spark__ implementations. dbt-fabricspark speaks Spark SQL but doesn't declare dependencies=["spark"] in its AdapterPlugin, so dispatch falls through to default__ instead of spark__. Macros added across 22 files covering: - Cross-DB utils (timestamps, dateadd/diff, to_char, safe_cast, etc.) - Data types (string, bool, timestamp, type lists, normalization) - Table operations (create, delete/insert, temp relations, etc.) - System utils (buckets CTE) - Metadata collection (information schema) - Test utils (clean test tables) Also adds a defensive fallback in get_elementary_relation.sql using api.Relation.create() when adapter.get_relation() returns None, working around a bug in dbt-fabricspark's list_relations_without_caching. Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

linear · 2026-03-12T12:37:33Z

ELE-5295 Add dbt-fabricspark adapter support to dbt-data-reliability

devin-ai-integration · 2026-03-12T12:37:34Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment and CI monitoring

github-actions · 2026-03-12T12:37:39Z

👋 @devin-ai-integration[bot]
Thank you for raising your pull request.
Please make sure to add tests and document all user-facing changes.
You can do this by editing the docs files in the elementary repository.

coderabbitai · 2026-03-12T12:37:50Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 15e521ed-3549-488f-99cb-2c5f518462f1

📥 Commits

Reviewing files that changed from the base of the PR and between d0fb760 and e6072fa.

📒 Files selected for processing (1)

macros/utils/cross_db_utils/generate_elementary_profile_args.sql

📝 Walkthrough

Walkthrough

Adds ~20 Fabricspark-prefixed wrapper macros that delegate to existing elementary.spark__* or elementary.default__* implementations across metadata, system utilities, cross-db utilities, data types, and table operations. No core logic changes; these are aliasing/delegation additions.

Changes

Cohort / File(s)	Summary
EDR Metadata & System Utilities `macros/edr/metadata_collection/get_columns_from_information_schema.sql`, `macros/edr/system/system_utils/buckets_cte.sql`, `macros/edr/tests/test_utils/clean_elementary_test_tables.sql`	Added `fabricspark__` macros that return/delegate to corresponding `elementary.spark__` implementations (get columns, complete buckets CTE, clean test tables).
Cross-DB Utility Functions `macros/utils/cross_db_utils/current_timestamp.sql`, `macros/utils/cross_db_utils/datediff.sql`, `macros/utils/cross_db_utils/generate_elementary_profile_args.sql`, `macros/utils/cross_db_utils/incremental_strategy.sql`, `macros/utils/cross_db_utils/safe_cast.sql`, `macros/utils/cross_db_utils/target_database.sql`, `macros/utils/cross_db_utils/to_char.sql`	Added multiple `fabricspark__` wrappers (current timestamp, utc timestamp, datediff, generate profile args, default incremental strategy, safe_cast, target_database, to_char) delegating to `elementary.spark__` or `elementary.default__*`.
Data Types `macros/utils/data_types/data_type.sql`, `macros/utils/data_types/data_type_list.sql`, `macros/utils/data_types/get_normalized_data_type.sql`	Added `fabricspark__` macros for type mappings, data type list, and normalized data type that delegate to `elementary.spark__` or `elementary.default__*`.
Table Operations `macros/utils/table_operations/create_or_replace.sql`, `macros/utils/table_operations/create_table_as.sql`, `macros/utils/table_operations/delete_and_insert.sql`, `macros/utils/table_operations/get_relation_max_length.sql`, `macros/utils/table_operations/has_temp_table_support.sql`, `macros/utils/table_operations/insert_rows.sql`, `macros/utils/table_operations/make_temp_relation.sql`, `macros/utils/table_operations/replace_table_data.sql`	Added `fabricspark__` wrappers for create/replace, CTAS SQL, delete+insert queries, relation name length, temp-table support (returns `false`), escape special chars, temp relation creation, and replace-table-data — all delegating to corresponding `elementary.spark__`/`elementary.default__*` or returning a constant.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

feat: add Docker-based Spark integration test setup #946: Adds/updates Spark-specific helpers (e.g., spark__escape_special_chars, spark__get_default_incremental_strategy) that these new fabricspark__ wrappers call.

Suggested reviewers

haritamar

Poem

🐰 I hopped through macros, one by one,
I wrapped each spark call till the task was done,
Aliases neat with not a bug shown,
Fabricspark hums in a familiar tone,
A carrot-coded cheer — hopping home! 🥕

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main objective: adding dbt-fabricspark adapter support. It directly aligns with the 21 files changed across the codebase, each introducing fabricspark__ macros for the new adapter.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch devin/ELE-5295-1773318827

📝 Coding Plan for PR comments

Generate coding plan

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Per reviewer feedback, removing the api.Relation.create() fallback since callers rely on None returns to detect missing relations. Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

devin-ai-integration

Devin Review found 1 potential issue.

View 4 additional findings in Devin Review.

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

…bric Livy API params Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@macros/utils/cross_db_utils/generate_elementary_profile_args.sql`:
- Around line 314-332: In the fabricspark__generate_elementary_profile_args
macro, fix the mistaken mapping where the "lakehouse" parameter is set to
elementary_schema; change it to use elementary_database so the macro uses the
passed-in elementary_database (keep "schema" as elementary_schema and leave the
other parameters as-is), i.e., update the _parameter("lakehouse", ...) argument
inside the macro to reference elementary_database instead of elementary_schema.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0ce94cd2-950e-430e-8721-b2eac75c7efb

📥 Commits

Reviewing files that changed from the base of the PR and between b4fb043 and d0fb760.

📒 Files selected for processing (1)

macros/utils/cross_db_utils/generate_elementary_profile_args.sql

…_schema Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

devin-ai-integration Bot assigned haritamar Mar 12, 2026

This comment was marked as resolved.

Sign in to view

haritamar reviewed Mar 12, 2026

View reviewed changes

Comment thread macros/utils/graph/get_elementary_relation.sql Outdated

revert: remove get_elementary_relation.sql defensive fallback

5035255

Per reviewer feedback, removing the api.Relation.create() fallback since callers rely on None returns to detect missing relations. Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

devin-ai-integration Bot commented Mar 12, 2026

View reviewed changes

devin-ai-integration Bot and others added 2 commits March 12, 2026 12:49

merge: resolve conflicts with origin/master

b4fb043

Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

feat: dedicated fabricspark__generate_elementary_profile_args with Fa…

d0fb760

…bric Livy API params Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

coderabbitai Bot reviewed Mar 12, 2026

View reviewed changes

Comment thread macros/utils/cross_db_utils/generate_elementary_profile_args.sql

fix: map lakehouse param to elementary_database instead of elementary…

e6072fa

…_schema Co-Authored-By: Itamar Hartstein <haritamar@gmail.com>

haritamar approved these changes Mar 12, 2026

View reviewed changes

haritamar merged commit 5fafd8d into master Mar 12, 2026
25 of 27 checks passed

haritamar deleted the devin/ELE-5295-1773318827 branch March 12, 2026 13:57

devin-ai-integration Bot mentioned this pull request Mar 12, 2026

fix: include fabricspark in delta file_format config for integration tests #967

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add dbt-fabricspark adapter support (ELE-5295)#964

feat: add dbt-fabricspark adapter support (ELE-5295)#964
haritamar merged 5 commits intomasterfrom
devin/ELE-5295-1773318827

devin-ai-integration Bot commented Mar 12, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

linear Bot commented Mar 12, 2026

Uh oh!

devin-ai-integration Bot commented Mar 12, 2026

Uh oh!

github-actions Bot commented Mar 12, 2026

Uh oh!

coderabbitai Bot commented Mar 12, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

Poem

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

devin-ai-integration Bot commented Mar 12, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

feat: add dbt-fabricspark adapter support (ELE-5295)

Summary

Updates since last revision

Review & Testing Checklist for Human

Notes

Summary by CodeRabbit

Uh oh!

linear Bot commented Mar 12, 2026

Uh oh!

devin-ai-integration Bot commented Mar 12, 2026

🤖 Devin AI Engineer

Uh oh!

github-actions Bot commented Mar 12, 2026

Uh oh!

coderabbitai Bot commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

Poem

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

devin-ai-integration Bot commented Mar 12, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Mar 12, 2026 •

edited

Loading