feat: add dbt-fabricspark adapter support (ELE-5295)

[devin-ai-integration]

feat: add dbt-fabricspark adapter support (ELE-5295)

Summary

Adds fabricspark__ prefixed macros to support the dbt-fabricspark adapter (Microsoft Fabric Lakehouse via Spark/Livy API). The dbt-fabricspark adapter dispatches as fabricspark but does not declare dependencies=["spark"] in its AdapterPlugin, so the dispatch chain is fabricspark__ → default__ instead of fabricspark__ → spark__ → default__. Since fabricspark speaks Spark SQL (not T-SQL), the existing fabric__ macros are wrong for it and the default__ macros often produce invalid SQL.

Each new fabricspark__ macro is a trivial one-line delegation to the corresponding spark__ implementation, following the same pattern used for sqlserver__ → fabric__. Only macros where spark__ differs from default__ needed overrides — 21 files across cross-db utils, data types, table operations, system utils, metadata collection, and test utils.

Updates since last revision

• Removed the defensive fallback in get_elementary_relation.sql per reviewer feedback. Multiple callers rely on None returns to detect missing relations, so the api.Relation.create() fallback was dropped entirely. The PR now contains only the fabricspark__ macro delegations with no behavioral changes to existing code paths.

Review & Testing Checklist for Human

• Completeness check: Verify no additional macros are needed where fabricspark would fall through to default__ and produce invalid Spark SQL. The ticket notes the list is "non-exhaustive" — macros without a spark__ override were intentionally skipped since Spark also uses default__ for those.
• fabricspark__edr_type_bool / fabricspark__edr_type_timestamp: These delegate to default__ (not spark__) since Spark has no override for them. Confirm this is correct for the fabricspark dialect.
• fabricspark__generate_elementary_profile_args: Delegates to spark__ which produces Databricks-oriented connection params (http_path, token). These may not match Fabric Spark's actual connection model — consider whether a dedicated implementation is needed (or accept as a known limitation for now).
• Test plan: Since there's no fabricspark adapter in CI, the macro delegations can't be directly tested. Existing Spark adapter tests validate the underlying spark__ implementations. Recommend a manual smoke test with an actual Fabric Lakehouse environment if available, or at minimum confirming existing CI (Spark, Fabric) still passes to ensure no regressions.

Notes

• Requested by: @haritamar
• Link to Devin Session
• Consider filing an upstream PR on microsoft/dbt-fabricspark to add dependencies=["spark"] to their AdapterPlugin, which would make most of these overrides unnecessary long-term.

---

Summary by CodeRabbit

• New Features
  • Added Fabric Spark support across many utilities: metadata, time/timestamp helpers, date calculations, data type normalization/mappings, and data-type lists.
  • Expanded table operations for Fabric Spark: create/replace, create-table-as, delete/insert, replace/insert flows, temp relation helpers, and name-length handling.
  • Added test-table cleanup, escape helpers, safe-cast, default incremental strategy, target-database, and Fabric Spark profile argument helpers.
  • Introduced Fabric Spark aliases for improved Spark compatibility.

[linear]

ELE-5295 Add dbt-fabricspark adapter support to dbt-data-reliability

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[github-actions]

👋 @devin-ai-integration[bot]
Thank you for raising your pull request.
Please make sure to add tests and document all user-facing changes.
You can do this by editing the docs files in the elementary repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 15e521ed-3549-488f-99cb-2c5f518462f1

📥 Commits

Reviewing files that changed from the base of the PR and between d0fb760 and e6072fa.

📒 Files selected for processing (1)

• macros/utils/cross_db_utils/generate_elementary_profile_args.sql

---

📝 Walkthrough

Walkthrough

Adds ~20 Fabricspark-prefixed wrapper macros that delegate to existing elementary.spark__* or elementary.default__* implementations across metadata, system utilities, cross-db utilities, data types, and table operations. No core logic changes; these are aliasing/delegation additions.

Changes

Cohort / File(s)	Summary
EDR Metadata & System Utilities macros/edr/metadata_collection/get_columns_from_information_schema.sql, macros/edr/system/system_utils/buckets_cte.sql, macros/edr/tests/test_utils/clean_elementary_test_tables.sql	Added fabricspark__* macros that return/delegate to corresponding elementary.spark__* implementations (get columns, complete buckets CTE, clean test tables).
Cross-DB Utility Functions macros/utils/cross_db_utils/current_timestamp.sql, macros/utils/cross_db_utils/datediff.sql, macros/utils/cross_db_utils/generate_elementary_profile_args.sql, macros/utils/cross_db_utils/incremental_strategy.sql, macros/utils/cross_db_utils/safe_cast.sql, macros/utils/cross_db_utils/target_database.sql, macros/utils/cross_db_utils/to_char.sql	Added multiple fabricspark__* wrappers (current timestamp, utc timestamp, datediff, generate profile args, default incremental strategy, safe_cast, target_database, to_char) delegating to elementary.spark__* or elementary.default__*.
Data Types macros/utils/data_types/data_type.sql, macros/utils/data_types/data_type_list.sql, macros/utils/data_types/get_normalized_data_type.sql	Added fabricspark__* macros for type mappings, data type list, and normalized data type that delegate to elementary.spark__* or elementary.default__*.
Table Operations macros/utils/table_operations/create_or_replace.sql, macros/utils/table_operations/create_table_as.sql, macros/utils/table_operations/delete_and_insert.sql, macros/utils/table_operations/get_relation_max_length.sql, macros/utils/table_operations/has_temp_table_support.sql, macros/utils/table_operations/insert_rows.sql, macros/utils/table_operations/make_temp_relation.sql, macros/utils/table_operations/replace_table_data.sql	Added fabricspark__* wrappers for create/replace, CTAS SQL, delete+insert queries, relation name length, temp-table support (returns false), escape special chars, temp relation creation, and replace-table-data — all delegating to corresponding elementary.spark__*/elementary.default__* or returning a constant.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• feat: add Docker-based Spark integration test setup #946: Adds/updates Spark-specific helpers (e.g., spark__escape_special_chars, spark__get_default_incremental_strategy) that these new fabricspark__ wrappers call.

Suggested reviewers

• haritamar

Poem

🐰 I hopped through macros, one by one,
I wrapped each spark call till the task was done,
Aliases neat with not a bug shown,
Fabricspark hums in a familiar tone,
A carrot-coded cheer — hopping home! 🥕

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main objective: adding dbt-fabricspark adapter support. It directly aligns with the 21 files changed across the codebase, each introducing fabricspark__ macros for the new adapter.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch devin/ELE-5295-1773318827

📝 Coding Plan for PR comments

• Generate coding plan

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[devin-ai-integration]

Devin Review found 1 potential issue.

View 4 additional findings in Devin Review.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@macros/utils/cross_db_utils/generate_elementary_profile_args.sql`:
- Around line 314-332: In the fabricspark__generate_elementary_profile_args
macro, fix the mistaken mapping where the "lakehouse" parameter is set to
elementary_schema; change it to use elementary_database so the macro uses the
passed-in elementary_database (keep "schema" as elementary_schema and leave the
other parameters as-is), i.e., update the _parameter("lakehouse", ...) argument
inside the macro to reference elementary_database instead of elementary_schema.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0ce94cd2-950e-430e-8721-b2eac75c7efb

📥 Commits

Reviewing files that changed from the base of the PR and between b4fb043 and d0fb760.

📒 Files selected for processing (1)

• macros/utils/cross_db_utils/generate_elementary_profile_args.sql