Add multi-database profiling support for SQL Server by m-abulazm · Pull Request #2536 · databrickslabs/lakebridge

m-abulazm · 2026-06-26T10:04:55Z

Changes

What does this PR do?

The mssql profiler previously assessed only a single database per run. This PR adds automatic detection between single-database and multi-database profiling based on the SQL Server edition and the configured database, and introduces a reusable variant-resolution mechanism to support it.

mssql auto-detection

A configured database scopes profiling to that one database (single_db).
A blank database lets the edition decide: Azure SQL Database (EngineEdition = 5) falls back to single_db; on-prem SQL Server and Azure SQL Managed Instance profile every accessible database (multi_db).
Credential configuration now accepts a blank database name (blank = all databases) for mssql.

Multi-database extracts (mssql/multi_db/, new)

New pipeline config and SQL extracts that fan out across every online, accessible user database (system DBs excluded), tagging each row with a database_name column.
UNION-based dynamic SQL for catalog views (tables, views, columns, routines, indexed views); cursor + USE for database-scoped DMVs (db_sizes, table_sizes) that can't use three-part naming.
All extracts and DDL gain a DATABASE_NAME column. The original config moves to mssql/single_db/.

Linked issues

Progresses #2483

Functionality

added relevant user documentation
added new CLI command
modified existing command: databricks labs lakebridge execute-database-profiler --source-tech mssql
... +add your own

Tests

manually tested: Azure SQL and On-prem SQL Server on Docker
added unit tests
added integration tests

Adds single_db and multi_db mssql profiler variants, auto-selected at execute time by probing SERVERPROPERTY('EngineEdition')

When a database is configured for mssql, the profiler scopes to just that database (single_db) on any edition; leaving it blank profiles all databases (multi_db on on-prem / Managed Instance, or the connected database on Azure SQL Database). The configure-time database prompt becomes optional for mssql (blank = all databases); legacy_synapse, which shares the configurator, still requires it.

codecov · 2026-06-26T10:09:28Z

Codecov Report

❌ Patch coverage is 82.14286% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.50%. Comparing base (d2fdae7) to head (e00cc53).

Files with missing lines	Patch %	Lines
...databricks/labs/lakebridge/assessments/variants.py	88.63%	2 Missing and 3 partials ⚠️
...databricks/labs/lakebridge/assessments/profiler.py	40.00%	3 Missing ⚠️
...ks/labs/lakebridge/connections/database_manager.py	33.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2536      +/-   ##
==========================================
+ Coverage   69.41%   69.50%   +0.09%     
==========================================
  Files         106      107       +1     
  Lines        9578     9626      +48     
  Branches     1056     1065       +9     
==========================================
+ Hits         6649     6691      +42     
- Misses       2731     2734       +3     
- Partials      198      201       +3

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2026-06-26T10:24:02Z

✅ 174/174 passed, 7 flaky, 2 skipped, 2h4m46s total

Flaky tests:

🤪 test_installs_and_runs_local_bladebridge (13.05s)
🤪 test_installs_and_runs_pypi_bladebridge (24.805s)
🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (21.536s)
🤪 test_transpiles_informatica_to_sparksql (23.184s)
🤪 test_transpile_teradata_sql (6.75s)
🤪 test_transpile_teradata_sql_non_interactive[True] (5.792s)
🤪 test_transpile_teradata_sql_non_interactive[False] (5.708s)

_{Running from acceptance #5036}

dgomez04

Let's align on the engine-variant vs. database-scope model (See variants.py).

dgomez04 · 2026-06-26T14:50:26Z

-                "database": self.prompts.question("Enter the database name"),
+                # mssql: blank profiles every accessible database (on-prem / Managed Instance); a name scopes
+                # to that one database. legacy_synapse (shares this configurator) needs the dedicated-pool name.
+                "database": (


This skips a common middle case: customers want to scope the migration to a subset of databases on a shared instance. A comma-separated allowlist here would be a natural fit.

We can either address the changes in this pull request or open an issue and follow-up.

dgomez04 · 2026-06-26T15:05:48Z

Stepping back on the architecture before we land anything. I don't think single_db vs multi_db is the right axis.

Comparing the two pipeline configurations, they are identical except for 7 of the ~13 extracts. And those that do only differ in database scope.

What EngineEdition actually tells us is the engine variant - each variant differs in system-table surface as we had discussed. That's the thing that should select a query pool / pipeline configuration.

The curent == 5 ? single_db : multi_db check both ignores this axis and mis-fires on it.

Synapse (6), serverless (9), Fabric (12), and SQL Edge (11) all fall into the else and would run the on-prem-style sys.databases + USE + three-part-INFORMATION_SCHEMA SQL, which isn't valid there.

The shape I'd propose:

**EngineEdition maps to engine variant, which selects a query pool.** Pick the dialect-appropriate set of extracts, with a safe default for unknown editions instead of assuming on-prem multi database.

Database scope is a user input (all or comma-separated list), independent of the enigne.

Engine capability decides how a multi-database scope executes: cross-database dynamic SQL on on-prem/MI; per-connection looping (or "scope the connected DB only") on Azure SQL Database, which can't do cross-DB from one connection.

Note. Scoping a single query pool by all/subset needs a way to inject WHERE [name] IN (...) into the SQL, which the type: sql pipeline can't do today, so this likely comes with adding parameter substitution to the pipeline.

Synapse, serverless, Fabric are not covered by our mssql profiler

dgomez04 · 2026-06-26T15:07:50Z

+        return None
+
+    if AUTO in spec:
+        if variant != AUTO:


Every normal mssql run reaches here with variant=None, so None != AUTO is true and we emit the warning on each run. The user never passed a variant, so "ignoring" one is confusing.

Suggest guarding the warning: if variant and variant != AUTO: so it only fires when an explicit choice is discarded.

dgomez04 · 2026-06-26T15:08:53Z

+    with DatabaseManager("mssql", connect_config) as db_manager:
+        # SERVERPROPERTY returns sql_variant, which pyodbc cannot fetch (ODBC type -16); CAST to int.
+        result = db_manager.fetch("SELECT CAST(SERVERPROPERTY('EngineEdition') AS INT) AS engine_edition")
+    engine_edition = int(result.rows[0][0])


Optional Improvement: int(result.rows[0][0]) will IndexError if the probe ever returns no rows. Low risk for SERVERPROPERTY, but a guarded failure ("could not determine SQL Server edition; defaulting to ...") reads better than a raw IndexError.

dgomez04 · 2026-06-26T15:10:35Z

+    """
+    cred_manager = create_credential_manager("mssql", EnvGetter(), creds_path=cred_file_path)
+    connect_config = cred_manager.get_credentials("mssql")
+    if connect_config.get("database"):


Blank database on Azure SQL Database (EngineEdition == 5) returns single_db, after which MSSQLConnector connects with database=None (the login's default, often master). Is profiling the default/master database the intended result, or should blank+Azure-SQL-DB require a database name?

that is a misconfiguration by the user

dey-abhishek · 2026-06-29T09:20:14Z

Follow-up on the existing Azure SQL DB + blank/all-databases thread after the latest changes:

I verified the current path still effectively routes database=\"*\" to master:

configure_assessment.py: MSSQL database prompt defaults to *
assessments/variants.py: * triggers edition probe; Azure SQL DB (EngineEdition=5) resolves to single_db
connections/database_manager.py: database=\"*\" is coerced to master
assessments/profiler.py: extraction uses the same credential config

So for Azure SQL DB, accepting defaults can still profile master (or fail for contained users without master access), instead of the intended user database.

Suggested fix: for Azure SQL DB + single_db, require a concrete database name (or rewrite to a concrete DB before extraction), and avoid relying on the * -> master fallback in that path.

dey-abhishek · 2026-06-29T09:20:15Z

The new multi-db metadata extracts build dynamic UNION ALL across multiple databases:

mssql/multi_db/tables.sql
mssql/multi_db/views.sql
mssql/multi_db/columns.sql
mssql/multi_db/routines.sql
mssql/multi_db/indexed_views.sql

On instances with mixed database collations, unioning text columns directly can fail with collation conflicts (for example, schema/table/object name fields from different DB collations).

Suggested fix: normalize all projected string expressions in every UNION branch to one chosen collation consistently (e.g., CAST(... AS NVARCHAR(...)) COLLATE <chosen_collation> for each text column in each branch).

dey-abhishek · 2026-06-29T09:20:18Z

This PR adds leading DATABASE_NAME to existing MSSQL extract tables (tables, views, columns, routines, indexed_views, table_sizes).

Pipeline behavior today appears to be:

DDL step skips when table already exists
overwrite path does TRUNCATE <table> then INSERT INTO <table> SELECT * FROM _result_frame

So rerunning profiler into an existing same-day/same-version DuckDB file created before this schema change can fail due to column count/order mismatch (old table shape vs new result shape).

Suggested fix: add schema migration/recreate logic when expected DDL has changed, or use explicit column-list inserts after schema migration.

dey-abhishek

Added PR review comments above @m-abulazm

dgomez04

LGTM - adding a comment to the original issue for comma-separated lists.

m-abulazm · 2026-07-03T16:01:32Z

The new multi-db metadata extracts build dynamic UNION ALL across multiple databases:

mssql/multi_db/tables.sql

mssql/multi_db/views.sql

mssql/multi_db/columns.sql

mssql/multi_db/routines.sql

mssql/multi_db/indexed_views.sql

On instances with mixed database collations, unioning text columns directly can fail with collation conflicts (for example, schema/table/object name fields from different DB collations).

Suggested fix: normalize all projected string expressions in every UNION branch to one chosen collation consistently (e.g., CAST(... AS NVARCHAR(...)) COLLATE <chosen_collation> for each text column in each branch).

lets solve this in a followup. it does not fit on this PR

m-abulazm · 2026-07-03T16:03:04Z

So rerunning profiler into an existing same-day/same-version DuckDB file created before this schema change can fail due to column count/order mismatch (old table shape vs new result shape).

this is not supported by our profilers and not in scope of this PR to handle

dey-abhishek

LGTM

m-abulazm added 2 commits June 26, 2026 12:04

Add multi-database SQL Server profiling via auto-routed edition variants

08911ad

Adds single_db and multi_db mssql profiler variants, auto-selected at execute time by probing SERVERPROPERTY('EngineEdition')

m-abulazm force-pushed the feat/profiler/mssql-multidb branch from 881335e to 21177f5 Compare June 26, 2026 10:06

cleanup

a13b6b7

m-abulazm force-pushed the feat/profiler/mssql-multidb branch from d8575e6 to a13b6b7 Compare June 26, 2026 13:29

m-abulazm changed the title ~~Feat/profiler/mssql multidb~~ Add multi-database profiling support for SQL Server Jun 26, 2026

m-abulazm self-assigned this Jun 26, 2026

dgomez04 requested changes Jun 26, 2026

View reviewed changes

m-abulazm added 2 commits June 29, 2026 09:33

small fix in variants.py

f4ebfab

fix all DBs handling

218d330

m-abulazm marked this pull request as ready for review June 29, 2026 08:54

m-abulazm requested a review from a team as a code owner June 29, 2026 08:54

m-abulazm requested review from dey-abhishek and dgomez04 June 29, 2026 08:54

m-abulazm added feat/profiler Issues related to profilers profiler/sql-server labels Jun 29, 2026

Merge branch 'main' into feat/profiler/mssql-multidb

2845701

dey-abhishek requested changes Jun 29, 2026

View reviewed changes

dgomez04 approved these changes Jun 29, 2026

View reviewed changes

dgomez04 mentioned this pull request Jun 29, 2026

MSSQL profiler: a single profiler run can only profile one user database #2483

Open

Merge branch 'main' into feat/profiler/mssql-multidb

ff3d115

raise in case of no database for azure sql server

4cdf7e8

m-abulazm requested a review from dey-abhishek July 3, 2026 16:14

dey-abhishek approved these changes Jul 3, 2026

View reviewed changes

m-abulazm added 2 commits July 13, 2026 10:55

Merge branch 'main' into feat/profiler/mssql-multidb

9269d40

fix test

e00cc53

m-abulazm added this pull request to the merge queue Jul 13, 2026

Merged via the queue into main with commit 7f61051 Jul 13, 2026
14 checks passed

m-abulazm deleted the feat/profiler/mssql-multidb branch July 13, 2026 10:01

Uh oh!

Conversation

m-abulazm commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

What does this PR do?

Linked issues

Functionality

Tests

Uh oh!

codecov Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dgomez04 left a comment

Choose a reason for hiding this comment

Uh oh!

dgomez04 Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

dgomez04 Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

m-abulazm Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

dgomez04 Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

dgomez04 Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

dgomez04 Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

m-abulazm Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dey-abhishek commented Jun 29, 2026

Uh oh!

dey-abhishek commented Jun 29, 2026

Uh oh!

dey-abhishek commented Jun 29, 2026

Uh oh!

dey-abhishek left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dgomez04 left a comment

Choose a reason for hiding this comment

Uh oh!

m-abulazm commented Jul 3, 2026

Uh oh!

m-abulazm commented Jul 3, 2026

Uh oh!

dey-abhishek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

m-abulazm commented Jun 26, 2026 •

edited

Loading

codecov Bot commented Jun 26, 2026 •

edited

Loading

github-actions Bot commented Jun 26, 2026 •

edited

Loading

m-abulazm Jun 29, 2026 •

edited

Loading

dey-abhishek left a comment •

edited

Loading