feat: release v0.13.0 — expand Reconcile capabilities, add SQL/Oracle/Redshift profilers, and upgrade runtime dependencies#2
Open
a0x8o wants to merge 68 commits into
Open
Conversation
…l spark (#2394) ### What does this PR do? Migrates the reconcile integration tests off local Spark and onto the Databricks `spark` fixture (UC), and removes the no-longer-needed `local_test_run` plumbing that existed only to support that local-Spark path. ### Relevant implementation details - Replaces the local `mock_spark` fixture (Spark Connect to `sc://localhost`) with the `spark` fixture from `databricks-labs-pytester`, so all tests run against Databricks Spark with UC. `DELTA_FAILED_TO_MERGE_FIELDS` from the hand-rolled `report_tables_schema`). - Drops the `local_test_run` flag - Metadata tables are always addressed as `catalog.schema.<table>`. Tests updated accordingly (every assertion now reads `{catalog}.{schema}.MAIN` and derived from a real `make_schema` + `make_volume` - Skips tests that depend on currently-unavailable infra (Snowflake) ### Caveats/things to watch out for when reviewing: - Integration tests now require a Databricks workspace + UC. They will not run against a bare local Spark anymore. - snowflake infra is not working and respective integration tests are ignored ### Functionality - [ ] added relevant user documentation - [ ] added new CLI command - [ ] modified existing command: `databricks labs lakebridge ...` ### Tests - [x] manually tested - [ ] added unit tests - [x] added integration tests --------- Co-authored-by: Andrew Snare <andrew.snare@databricks.com> Co-authored-by: Andrew Snare <asnare@users.noreply.github.com>
#2405) ## Changes This PR implements minimal support for using a Maven mirror to install morpheus, instead of requiring direct access to Maven Central. This has been implemented by allowing an environment variable, `LAKEBRIDGE_MAVEN_URL` to override the repository URL so that a mirror is used instead of Maven Central. If credentials are required to access the mirror, they can be specified in `~/.netrc`. (Alternately, `NETRC` can be set in the environment to specify the location of this file.) ### Relevant implementation details Support for `~/.netrc` (and `NETRC`) is provided automatically by the `requests` library that we use to issue HTTP calls against the maven repository. ### Notes for reviewers This PR also syncs the `jfrog-auth` action to the latest version: this sets up the netrc file needed for this to work during CI. Lots of integration tests are still failing on `main`. Relevant integration tests for this PR include: - `test_gets_maven_artifact_version` - `test_downloads_from_maven` - `test_installs_and_runs_maven_morpheus` - `test_transpiles_all_dbt_project_files` - `test_transpile_sql_file` _Documentation updates will follow._ ### Linked issues Resolves #2359 ### Functionality - modified existing command: `databricks labs lakebridge install-transpile` ### Tests - manually tested - added unit tests - existing integration tests
## Summary The `test-profiler-connection` CLI command always exited 0 on failure because `logger.fatal()` only logs — it doesn't terminate the process. The blueprint framework's `except Exception` catch-all swallowed everything else. This replaces all three broken error handlers with `raise SystemExit(...)` ### Functionality - [ ] added relevant user documentation - [ ] added new CLI command - [ ] modified existing command: `databricks labs lakebridge ...` - [ ] ... +add your own ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [x] manually tested - [ ] added unit tests - [x] added integration tests
<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST -->
## Changes
<!-- Summary of your changes that are easy to understand. Add
screenshots when necessary, they're helpful to illustrate the before and
after state -->
### What does this PR do?
**Replaced direct JDBC connections with Databricks Unity Catalog
remote_query().**
Previously, reconciliation connected to external databases (Oracle,
Snowflake, SQL Server) by managing JDBC credentials through Databricks
secret scopes — each connector built JDBC URLs, handled
authentication (passwords, PEM keys), and executed queries directly.
Now, connections are managed through **UC Connections** — a centralized
Databricks-native way to connect to foreign data sources. The connectors
call `remote_query()` via Spark SQL, and Databricks handles
authentication and connectivity.
### Why
- **No more secret management** — credentials are managed in Unity
Catalog, not in application code or secret scopes
- **Simpler configuration** — users provide a UC connection name instead
of configuring secret scopes with individual credential keys
- **Centralized governance** — UC Connections are auditable,
permission-controlled, and managed through standard Databricks tooling
### Config change (v1 → v2)
```yaml
# Before (v1)
data_source: oracle
secret_scope: remorph_oracle
database_config:
source_schema: HR
target_catalog: main
target_schema: reconcile_target
# After (v2)
source:
dialect: oracle
catalog: ORCL
schema: HR
uc_connection_name: my_oracle_connection
target:
catalog: main
schema: reconcile_target
Existing v1 configs are auto-migrated on load.
```
### Functionality
- [ ] added relevant user documentation
- [ ] added new CLI command
- [ ] modified existing command: `databricks labs lakebridge ...`
- [ ] ... +add your own
### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->
- [ ] manually tested
- [ ] added unit tests
- [ ] added integration tests
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes This PR updates the configuration used to build the source and wheel distributions from this repository. With these changes: - The source distribution is correct; - The wheel can be built from either the source distribution (or directly from the project repository). As a bonus a lot of unintended resources (documentation) are no longer included in the wheel that we build. ### Relevant implementation details The build configuration was previously incorrect, and worked coincidentally when building a wheel directly from this repository but not when building from the source distribution. A quick summary of the gory details: - A source distribution is intended to facilitate, in a reproducible manner: a) testing the package; b) building a binary wheel distribution. - Wheels can be built _either_ from a source distribution _or_ from the project repository. - Previously, using hatch, the wheel was produced directly from the project repository. A source distribution was produced, but its layout was different: a wheel built from this was bogus and didn't contain the intended package sources. - Now, using uv, the wheel is produced in a 2-step process: 1. Build the source distribution; 2. Build the wheel from the source distribution. This exposed the problem with the source distribution. With these changes building the wheel directly from the project repo or the source distribution yield the same results. This also provided an opportunity to ensure that both the source distribution and wheel only include the intended files from the repository. ### Tests - manually tested
## Changes ### What does this PR do? This PR adds a new profiler for `mssql`. ### Relevant implementation details This profiler closely follows the implementation of the Azure Synapse profiler. However, this implementation provides a `last_execution_time` param for all `mssql` server queries, allowing for a future scheduler component to hook in an repeat queries. ### Caveats/things to watch out for when reviewing: This profiler will not support on-prem `mssql` servers. ### Linked issues N/A ### Functionality - [ ] added relevant user documentation - [ ] added new CLI command - [ ] modified existing command: `databricks labs lakebridge ...` - [X] added new profiler component - [ ] ... +add your own ### Tests - [X] manually tested - [ ] added unit tests - [ ] added integration tests --------- Co-authored-by: M Abulazm <mohamed.abulazm@databricks.com>
…eration) (#2422) ## Summary - Fixes #2366 - Foreign Catalogs (created via Lakehouse Federation) lack the Databricks-specific `full_data_type` column in `information_schema.columns`, causing `UNRESOLVED_COLUMN` errors for all reconcile report types (`schema`, `data`, `row`, `all`) - Add `DatabricksNonUnityCatalogDataSource(DatabricksDataSource)` that overrides only `get_schema` to use `DESCRIBE TABLE`, covering hive_metastore, global_temp views, and Foreign Catalogs - `DatabricksDataSource` is scoped to native Unity Catalog only (`information_schema.columns` with `full_data_type`) - `source_adapter.create_adapter` dispatches via `is_target`: target → `DatabricksDataSource`, source → `DatabricksNonUnityCatalogDataSource` - `catalog` parameter on `get_schema` / `read_data` tightened from `str | None` to `str` (now required by `SourceConnectionConfig`) ## Test plan - [x] Existing unit tests pass (1159) - [x] New / updated unit tests: `test_get_schema_uses_information_schema`, `test_get_schema_non_uc_uses_describe_table`, `test_get_schema_non_uc_foreign_catalog`, `test_get_schema_information_schema_exception_handling`, `test_get_schema_describe_exception_handling` - [x] Source adapter tests: `test_create_adapter_for_databricks_dialect_source`, `test_create_adapter_for_databricks_dialect_target` - [x] Integration test: Reconcile with Foreign Catalog (Lakebase PostgreSQL via Lakehouse Federation) as direct source --- Reopened from #2367 on an upstream branch to bypass the fork-PR OIDC restriction on JFrog auth (CI cannot run on fork PRs). All review comments and history are preserved on the original PR.
Add Redshift connector to Recon --------- Co-authored-by: M Abulazm <mohamed.abulazm@databricks.com>
# Summary This PR revamps the Lakebridge documentation with a focus on clarity, structure, and first-time user experience. - **New pages:** Added a Getting Started tutorial (end-to-end SQL Server → Databricks SQL walkthrough), a tool selection decision guide (Choosing Tools), a dedicated Morpheus transpiler page, and a Switch architecture page split out from the Switch index - **Restructured Reconcile docs**: Consolidated from 5 files (~1,400 lines) to 3 (~750 lines). The index was rewritten as a concise overview with a report-type comparison table. The former reconcile_configuration.mdx and example_config.mdx were merged into a single Configuration Reference page; recon_notebook.mdx and reconcile_automation.mdx were merged into a single Running Reconcile page covering both CLI and notebook execution paths - **Restructured SSIS docs**: Moved the three SSIS pages (ssis.mdx, ssis_supported_components.mdx, ssis_examples.mdx) into a dedicated ssis/ subfolder, creating a collapsible SSIS category in the sidebar consistent with how other multi-page sections are structured; updated all cross-links between the pages - **Other restructured pages**: Rewrote Installation for brevity; expanded FAQ from 3 to 25+ questions; merged the Transpile overview into the Transpile index; split Switch docs into a quickstart index and a separate architecture page - **Deduplication**: Removed install-transpile and configure-reconcile commands from Getting Started (they now live exclusively in Installation); Getting Started Step 3 is reframed as configuration guidance with a back-link to Installation, and Step 5 runs reconcile directly - **Navigation overhaul**: Reordered the sidebar to match the actual user journey: Installation → Getting Started → Choosing Tools → Assessment → Transpile → Reconcile → SQL Splitter → FAQ — with Installation moved before Getting Started since it is a prerequisite. Resolved a position conflict between Installation and Choosing Tools which were both set to sidebar_position: 2 --------- Co-authored-by: Andrew Snare <asnare@users.noreply.github.com> Co-authored-by: M Abulazm <mohamed.abulazm@databricks.com>
<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary, they're helpful to illustrate the before and after state --> ### What does this PR do? * test oracle reconcile * improve parsing of remote query options as it was not working for oracle * remove unused `WorkspaceClient` * remove old tests of oracle and docker script that were not automated and needed lots of manual efforts to run ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> Resolves #2416 ### Functionality - [ ] added relevant user documentation - [ ] added new CLI command - [ ] modified existing command: `databricks labs lakebridge ...` - [ ] ... +add your own ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [x] manually tested - [ ] added unit tests - [x] added integration tests
## Changes This primary purpose of this PR is: - Fixing some changes to `uv.lock` that were inadvertently introduced in #2151. - Updating our CI to detect issues in PRs relating to the lock-files. The intention within this repo is: - The lock-files should only update under 2 circumstances: - A refresh-only update of the lock-files. - A dependency in `pyproject.toml` has been changed. - Outside of this, `uv.lock` and `.build-constraints.txt` should not be changing. - When they change, it should be via `make lock-dependencies` which ensures `uv.lock` refers to PyPI properly. (Manual review is unlikely to spot the error if a dev proxy is in there, so this PR automates the check.) Further to these linting-releated changes: - Running `make lock-dependencies` now locks to the latest versions of dependencies instead of keeping the existing versions if they're still within their allowed version range. - The use of `databricks-bb-analyzer==0.3.1` is blocked: the analyzer binary in this release doesn't work properly. (Issue #2373.) Finally, to verify the linting this PR updates some dependencies: - Versions are locked against the highest version. - Outstanding dependabot updates are included. ### Relevant implementation details The linting script has been written so that it can be run locally by developers, it doesn't require the GHA environment to be in place. ### Linked Issues Resolves: #2373 Subsumes: - #2409 - #2410 - #2411 - #2412 - #2413 - #2415 ### Tests - manually tested - new CI check
<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes Traditionally GitHub Actions associate the use of an `environment` as deploying in some way. These days we can mark environment usage as unrelated to deployments, which reduces a bunch of clutter in PRs and various places in the UI.
## Changes This PR adjusts the some of the lockfile handling: - We now use `https://pypi.org/simple` (no trailing-slash) instead of `https://pypi.org/simple/` as the PyPI registry URL: this avoids spurious updates from dependabot. - Detection of lockfile drift is more robust: sometimes uv uses hatchling to examine the project, and hatchling needs more than the skeleton `pyproject.toml` that was being checked. Some trivial dependency updates are included in this PR. ### Linked issues Follows: #2434 Subsumes: - #2436 - #2439 - #2440 ### Functionality - updated uv linting ### Tests - manually tested - `lint-uv` job (workflow: `push.yml`
Bump version to 0.13.0
) ## Summary - Add optional `--switch-config-path` parameter to `llm-transpile` CLI command - Parameter allows users to specify a custom Switch configuration file in the workspace - Path must start with `/Workspace/` (validation included) - When not specified, Switch uses its default configuration Closes #2255 ## Test plan - [x] Unit tests added and passing - [x] E2E test: verify `--switch-config-path` works with actual workspace execution - [x] Verify Switch side correctly reads `switch_config_path` job parameter --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com>
<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary, they're helpful to illustrate the before and after state --> ### What does this PR do? creates a test cluster to run recon e2e tests ### Relevant implementation details use pytester and pin spark version ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [X] manually tested - [ ] added unit tests - [ ] added integration tests --------- Co-authored-by: Andrew Snare <andrew.snare@databricks.com>
## Changes ### What does this PR do? This PR updates this project so that the CLI supports Python 3.14. Aside the from the project metadata, this involved: - Updating to current versions of `mypy`, `pylint` and `pytest` that work properly with python 3.14. Some code-changes were required to satisfy the new versions of these tools. - Expands both unit and integration testing to cover all supported python versions. ### Relevant implementation details Downstream dependencies already been released with support for Python 3.14: - Blueprint - Bladebridge The `pytester` and `lsql` dependencies already work with python 3.14, although the project metadata may not reflect this. A further improvement here is integration testing across the python version matrix. ### Caveats/things to watch out for when reviewing: We have our first system imports that depend on the version of python: - `importlib.abc.Traversable` was moved to `importlib.abs.resources.Traversable` in Python 3.11, with the old location deprecated and removed in 3.14. - An import-guard based on the python version is the only way to both ensure the import works _and_ that mypy is aware of which import is currently active. Due to the expanded unit and integration tests, the set of mandatory checks on PRs for this repository will need to be adjusted. ### Linked issues Resolves #2137. ### Functionality - updated relevant user documentation - modifies existing commands: - `databricks labs install lakebridge` - `databricks labs lakebridge install-transpile` - etc. ### Tests - manually tested - existing unit tests, extended to cover python 3.14. - existing integration tests, extended to cover all supported python versions.
<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary, they're helpful to illustrate the before and after state --> ### What does this PR do? add health check on database clients to allow e.g. oracle to implement health check which works differently than current implementation ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> Unblocks #2187
…frastructure to `assessments/common/` (#2431) # Collapse DuckDB insert helpers and move shared profiler helpers to `assessments/common/` Closes #2093. ## Summary `synapse/common/duckdb_helpers.py` carried two parallel write helpers — `save_resultset_to_db` (for `FetchResult` from SQL pools, supported `overwrite`/`append` with an inlined per-table schemas dict) and `insert_df_to_duckdb` (for `pd.DataFrame` from Azure SDK / monitoring, only `overwrite`, no schema). They behaved differently on errors (one swallowed, one re-raised), shared no code, and had drifted enough that adding the SQL Server profiler in #2151 was forced to reach across into `synapse/common/` and inline 13 new MSSQL schemas into the same Synapse-named function. This pull request replaces both helpers with a single `save_to_duckdb(df, table_name, db_path, mode="overwrite", schema=None)` and lifts the shared profiler helpers out of `synapse/common/` to a new `assessments/common/` package so MSSQL (and future profilers) no longer have to import from a Synapse-named module. ### Folder layout after this PR ``` assessments/ ├── common/ NEW (sibling of synapse/, mssql/) │ ├── cli.py arguments_loader │ └── duckdb_helpers.py save_to_duckdb, get_max_column_value_duckdb, _table_exists ├── synapse/ │ └── common/ │ ├── functions.py only Azure SDK clients now (arguments_loader removed) │ └── schemas.py NEW: SYNAPSE_SCHEMAS └── mssql/ └── common/ └── schemas.py NEW: MSSQL_SCHEMAS ``` `synapse/common/duckdb_helpers.py` is gone. Zero cross-backend imports remain between `synapse/` and `mssql/`. ### `save_to_duckdb` behavior matrix | mode | schema | table state | action | |---|---|---|---| | `overwrite` | provided | any | DROP, recreate with schema, INSERT (skip insert if df empty) | | `overwrite` | none | exists | TRUNCATE, INSERT (preserves DDL types from prior run) | | `overwrite` | none | missing | `CREATE TABLE AS SELECT *` (with `LIMIT 0` if df is empty) | | `append` | provided | exists | INSERT | | `append` | provided | missing | CREATE with schema, INSERT | | `append` | none | exists | INSERT (DataFrame schema must match) | | `append` | none | missing | `CREATE TABLE AS SELECT *` from first batch (logs a warning — brittle for incremental appends) | | `append` | any | empty df | no-op | The "schema = explicit string" path is what makes incremental appends robust: pandas dtype inference can drift between batches when a column is null-only in one batch but typed in the next, breaking the INSERT. With an explicit schema DuckDB owns the type and casts on insert. ## Commits (7, each independently buildable and bisectable) 1. **`Introduce shared profiler helpers in assessments/common/`** — adds the new shared modules, the per-backend `synapse/common/schemas.py`, slims `synapse/common/duckdb_helpers.py` to back-compat shims so unmigrated callers keep working, adds 10 unit tests. 2. **`Migrate workspace_extract.py to assessments/common/`** — 14 sites. 3. **`Migrate monitoring_metrics_extract.py to assessments/common/`** — 3 sites. 4. **`Migrate dedicated_sqlpool_extract.py to assessments/common/`** — 7 sites; uses `SYNAPSE_SCHEMAS`. 5. **`Migrate serverless_sqlpool_extract.py to assessments/common/`** — 10 sites; uses `SYNAPSE_SCHEMAS`. 6. **`Migrate mssql extracts to assessments/common/ and pin MSSQL_SCHEMAS`** — adds `mssql/common/schemas.py` (13 entries lifted from #2151), migrates `mssql/info_extract.py` (9 sites) and `mssql/activity_extract.py` (4 sites). 7. **`Remove deprecated profiler helpers from synapse/common/`** — deletes the file and removes the `arguments_loader` re-export. Total: 47 call sites across 6 files migrated. ## Behavior changes (worth flagging in review) - **Errors propagate consistently.** The old `save_resultset_to_db` swallowed exceptions, which meant a profiler run could silently drop data while reporting success. `pipeline.py`'s `_run_python_script` already converts re-raised script errors into the `{"status": "error"}` JSON it expects, so this is the correct behavior. - **`overwrite` without schema now TRUNCATEs an existing table** instead of dropping and recreating with whatever pandas inferred. This preserves the DDL-declared types from a prior run — relevant for the workspace/monitoring extracts where the table was first created via CTAS. - **Empty DataFrame with no columns** is now skipped with a warning instead of attempting an invalid `CREATE TABLE`. ## Argument-order swap `insert_df_to_duckdb(df, db_path, table_name)` → `save_to_duckdb(df, table_name, db_path)`. Mechanical, but worth a sanity scan in the workspace and monitoring migrations. ## Why? #2151 had to reach into `synapse/common/` from MSSQL because there was no shared profiler module to import from. Splitting "collapse the helpers" and "give them a proper home" into two PRs would mean PR #2 depends on PR #1 in non-trivial ways, and the merge conflict that kicked off this rebase would just resurface for the next contributor. Done together, the result is one coherent story and removes the cross-cutting smell entirely. The `arguments_loader` move is in the same spirit and was free to bundle here because every migrated file was already updating its imports. ## Testing - 10 new unit tests in `tests/unit/assessment/test_duckdb_helpers.py` cover overwrite/append, explicit-schema dtype pinning, empty DataFrames, and a regression guard for the incremental-append dtype drift case. - Full `tests/unit/assessment/` suite passes (85 tests). - `ruff check` and `pylint -E,F` clean across the touched packages. - **Integration tests haven't been ran as of the current state of the pull request for either Synapse or MSSQL. Reviewers please advice.** ## Follow-ups (out of scope) - **Cross-run history accumulation for `workspace_*` and `metrics_*` tables.** They're written in `overwrite` mode today, so each profiler run wipes the prior run's data. With `save_to_duckdb` accepting an optional schema, switching them to `append` is now a one-line-per-call change once a watermark column is agreed on. Will open a separate issue. --------- Co-authored-by: M Abulazm <mohamed.abulazm@databricks.com>
…database_manager (#2305) ## Changes This PR adds Redshift as a supported profiler assessment platform. ### What does this PR do? - Adds Redshift as a supported platform for the profiler assessment (alongside Synapse). - Implements Redshift credential flows: database password, federated user, secrets manager (ARN), temporary credentials db user, temporary credentials IAM; allows SSL for all; configurator prompts and connection handling in database_manager.py. - Files changed - .gitignore - pyproject.toml - src/databricks/labs/lakebridge/connections/database_manager.py ### Relevant implementation details - Credentials: database password, federated user, secrets manager (ARN), temporary credentials db user, temporary credentials IAM. Federated and Secrets Manager use an optional lazy boto3 import. ### Caveats/things to watch out for when reviewing: ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> Resolves #.. ### Functionality - [ ] added relevant user documentation - [ ] added new CLI command - [ ] modified existing command: `databricks labs lakebridge ...` - [ ] ... +add your own ### Tests 1. Manually Tested all credential flows on all clusters in AWS Sandbox account aws-sandbox-field-eng (332745928618) tests/resources/assessments/pipeline_config_main_redshift.yml: pipeline config that runs that script. - [x] manually tested - [ ] added unit tests - [ ] added integration tests
## Changes This PR updates the installation documentation, with a primary goal of how to use mirrors and/or proxies when direct access to GitHub, PyPI or Maven Central is unavailable. ### Linked issues Documents the changes made for #2359. ### Functionality - added user documentation ### Tests Local preview, content renders as: <img width="2514" height="4870" alt="image" src="https://github.com/user-attachments/assets/98b7fc8b-8342-4244-bd28-869b7ff906ce" />
## Changes This PR updates our documentation to build with node 26, including re-locking the yarn dependencies. This is necessary because although the documentation still builds with node 25, that version is no longer the active version and it's currently unavailable via Homebrew: this means the local dev-loop is broken. ### Tests - manually tested - CI automation
<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes This PR removes the explicit dependency on the `cryptography` package: previously we used this directly in our code for setting up connections to Snowflake, but it hasn't been needed since #2362. ### Caveats/things to watch out for when reviewing: The package is still present as a transient dependency, and therefore present in the lockfile, but we no longer need to declare an explicit dependency or track the supported versions ourselves. ### Linked Issues This removes the need for #2447. ### Tests - existing unit tests - existing integration tests
Bumps [black](https://github.com/psf/black) from 25.9.0 to 26.3.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/psf/black/releases">black's releases</a>.</em></p> <blockquote> <h2>26.3.1</h2> <h3>Stable style</h3> <ul> <li>Prevent Jupyter notebook magic masking collisions from corrupting cells by using exact-length placeholders for short magics and aborting if a placeholder can no longer be unmasked safely (<a href="https://redirect.github.com/psf/black/issues/5038">#5038</a>)</li> </ul> <h3>Configuration</h3> <ul> <li>Always hash cache filename components derived from <code>--python-cell-magics</code> so custom magic names cannot affect cache paths (<a href="https://redirect.github.com/psf/black/issues/5038">#5038</a>)</li> </ul> <h3><em>Blackd</em></h3> <ul> <li>Disable browser-originated requests by default, add configurable origin allowlisting and request body limits, and bound executor submissions to improve backpressure (<a href="https://redirect.github.com/psf/black/issues/5039">#5039</a>)</li> </ul> <h2>26.3.0</h2> <h3>Stable style</h3> <ul> <li>Don't double-decode input, causing non-UTF-8 files to be corrupted (<a href="https://redirect.github.com/psf/black/issues/4964">#4964</a>)</li> <li>Fix crash on standalone comment in lambda default arguments (<a href="https://redirect.github.com/psf/black/issues/4993">#4993</a>)</li> <li>Preserve parentheses when <code># type: ignore</code> comments would be merged with other comments on the same line, preventing AST equivalence failures (<a href="https://redirect.github.com/psf/black/issues/4888">#4888</a>)</li> </ul> <h3>Preview style</h3> <ul> <li>Fix bug where <code>if</code> guards in <code>case</code> blocks were incorrectly split when the pattern had a trailing comma (<a href="https://redirect.github.com/psf/black/issues/4884">#4884</a>)</li> <li>Fix <code>string_processing</code> crashing on unassigned long string literals with trailing commas (one-item tuples) (<a href="https://redirect.github.com/psf/black/issues/4929">#4929</a>)</li> <li>Simplify implementation of the power operator "hugging" logic (<a href="https://redirect.github.com/psf/black/issues/4918">#4918</a>)</li> </ul> <h3>Packaging</h3> <ul> <li>Fix shutdown errors in PyInstaller builds on macOS by disabling multiprocessing in frozen environments (<a href="https://redirect.github.com/psf/black/issues/4930">#4930</a>)</li> </ul> <h3>Performance</h3> <ul> <li>Introduce winloop for windows as an alternative to uvloop (<a href="https://redirect.github.com/psf/black/issues/4996">#4996</a>)</li> <li>Remove deprecated function <code>uvloop.install()</code> in favor of <code>uvloop.new_event_loop()</code> (<a href="https://redirect.github.com/psf/black/issues/4996">#4996</a>)</li> <li>Rename <code>maybe_install_uvloop</code> function to <code>maybe_use_uvloop</code> to simplify loop installation and creation of either a uvloop/winloop evenloop or default eventloop (<a href="https://redirect.github.com/psf/black/issues/4996">#4996</a>)</li> </ul> <h3>Output</h3> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/psf/black/blob/main/CHANGES.md">black's changelog</a>.</em></p> <blockquote> <h2>Version 26.3.1</h2> <h3>Stable style</h3> <ul> <li>Prevent Jupyter notebook magic masking collisions from corrupting cells by using exact-length placeholders for short magics and aborting if a placeholder can no longer be unmasked safely (<a href="https://redirect.github.com/psf/black/issues/5038">#5038</a>)</li> </ul> <h3>Configuration</h3> <ul> <li>Always hash cache filename components derived from <code>--python-cell-magics</code> so custom magic names cannot affect cache paths (<a href="https://redirect.github.com/psf/black/issues/5038">#5038</a>)</li> </ul> <h3><em>Blackd</em></h3> <ul> <li>Disable browser-originated requests by default, add configurable origin allowlisting and request body limits, and bound executor submissions to improve backpressure (<a href="https://redirect.github.com/psf/black/issues/5039">#5039</a>)</li> </ul> <h2>Version 26.3.0</h2> <h3>Stable style</h3> <ul> <li>Don't double-decode input, causing non-UTF-8 files to be corrupted (<a href="https://redirect.github.com/psf/black/issues/4964">#4964</a>)</li> <li>Fix crash on standalone comment in lambda default arguments (<a href="https://redirect.github.com/psf/black/issues/4993">#4993</a>)</li> <li>Preserve parentheses when <code># type: ignore</code> comments would be merged with other comments on the same line, preventing AST equivalence failures (<a href="https://redirect.github.com/psf/black/issues/4888">#4888</a>)</li> </ul> <h3>Preview style</h3> <ul> <li>Fix bug where <code>if</code> guards in <code>case</code> blocks were incorrectly split when the pattern had a trailing comma (<a href="https://redirect.github.com/psf/black/issues/4884">#4884</a>)</li> <li>Fix <code>string_processing</code> crashing on unassigned long string literals with trailing commas (one-item tuples) (<a href="https://redirect.github.com/psf/black/issues/4929">#4929</a>)</li> <li>Simplify implementation of the power operator "hugging" logic (<a href="https://redirect.github.com/psf/black/issues/4918">#4918</a>)</li> </ul> <h3>Packaging</h3> <ul> <li>Fix shutdown errors in PyInstaller builds on macOS by disabling multiprocessing in frozen environments (<a href="https://redirect.github.com/psf/black/issues/4930">#4930</a>)</li> </ul> <h3>Performance</h3> <ul> <li>Introduce winloop for windows as an alternative to uvloop (<a href="https://redirect.github.com/psf/black/issues/4996">#4996</a>)</li> <li>Remove deprecated function <code>uvloop.install()</code> in favor of <code>uvloop.new_event_loop()</code> (<a href="https://redirect.github.com/psf/black/issues/4996">#4996</a>)</li> <li>Rename <code>maybe_install_uvloop</code> function to <code>maybe_use_uvloop</code> to simplify loop installation and creation of either a uvloop/winloop eventloop or default eventloop (<a href="https://redirect.github.com/psf/black/issues/4996">#4996</a>)</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/psf/black/commit/c6755bb741b6481d6b3d3bb563c83fa060db96c9"><code>c6755bb</code></a> Prepare release 26.3.1 (<a href="https://redirect.github.com/psf/black/issues/5046">#5046</a>)</li> <li><a href="https://github.com/psf/black/commit/69973fd6950985fbeb1090d96da717dc4d8380b0"><code>69973fd</code></a> Harden blackd browser-facing request handling (<a href="https://redirect.github.com/psf/black/issues/5039">#5039</a>)</li> <li><a href="https://github.com/psf/black/commit/4937fe6cf241139ddbfc16b0bdbb5b422798909d"><code>4937fe6</code></a> Fix some shenanigans with the cache file and IPython (<a href="https://redirect.github.com/psf/black/issues/5038">#5038</a>)</li> <li><a href="https://github.com/psf/black/commit/2e641d174469c505d5ae905e75d4c769597e681f"><code>2e641d1</code></a> docs: remove outdated Black Playground references (<a href="https://redirect.github.com/psf/black/issues/5044">#5044</a>)</li> <li><a href="https://github.com/psf/black/commit/c014b22a2d5e0632587b47b81151658bddfa0b88"><code>c014b22</code></a> Remove unused internal code (<a href="https://redirect.github.com/psf/black/issues/5041">#5041</a>)</li> <li><a href="https://github.com/psf/black/commit/0dae20b2d009f2f03de8696d06b0c947d3abafc9"><code>0dae20b</code></a> Add new changelog (<a href="https://redirect.github.com/psf/black/issues/5036">#5036</a>)</li> <li><a href="https://github.com/psf/black/commit/c5c1cbddd92cecb554ac2a77a24139dd76831030"><code>c5c1cbd</code></a> Minor release patches (<a href="https://redirect.github.com/psf/black/issues/5035">#5035</a>)</li> <li><a href="https://github.com/psf/black/commit/7e5a828c37d71b6a6666e28eed444816def6a8f4"><code>7e5a828</code></a> docs: clarify relationship between Black style and PEP 8 (<a href="https://redirect.github.com/psf/black/issues/5025">#5025</a>)</li> <li><a href="https://github.com/psf/black/commit/69705deb8776e7c5e585668da106d1abe2cb8d77"><code>69705de</code></a> docs: add clearer pyproject configuration guidance (<a href="https://redirect.github.com/psf/black/issues/5026">#5026</a>)</li> <li><a href="https://github.com/psf/black/commit/35ea67920b7f6ac8e09be1c47278752b1e827f76"><code>35ea679</code></a> Prepare release 26.3.0 (<a href="https://redirect.github.com/psf/black/issues/5032">#5032</a>)</li> <li>Additional commits viewable in <a href="https://github.com/psf/black/compare/25.9.0...26.3.1">compare view</a></li> </ul> </details> <br /> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Andrew Snare <andrew.snare@databricks.com>
<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary, they're helpful to illustrate the before and after state --> ### What does this PR do? profile legacy sql dw ### Relevant implementation details added extraction queries ### Caveats/things to watch out for when reviewing: ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> Resolves #2224 ### Functionality - [ ] added relevant user documentation - [ ] added new CLI command - [X] modified existing command: `databricks labs lakebridge configure-database-profiler` ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [x] manually tested - [X] added unit tests - [ ] added integration tests
## Changes Creating the oracle extractor. Unit tests passed for: - dialog to configure the extractor - run scripts to gather data - local database population with extracted data from the Oracle Database. ### What does this PR do? Oracle profiler data extraction ### Relevant implementation details The data analysis is still WIP. ### Caveats/things to watch out for when reviewing: ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> Resolves #.. ### Functionality - [ ] added relevant user documentation - [ ] added new CLI command - [ ] modified existing command: `databricks labs lakebridge ...` - [ ] ... +add your own ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [x] manually tested - [x] added unit tests - [ ] added integration tests Co-authored-by: M Abulazm <mohamed.abulazm@databricks.com>
## Changes
### What does this PR do?
* Standardizes the reconcile sampling query across all dialects
* Azure Synapse Dedicated SQL Pool does not accept VALUES as a derived
table, so the previous T-SQL path failed at runtime on Synapse
connections.
### Implementation Details
replaces the previous CTE form (non-T-SQL) and `VALUES` derived-table
form (T-SQL) with a single derived-table-join shape that works
everywhere.
```sql
-- Before (T-SQL path):
INNER JOIN (VALUES (...), (...)) AS recon([col1], [col2]) ON ...
-- Before (other dialects):
WITH recon AS (SELECT ... UNION SELECT ...), src AS (...)
SELECT ... FROM src INNER JOIN recon ON ...
-- After (all dialects):
SELECT ... FROM (...) AS src
INNER JOIN (SELECT ... UNION SELECT ...) AS recon ON ...
```
### Linked issues
Resolves #2418
### Functionality
- [ ] added relevant user documentation
- [ ] added new CLI command
- [ ] modified existing command: `databricks labs lakebridge ...`
### Tests
- [x] manually tested
- [x] added unit tests
- [x] added integration tests
…QL scripts (#2482) ## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary, they're helpful to illustrate the before and after state --> ### What does this PR do? * Converts the MSSQL profiler from `type: python` venv steps to `type: sql` / `type: ddl` in-process steps ### Key changes - Replace `activity_extract.py`, `info_extract.py`, and `mssql/common/*`. - Add and modify 13 query/DDL SQL pairs to profile. - Use exisitng connector `MSSQLConnector` instead of duplicate in `get_sqlserver_reader`. - DDL types match documented source types (`SMALLINT`, `INTEGER`, `DOUBLE`, `TIMESTAMP`). - `sys_info.sql` enumerates 37 columns explicitly (no `SELECT *`). - `database` configurator prompt no longer defaults to `master`; user supplies the actual target DB. Docs (`mssql.mdx`, `legacy_synapse.mdx`) aligned with actual prompts. ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> Resolves #2459 ### Functionality - [ ] modified existing command: `databricks labs lakebridge execute-database-profiler --source-tech mssql` ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [X] manually tested - [ ] added unit tests - [ ] added integration tests
…ation schema (#2304) ## Changes This PR adds Redshift as a supported profiler assessment platform. It introduces Redshift resources for serverless, provisioned, and provisioned_multi_az ### What does this PR do? - Adds Redshift as a supported platform for the profiler assessment (alongside Synapse). - Adds Redshift resources for all three variants (serverless, provisioned, provisioned_multi_az): query/SQL files (e.g. query_view, rs_managed_storage_gb, rs_nodes, chart queries), pipeline configs, and validation schema. - Files (2 levels: folder → subfolder, 3rd level: files under each)): - resources/assessments/redshift/ → provisioned/, provisioned_multi_az/, serverless/ with their .sql and pipeline_config.yml - resources/assessments/validation/ → redshift_extract_schema.yml ### Relevant implementation details - Resources: Same query/file layout for serverless, provisioned, and provisioned_multi_az under resources/assessments/redshift/{variant}/ (e.g. 0_query_view.sql … 9_chart_*.sql, pipeline_config.yml); serverless also has 10_cost_incurred.sql. Shared validation: resources/assessments/validation/redshift_extract_schema.yml. ### Caveats/things to watch out for when reviewing: - Multi-AZ Redshift has limited system views (e.g. some STV views unavailable); provisioned_multi_az SQL reflects that where applicable. - Serverless Redshift has one additional query ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> Resolves #.. ### Functionality - [ ] added relevant user documentation - [ ] added new CLI command - [ ] modified existing command: `databricks labs lakebridge ...` - [ ] ... +add your own ### Tests 1. Manually Tested all credential flows on all clusters in AWS Sandbox account aws-sandbox-field-eng (332745928618) tests/resources/assessments/pipeline_config_main_redshift.yml: pipeline config that runs that script. - [x] manually tested - [ ] added unit tests - [ ] added integration tests --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: M Abulazm <mohamed.abulazm@databricks.com>
…CO summaries (#2420) ## Changes ### What does this PR do? Adds a **Snowflake source** to the Lakebridge profiler (`databricks labs lakebridge configure-database-profiler` / `execute-database-profiler`). End-to-end this: 1. Prompts for Snowflake connection details + a Programmatic Access Token (PAT) and saves credentials. 2. Extracts from `SNOWFLAKE.ACCOUNT_USAGE` (warehouse usage, query history, storage, users, account info, plus optional pipe / autoclustering / mat-view refresh credits) into a timestamped DuckDB file (`profiler_extract_YYYYMMDD_HHMMSS.db`). 3. Runs post-extraction DuckDB computations to produce summary tables: `query_mapper`, `warehouse_mapper`, `classified_queries`, `annual_usage_by_category`, `storage_summary`, and `migration_complexity`. 4. Ships a full PAT-setup walkthrough (with screenshots) under `docs/lakebridge/docs/assessment/profiler/snowflake.mdx`. ### Relevant implementation details - **New profiler source registered** in `assessments/_constants.py` (`PROFILER_SOURCE_SYSTEM`, `PLATFORM_TO_SOURCE_TECHNOLOGY_CFG`, `CONNECTOR_REQUIRED`). - **`SnowflakeConnector`** in `connections/database_manager.py` is now implemented (was `NotImplementedError`). Uses `snowflake-sqlalchemy` + PAT (password authenticator). Eagerly opens a connection so PAT / network-policy errors surface up front with actionable guidance and a docs link. - **`ConfigureSnowflakeAssessment`** in `assessments/configure_assessment.py` collects connection + profiler config (lookback 1–365 days, default 365 to match the legacy notebook). - **Pipeline changes** (`assessments/pipeline.py`): - DB filename now timestamped (`profiler_extract_<ts>.db`) so reruns don't clobber prior extracts. - SQL templates support `{lookback_days}` substitution from saved config. - Optional steps that fail no longer abort the run; they log a warning and (for `adjusted_rate`) write a sensible default row. - Every extracted table gets an `extract_timestamp` column. - **Migration complexity analysis** (`assessments/snowflake_computations.py`) classifies each unique query as native/simple/medium/complex by parsing function calls against bundled Snowflake + Photon function lists. Defaults to a 6h wall-clock timeout; on timeout the user is prompted once to retry without a limit. - **Resource-path resolution** (`_constants.py::_find_product_path_prefix`) now works for both `databricks labs install` layouts *and* editable `pip install -e .` source layouts — earlier the CLI couldn't find SQL resources outside the labs install path. - **Auth ergonomics**: `cli.py::_warn_on_databricks_auth_ambiguity` warns (non-blocking) when `~/.databrickscfg` has multiple profiles on the same host or when `DATABRICKS_TOKEN` is set alongside cfg profiles — both produced opaque SDK errors during testing. - **New deps**: `snowflake-connector-python>=3.6.0`, `snowflake-sqlalchemy>=1.7.0`. ### Caveats/things to watch out for when reviewing - **Post-merge follow-up tracked**: the two `# TODO` comments in `assessments/configure_assessment.py` and `connections/database_manager.py` reference a branch URL for the docs walkthrough (`feature/snowflake-profiler`). After this PR merges and the docs site rebuilds, both must be flipped to the published URL: - `configure_assessment.py` → `https://databrickslabs.github.io/lakebridge/docs/assessment/profiler/snowflake#create-a-snowflake-personal-access-token-pat` - `database_manager.py` (`docs_url` stem) → `https://databrickslabs.github.io/lakebridge/docs/assessment/profiler/snowflake` (anchors are concatenated at usage sites — `#part-3--bypass-the-network-policy-temporary`, `#part-2--create-a-personal-access-token`) - Greppable with: `grep -rn "feature/snowflake-profiler/docs"`. Will be done in a small follow-up PR. - `tests/unit/test_install.py::test_configure_all_override_installation` switched from picking source `0` to `1` because `Prompts.choice` sorts alphabetically — adding `snowflake` shifted `synapse` from index 0 to 1. Worth a sanity-check that the test is asserting the intent we want. - `pipeline.py` now silently swallows failures of optional steps with a warning + default row for `adjusted_rate`. Other optional steps just log; if you'd rather keep the explicit error, flag it. - Migration complexity is intentionally synchronous + slow on big histories (6h default, opt-in unbounded retry). It runs *after* the rest of the extract is on disk, so a Ctrl+C still leaves a usable DuckDB. - PAT auth uses Snowflake's `password` authenticator (PATs are accepted there). External browser auth is wired up but untested in this PR. - New Snowflake deps are pulled in at the project level — they're heavy. Acceptable, but worth confirming. ### Linked issues <!-- Add a Resolves #... line here if this maps to an existing issue. --> ### Functionality - [x] added relevant user documentation (`docs/lakebridge/docs/assessment/profiler/snowflake.mdx` + index update) - [ ] added new CLI command - [x] modified existing command: `databricks labs lakebridge configure-database-profiler` - [x] modified existing command: `databricks labs lakebridge execute-database-profiler` - [x] new profiler source: Snowflake (via `ACCOUNT_USAGE`) ### Tests - [x] manually tested end-to-end against a live Snowflake account (PAT auth, full 365-day lookback, all summary tables produced) - [x] updated `tests/unit/test_install.py::test_configure_all_override_installation` for the new alphabetical source ordering - [x] **added unit tests** (30 tests, all passing locally): - `tests/unit/assessment/test_snowflake_configurator.py` — full credential flow + lookback validation loop. - `tests/unit/assessment/test_snowflake_computations.py` — seeds an in-memory DuckDB extract and asserts all summary tables, mapper sizes, credit math, latest-snapshot storage filter, `extract_timestamp` coverage, and the `duckdb.InterruptException` → `MigrationComplexityTimeout` conversion. - `tests/unit/connections/test_snowflake_connector.py` — `_raise_friendly_connect_error` branches (network-policy + PAT credential errors point at the right docs anchors), unmapped errors propagate unchanged, `parse_snowflake_account` / `validate_snowflake_account` URL handling. - Engine-mocking tests for `_connect` deliberately skipped — would require mocking `snowflake.sqlalchemy` and add little coverage value beyond the friendly-error tests. - [ ] added integration tests (manual run against live Snowflake covers this; full CI integration would require a managed Snowflake account) This pull request and its description were written by Isaac.
## What Adds a `rate_sheet` extract to the Snowflake profiler that pulls the effective per-credit/unit rate from `SNOWFLAKE.ORGANIZATION_USAGE.RATE_SHEET_DAILY` — a 90-day average, scoped to the current account and split by rating type. The same rows also carry the account tier (`SERVICE_LEVEL`) and the billing currency. Also logs the absolute path of the DuckDB extract on successful completion, so users can find the output file (this benefits all profilers, not just Snowflake). ## Why The downstream TCO / value model needs the customer's effective `$/credit` rate and Snowflake tier to convert consumed credits into dollars and to derive their contract discount. The profiler emitted credits but no rate, so the dollar conversion had to be sourced manually. ## Changes - New `resources/assessments/snowflake/rate_sheet.sql` - Register the `rate_sheet` step in the Snowflake `pipeline_config.yml` - Log the extract output path on successful completion in `pipeline.py` ## Access requirement `ORGANIZATION_USAGE` lives in the shared `SNOWFLAKE` database. The existing default role `ACCOUNTADMIN` can read it from an ORGADMIN-enabled account (or grant a custom role the `ORGANIZATION_USAGE_VIEWER` database role). When access isn't available the step simply returns 0 rows and is skipped — the run still succeeds. ## Testing - Ran the Snowflake profiler end-to-end against an ORGADMIN-enabled account; `rate_sheet` populated with tier (Enterprise), region, currency (USD), and effective rate ($3.00/credit). - `tests/unit/assessment/` unit tests pass (87 passed). Classifying usage into the value model (BI/ETL split, discount derivation) is downstream work outside this PR.
Adds a BigQuery source-tech to the Lakebridge profiler. ## What it does Runs 16 region-qualified `INFORMATION_SCHEMA` queries against the customer's BigQuery project(s) (configurable via `project.region` pairs) and writes 12 analysis tables into a local DuckDB file at `~/.databricks/labs/lakebridge_profilers/bigquery_assessment/profiler_extract.db`. Pipeline structure mirrors the existing Synapse / MSSQL / legacy_synapse / Oracle source-techs: - `configure-database-profiler` interactive prompt → writes `credentials.yml` - `execute-database-profiler --source-tech bigquery` → runs `bq_metadata_extract.py` ## Design highlights - **Auth**: ADC-only (`gcloud auth application-default login` or `GOOGLE_APPLICATION_CREDENTIALS` for non-interactive). Service-account impersonation supported via ADC. - **Pair syntax**: `project.region` — matches Google's [fully-qualified resource-path convention](https://cloud.google.com/iam/docs/full-resource-names#bigquery), bulk-entry friendly for multi-project sizing. - **Region qualifier in SQL**: required by BQ for INFORMATION_SCHEMA views per [docs.cloud.google.com](https://docs.cloud.google.com/bigquery/docs/information-schema-intro#region_qualifier) — the client's `location` parameter doesn't substitute for it. - **Shared helpers**: uses `assessments.common.{cli, duckdb_helpers}` from #2431 — no duplication. ## Verification - `make fmt && make lint && make test` clean (1187 unit tests pass, 2 skipped, 3 xfailed). - End-to-end smoke test produces the expected 12 DuckDB tables. ## Follow-up PRs - BigQuery profiler dashboard (Lakeview JSON) — separate PR after this merges. - Integration test + `sandbox_bigquery_*` fixtures — separate PR after this merges. - Continuation of #2445 (closed; see #2449 for the JFrog/OIDC fork-PR tracking). --- ### Review follow-ups Addressed the blocking `table_storage` billing-model split in this PR (commit `60e08f13`; verified against a real BQ project). The non-blocking notes are tracked for a follow-up in #2493: - streaming/write-API profiling-window filter - `timeline_analysis` `bytes_processed` diagnostic - catch `GoogleAPIError` in the per-SQL handler - CHANGELOG note for the profiler-schema default change --------- Co-authored-by: M Abulazm <mohamed.abulazm@databricks.com>
## Changes This PR fixes a bug in the Redshift serverless profiler managed storage extract. ### What does this PR do? - Fixes `rs_managed_storage_gb` aggregation for the Redshift serverless variant. - Files changed - src/databricks/labs/lakebridge/resources/assessments/redshift/serverless/2_rs_managed_storage_gb.sql ### Relevant implementation details `sys_serverless_usage` holds hourly snapshots of allocated storage. The previous `sum(round(data_storage / 1024.0, 2))` accumulated those snapshots, so the reported managed storage grew linearly with the lookback window (e.g., 24 x actual_GB for a one-day extract). Replaced with `round(avg(data_storage) / 1024.0, 2)` so the extract returns the representative allocation. ### Caveats/things to watch out for when reviewing: Affects only the `redshift_serverless` variant. The `provisioned` and `provisioned_multi_az` variants use different source tables (`stv_node_storage_capacity` / placeholder) and are unchanged. Output schema (`set_name VARCHAR`, `rs_managed_storage_gb DOUBLE`) is identical so no validator schema, schema-definition YAML, mock DuckDB fixture, or test changes are required. ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> Resolves #.. ### Functionality - [ ] added relevant user documentation - [ ] added new CLI command - [ ] modified existing command: `databricks labs lakebridge ...` - [ ] ... +add your own ### Tests 1. Manually tested in AWS Sandbox account aws-sandbox-field-eng (332745928618) against a serverless Redshift workspace. - [x] manually tested - [ ] added unit tests - [ ] added integration tests
<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary, they're helpful to illustrate the before and after state --> ### What does this PR do? add source system and report type to user agent extras
## Changes
This PR adds Redshift as a supported profiler assessment platform. It
wires Redshift into the CLI and assessment config, and parameterizes
profiler/validator tests for both synapse and Redshift using mocks (no
live cluster).
### What does this PR do?
- Adds Redshift as a supported platform for the profiler assessment
(alongside Synapse).
- Wires Redshift into the CLI (source technology / transfer) and
assessment constants (_constants.py: variants, config path template,
PLATFORM_TO_SOURCE_TECHNOLOGY_CFG, PROFILER_SOURCE_SYSTEM).
- Parameterizes existing profiler and profiler-validator tests for
synapse and redshift; adds Redshift mock extract and test resources so
all tests run for both platforms without a live Redshift cluster.
- Full list of changed files:
- Source
- src/databricks/labs/lakebridge/assessments/__init__.py
- src/databricks/labs/lakebridge/assessments/_constants.py
- src/databricks/labs/lakebridge/assessments/configure_assessment.py
- src/databricks/labs/lakebridge/assessments/profiler.py
- src/databricks/labs/lakebridge/cli.py
- src/databricks/labs/lakebridge/connections/credential_manager.py
- Tests – integration
- tests/integration/assessments/profiler_extract_utils.py
- tests/integration/assessments/test_profiler.py
- tests/integration/assessments/test_profiler_validator.py
- Tests – resources
- tests/resources/assessments/db_extract_redshift.py
- tests/resources/assessments/pipeline_config_main_redshift.yml
- tests/resources/assessments/redshift_schema_def.yml
### Relevant implementation details
- Tests: Session-scoped mock_redshift_profiler_extract and
PLATFORM_VALIDATOR_CONFIG drive parameterized tests;
db_extract_redshift.py and pipeline_config_main_redshift.yml provide a
script-based Redshift-shaped DuckDB for execution tests with no real
cluster.
### Caveats/things to watch out for when reviewing:
### Linked issues
<!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes,
fixed, resolve, resolves, resolved. See
https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword
-->
Resolves #..
### Functionality
- [ ] added relevant user documentation
- [ ] added new CLI command
- [ ] modified existing command: `databricks labs lakebridge ...`
- [ ] ... +add your own
### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->
1. Redshift mock extract
(tests/integration/assessments/profiler_extract_utils.py)
Added Redshift table definitions: query_view, rs_managed_storage_gb,
rs_nodes (same pattern as Synapse: 3 tables, 1 empty).
Added RedshiftProfilerBuilder and build_mock_redshift_extract() so
validator tests can use a Redshift-shaped DuckDB without a real cluster.
2. Test schema for Redshift
(tests/resources/assessments/redshift_schema_def.yml)
Minimal schema for validator tests: query_view, rs_managed_storage_gb,
rs_nodes.
rs_managed_storage_gb.rs_managed_storage_gb is defined as VARCHAR (mock
has DOUBLE) so test_validate_invalid_schema_check still fails as
intended.
3. Profiler validator tests
(tests/integration/assessments/test_profiler_validator.py)
Added session-scoped mock_redshift_profiler_extract and
PLATFORM_VALIDATOR_CONFIG for platform-specific table names and counts.
All 7 validator tests are parameterized with
@pytest.mark.parametrize("platform", ["synapse", "redshift"]).
Each test runs once per platform; logic is shared, no new test cases.
test_get_profiler_extract_path is unchanged (no platform param).
4. Profiler tests (tests/integration/assessments/test_profiler.py)
All 5 tests parameterized with @pytest.mark.parametrize("platform",
["synapse", "redshift"]).
Execution tests use script-based configs only (no live DB):
Synapse: pipeline_config_main.yml + db_extract.py.
Redshift: pipeline_config_main_redshift.yml + db_extract_redshift.py.
test_profile_execution_config_override copies the platform-specific
script (db_extract.py or db_extract_redshift.py) into a temp dir and
runs it.
5. Redshift script and config (for execution tests without live
Redshift)
tests/resources/assessments/db_extract_redshift.py: standalone script
that creates a DuckDB at --db-path with the same 3 Redshift tables (no
imports from tests).
6. Manually Tested all credential flows on all clusters in AWS Sandbox
account aws-sandbox-field-eng (332745928618)
tests/resources/assessments/pipeline_config_main_redshift.yml: pipeline
config that runs that script.
- [x] manually tested
- [x] added unit tests
- [x] added integration tests
---------
Co-authored-by: M Abulazm <mohamed.abulazm@databricks.com>
## Changes This PR Redshift profiler docs to the assessment platform. ### What does this PR do? - Adds Redshift profiler docs for the assessment (alongside Synapse). - Files changed - .gitignore - docs/lakebridge/docs/assessment/profiler/index.mdx - docs/lakebridge/docs/assessment/profiler/redshift.mdx ### Relevant implementation details File index.mdx changed to add reference to Redshift profiler. File redshift.mdx is added. ### Caveats/things to watch out for when reviewing: ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> Resolves #.. ### Functionality - [ ] added relevant user documentation - [ ] added new CLI command - [ ] modified existing command: `databricks labs lakebridge ...` - [ ] ... +add your own ### Tests 1. Manually Tested all credential flows on all clusters in AWS Sandbox account aws-sandbox-field-eng (332745928618) tests/resources/assessments/pipeline_config_main_redshift.yml: pipeline config that runs that script. - [x] manually tested - [ ] added unit tests - [ ] added integration tests --------- Co-authored-by: dgomez04 <diego.gomez@databricks.com>
Bump version to 0.14.0
## Changes This PR updates the Java requirements for the CLI from 11 to 21. The most recent builds of Morpheus switched from Java 11 to 21, so this is now a requirement. ### Linked issues Context: databrickslabs/morpheus#952 (private) ### Functionality - updated relevant user documentation - modified existing command: `databricks labs lakebridge install-transpile` ### Tests - updated integration tests
<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary, they're helpful to illustrate the before and after state --> ### What does this PR do? Fix integration test assertion after databricks workspace behavior change - before: `` [`system`.`builtin`, `system`.`session`, `catalog`.`schema`] `` - after: `` [`system`.`session`, `system`.`builtin`, `system`.`ai`, `catalog`.`schema`] `` ### Caveats/things to watch out for when reviewing: A more resilient fix (wildcarding the path inside the assertion) was considered and deliberately skipped in favour of the minimal change. ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> Resolves #2515 ### Functionality - [X] update test code ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [ ] manually tested - [ ] added unit tests - [X] added integration tests
<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary, they're helpful to illustrate the before and after state --> ### What does this PR do? Introduce the concept of a variant of a profiler. the user is prompted to select a variant when executing profiler if supported ### Relevant implementation details - **Variant model** — add `SOURCE_SYSTEM_VARIANTS` ; `PROFILER_SOURCE_SYSTEM` is now the flat list of real source systems. Remove `REDSHIFT_VARIANTS`, and `source_system_family()`. - **Pipeline resolution** — Remove `SOURCE_SYSTEM_TO_PIPELINE_CFG`, replace the path map with convention-based `get_pipeline(source_system, variant)` → `resources/assessments/<source>/<variant>/pipeline_config.yml`. - **CLI** — `execute-database-profiler` accepts `--variant`. `parse_profiler_variant` prompts for a variant when the source has variants and none is given. ### Caveats/things to watch out for when reviewing: - `configure-profiler` and `test-profiler-connection` currently do not accept variant. - integration with desktop app needs to be defined ### Functionality - [ ] added relevant user documentation - [ ] added new CLI command - [X] modified existing command: `databricks labs lakebridge execute-database-profiler` ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [ ] manually tested - [X] added unit tests - [ ] added integration tests
<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary, they're helpful to illustrate the before and after state --> ### What does this PR do? Remove the profiler dashboards ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> Resolves #2473 ### Functionality - [X] removed existing command: `databricks labs lakebridge create-profiler-dashboard`
#2512) <!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary, they're helpful to illustrate the before and after state --> ### What does this PR do? Fix connection testing implementation that relied on magic if conditions and duplicate config that was not modeled correctly and could be broken easily ### Relevant implementation details Introduce `AssessmentConfigurator.test_connection()` to fetch the saved creds and delegates to `_check_connection()`: - base → `DatabaseManager` health check (snowflake, oracle, mssql, legacy_synapse, redshift) - `ConfigureSynapseAssessment` → `validate_synapse_pools` (per SQL pool) - `ConfigureBigQueryAssessment` → `validate_bigquery_pairs` (per project/region) This removes the `if source_tech == "synapse"` special-case in `test-profiler-connection` and the hand-maintained `CONNECTOR_REQUIRED` source-system flag which can be derived automatically ### Functionality - [X] modified existing command: `databricks labs lakebridge test-profiler-connection` - [X] modified existing command: `databricks labs lakebridge configure-database-profiler` ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [ ] manually tested - [X] added unit tests - [ ] added integration tests
…ests (#2521) ## Summary This PR addresses [#2479](#2479) by migrating reconcile integration tests to use the real `pytester` `ws` fixture instead of `mock_workspace_client`. ### Changes made - Replaced `mock_workspace_client` fixture usage with `ws` in: - `tests/integration/reconcile/test_recon_capture.py` - `tests/integration/reconcile/test_aggregates_recon_capture.py` - `tests/integration/reconcile/query_builder/test_execute.py` - Removed now-unnecessary assignments like `ws = mock_workspace_client`. - Renamed a local mock variable in `test_execute.py` (`mock_workspace_client` -> `ws_client_mock`) to avoid fixture-style confusion while keeping test behavior unchanged. ## Why Integration tests now run with a real Spark workspace context, so `WorkspaceClient` should come from the shared `ws` fixture for consistency with the current integration test model. Fixed issue: #2479 ## Test plan - [x] `python -m pytest tests/integration/reconcile/test_aggregates_recon_capture.py --collect-only -q` - [x] `python -m pytest tests/integration/reconcile/test_recon_capture.py --collect-only -q` - [x] `python -m pytest tests/integration/reconcile/query_builder/test_execute.py --collect-only -q` - [ ] Run full integration test execution in CI / Databricks-enabled environment
## What does this PR do? Adds **Teradata profiler ** to Lakebridge. The profiler connects to a Teradata system, runs a set of extraction queries defined entirely in configuration, and produces a local DuckDB extract. Teradata ships in two **variants** — `core` and `pdcr` — selected at extraction time via the existing `--variant` flag, each backed by its own pipeline configuration. ## Relevant implementation details - **Source registration** (`assessments/_constants.py`): registers `teradata` in `PROFILER_SOURCE_SYSTEM` and its variants (`core`, `pdcr`) in `SOURCE_SYSTEM_VARIANTS`, mirroring Redshift. - **Connector** (`connections/database_manager.py`): `TeradataConnector` (SQLAlchemy `teradatasql` driver) + factory registration; supports a configurable `port` (default `1025`). Adds the `teradatasqlalchemy` dependency. - **Credential dialog** (`assessments/configure_assessment.py`): `ConfigureTeradataAssessment` collects host / port / user / password / default database, with `local | env` secret-vault handling. - **Extraction resources** (`resources/assessments/teradata/`): per-variant `core/pipeline_config.yml` and `pdcr/pipeline_config.yml` plus shared SQL/DDL assets for system info, nodes info, disk/usage aggregates, storage, object types, user databases, UDFs, PDCR aggregates, and DBQL-core. - **Variant behavior**: the `core` variant extracts `td_dbql_core_info_extract` (DBQL-core); the `pdcr` variant extracts `td_pdcr_info_agg_extract` and `td_pdcr_sp_exe_info_agg_extract` (PDCR aggregated history). Selected via `--source-tech teradata --variant core|pdcr`. - **Docs** (`docs/.../assessment/profiler/teradata.mdx` + index): Teradata profiler guide covering prerequisites, the two variants, and how to choose between them. This PR makes **no changes** to the generic CLI, installer, deployment, or dashboard code — Teradata is supported purely through the existing config-driven extractor + variant framework, mirroring Redshift/Oracle. ## Caveats / things to watch out for when reviewing - Scope is **extractor only**: no Unity Catalog ingestion, no Lakeview dashboard publishing, no `create-profiler-dashboard` CLI command, and no SQLTextInfo masking/tagging in this PR. (The DBQL-core extract redacts `SQLTextInfo` to `[REDACTED]` at extraction time, so raw query text never leaves the source.) - `teradatasqlalchemy` is added as a runtime dependency. - Variant selection (`core` vs `pdcr`) is explicit via `--variant`; if PDCR (`PDCRINFO`) is unavailable, use `--variant core`. ## Linked issues Resolves #.. ## Functionality - [x] added relevant user documentation - [ ] added new CLI command - [ ] modified existing command - [x] added Teradata profiler extraction resources (config-driven; no framework changes) ## Tests - [x] manually tested (extractor CLI run end-to-end) - [x] added unit tests - `test_teradata_profiler.py` — variant (`core`/`pdcr`) pipeline-config resolution + missing/invalid-config handling - `test_teradata_variants.py` — variant → pipeline-config path mapping - `test_assessment.py` — `ConfigureTeradataAssessment` dialog + factory - `test_database_manager.py` — `TeradataConnector` - [ ] added integration tests `make test` green (1276 passed, aside from an unrelated pre-existing `test_cli_analyze` timeout flake); `make fmt` / ruff / mypy / pylint 10.00/10.
…profiler (#2531) ## Summary This pull request aligns the four schema strings to match the `SELECT` column order in `queries.py`. Because rows land positionally into the declared schema, queries on these tables against an extract were silently reading the wrong column: - `dedicated_tables` and `serverless_tables` had `TABLE_NAME` and `TABLE_SCHEMA` swapped relative to what `SynapseQueries.list_tables()` actually returns (`TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE, POOL_NAME`). - `dedicated_views` and `serverless_views` had `CHECK_OPTION` and `IS_UPDATABLE` listed first, when the query returns `TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, CHECK_OPTION, IS_UPDATABLE, VIEW_DEFINITION, POOL_NAME`. Closes #2530
…ing` (#2534) `AUTOMATIC_CLUSTERING_HISTORY` includes clustering activity on Snowflake-owned system databases (`SNOWFLAKE`, `UTIL_DB`, `SNOWFLAKE_SAMPLE_DATA`). These are not customer workloads and their inclusion inflates the clustering credit picture in the extract. `database_objects.sql` already excludes these same databases in every `UNION` branch. This pull requst adds the equivalent filter to `automatic_clustering.sql` to bring it in line.
…#2535) ## Summary - Add intermediate `write_and_read_df_with_volumes` checkpoints in reconcile computation paths that were still missing materialization (marked with TODOs in #2257). - Materialize threshold mismatch rows, aggregate per-rule outputs, and sampled mismatch capture DataFrames before downstream `count()` / sampling work. ## Motivation Large Spark plans in reconcile can retain long lineages and cause backpressure. The reconcile runtime already materializes some intermediate results (e.g. after source/target join); this PR fills the remaining gaps called out in [#2257](#2257). ## Changes | Phase | Location | Change | |---|---|---| | Threshold comparison | `reconciliation._compute_threshold_comparison` | Materialize filtered `mismatched_df` before `count()` | | Aggregate per-rule | `compare.reconcile_agg_data_per_rule` | Thread `persistence` and materialize `mismatch`, `missing_in_src`, `missing_in_tgt` | | Mismatch capture (sampling) | `compare.capture_mismatch_data_and_columns` | Require `persistence`; wired from `Reconciliation._get_mismatch_data` | Fixed issue: #2257 **Note:** These are temporary intermediate writes via `ReconIntermediatePersist` (UC volume / parquet), not changes to final `ReconCapture` MAIN/METRICS/DETAILS output tables. Temp data is still cleaned up in `trigger_recon` / `trigger_recon_aggregates`. ## Test plan - [x] CI integration reconcile tests (`test_compare.py`, `test_aggregates_reconcile.py`, `test_execute.py`) — covers aggregate per-rule, mismatch capture, and threshold paths via existing e2e coverage - [ ] Manual smoke on Databricks (optional): run reconcile job and confirm intermediate volume writes under `metadata_config.volume`
…2503) ## Summary - Refactored reconcile runtime internals to use `SourceConnectionConfig` and `TargetConnectionConfig` instead of `DatabaseConfig`. - Updated `Reconciliation`, `TriggerReconService`, and `TriggerReconAggregateService` to pass source/target connection config objects explicitly. - Updated integration tests in reconcile query-builder and aggregate/capture paths to align with the new config model. - Removed remaining `DatabaseConfig` usage from `tests/integration/reconcile/test_recon_capture.py`. ## Backward Compatibility - Preserved `ReconCapture` constructor compatibility for legacy `DatabaseConfig`-style callers during the migration period. ## Validation - `uv run python -m pytest tests/integration/reconcile/test_recon_capture.py -q` - `23 skipped` - `uv run python -m pytest tests/integration/reconcile/query_builder/test_execute.py -q` - `1 passed, 15 skipped` - `uv run python -m pytest tests/integration/reconcile/test_aggregates_reconcile.py -q` - `4 passed, 2 skipped` - `uv run python -m pytest tests/integration/reconcile/test_aggregates_recon_capture.py -q` - `1 skipped` ## Notes - `uv.lock` changes from local environment sync were intentionally excluded from this PR. ### Linked issues Fixed issue : #2417 ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [ ] manually tested - [ ] added unit tests - [ ] added integration tests
…s. (#2525) ### What does this PR do? This PR updates the `lock-dependencies` mechanism so that scrubbing also takes care of the CDN URLs that are embedded in there; some mirrors/proxies, in addition to rewriting the registry, also rewrite the URLs. ### Caveats/things to watch out for when reviewing: The `lint-uv` check will fail, but can be safely ignored/overridden to merge. ### Linked issues Relates: databrickslabs/blueprint#392 ### Functionality - updates `make lock-dependencies` ### Tests - manually tested: produces the `uv.lock` included in this PR. - automated CI checks the resulting lock-file.
<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary, they're helpful to illustrate the before and after state --> ### What does this PR do? Suppresses the `FutureWarning` from `google.api_core` ### Caveats/things to watch out for when reviewing: - Suppression is scoped to these two import sites (the only ones importing google in the repo). The warning fires only on first import of `google.api_core` process-wide. - Once the project moves to Python 3.11+, the warning stops firing on its own and these `with` blocks become harmless no-ops that can be removed. ### Functionality - [ ] added relevant user documentation - [ ] added new CLI command - [ ] modified existing command: `databricks labs lakebridge ...` - [ ] ... +add your own ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [ ] manually tested - [ ] added unit tests - [ ] added integration tests
<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary, they're helpful to illustrate the before and after state --> ### What does this PR do? * Add missing extract source otherwise the profiler doesnt load ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [X] manually tested - [ ] added unit tests - [ ] added integration tests
## What
Adds **BigQuery** as a reconcile source, at parity with the other
sources (`schema` / `row` / `data` / `all` / `aggregate`). BigQuery was
already supported by the transpiler and profiler; this closes the gap
for reconcile.
## How it works
BigQuery uses the same **Lakehouse Federation `remote_query`** path as
Snowflake/Oracle/etc. — no new dependencies. `BigQueryDataSource` uses
backtick-quoted 3-part `project.dataset.table` names and reads metadata
from `INFORMATION_SCHEMA.COLUMNS`. sqlglot already ships a BigQuery
dialect, so query generation and schema comparison are reused unchanged.
## Changes
- **Connector** `reconcile/connectors/bigquery.py`
(`BigQueryDataSource`); registered in `ReconSourceType` and
`source_adapter.create_adapter`; install prompt + `recon_capture`
display name.
- **Schema type handling**: the connector's `INFORMATION_SCHEMA` query
canonicalizes the few BigQuery types sqlglot can't bridge to Databricks
(`BIGNUMERIC`→`string`, bare `NUMERIC`→`decimal(38,9)`, `TIME`→`string`,
`JSON`→`variant`, `RANGE<T>`→`struct<…>`); everything else is left to
sqlglot.
- **Row hashing**: adds a BigQuery hash algorithm (`TO_HEX(SHA256())`,
matching Databricks `sha2(...,256)`) and a **scale-aware decimal
transform** (`FORMAT('%.<scale>f', col)`) so BigQuery's
trailing-zero-stripped numeric strings match Spark's scale-padded
`DECIMAL` strings.
- **Docs + tests**: supported-sources row, a BigQuery config tab (with a
compute note), connector tests, a type-coverage guardrail test, and an
adapter test.
## Testing
- `make fmt` / `make lint` (pylint 10.0/10.0) and `make test` — green
(1319 passed).
- **End-to-end on a real workspace**: a BigQuery source (UC Federation
connection) reconciled against an identical Databricks copy via the
deployed reconcile **job** — `schema`, `row`, and `data` all matched (0
mismatches / 0 missing).
## Notes / limitations
- **Compute**: BigQuery reads use `remote_query`, which requires
**Databricks Runtime 17.3+** or **serverless** compute (the reconcile
job's default cluster may run an older runtime — point it at a DBR 17.3+
cluster via `job_overrides.existing_cluster_id`, or run serverless).
This applies to all Lakehouse Federation reconcile sources. Documented
in the config tab.
- `INTERVAL` maps to two Databricks columns, which the 1:1 schema
comparison can't represent — surfaces as a visible mismatch
(documented).
---------
Co-authored-by: M Abulazm <mohamed.abulazm@databricks.com>
## Feedback Concurrency numbers are currently unreliable when the same pool name exists across multiple workspaces. Because the Synapse session/request extract tables only track `POOL_NAME` + `EXTRACT_TS`, data from different workspaces collides once it lands in the extract, making it impossible to differentiate them. ## Fix Added a `WORKSPACE_NAME` column to uniquely identify extracts. This value is pulled from the workspace config's name property, which is already available in both extracts, meaning no extra API calls were required. Downstream consumers can now accurately group metrics by `(WORKSPACE_NAME, POOL_NAME)` when required. This pull request and its description were written by Isaac.
<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary, they're helpful to illustrate the before and after state --> ### What does this PR do? The mssql profiler previously assessed only a single database per run. This PR adds automatic detection between single-database and multi-database profiling based on the SQL Server edition and the configured database, and introduces a reusable variant-resolution mechanism to support it. **mssql auto-detection** - A configured database scopes profiling to that one database (`single_db`). - A blank database lets the edition decide: Azure SQL Database (`EngineEdition = 5`) falls back to `single_db`; on-prem SQL Server and Azure SQL Managed Instance profile every accessible database (`multi_db`). - Credential configuration now accepts a blank database name (blank = all databases) for mssql. **Multi-database extracts** (`mssql/multi_db/`, new) - New pipeline config and SQL extracts that fan out across every online, accessible user database (system DBs excluded), tagging each row with a `database_name` column. - UNION-based dynamic SQL for catalog views (tables, views, columns, routines, indexed views); cursor + `USE` for database-scoped DMVs (`db_sizes`, `table_sizes`) that can't use three-part naming. - All extracts and DDL gain a `DATABASE_NAME` column. The original config moves to `mssql/single_db/`. ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> Progresses #2483 ### Functionality - [ ] added relevant user documentation - [ ] added new CLI command - [X] modified existing command: `databricks labs lakebridge execute-database-profiler --source-tech mssql` - [ ] ... +add your own ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [X] manually tested: Azure SQL and On-prem SQL Server on Docker - [X] added unit tests - [ ] added integration tests
…#2497) <!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST --> ## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary, they're helpful to illustrate the before and after state --> ### What does this PR do? Adds a `sampling_options` block to the reconcile table config so users can control how many sample rows are captured per result set (mismatch, missing-in-source, missing-in-target) instead of the fixed 50. Previously large sample sizes on Databricks blew up the UNION-based sampling query against Spark Connect; the Databricks path now uses a temp view so it scales. ### Relevant implementation details - `SamplingSpecifications` now defaults to `count` / `50` and validates the value (coerces to int, rejects bools/non-numeric, keeps `fraction` disabled). `Table.get_max_sample_size()` exposes the configured value. - Sample-size bounds are applied in `NormalizeReconConfigService` (engine-aware), not in the samplers: - Databricks source: clamped to `[1, 50_000]`. - Non-Databricks source: honored up to `400`, larger values capped at 400 with a warning. - `value <= 0` falls back to 50. - `SamplingQueryBuilder` registers the recon keys as a temp view for the Databricks engine (`SELECT * FROM recon_keys_<uuid>`) so the generated SQL stays small regardless of sample size; other dialects keep the inline-UNION derived table. - The old fixed `[50, 400]` clamp was removed from the samplers. ### Caveats/things to watch out for when reviewing: - Non-Databricks sources are intentionally capped at 400 (matches prior behaviour); only Databricks honors larger sizes up to 50,000. - `fraction` sampling type stays disabled and raises a config error. ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> Resolves #2559 ### Functionality - [x] added relevant user documentation - [ ] added new CLI command - [ ] modified existing command: `databricks labs lakebridge ...` - [x] new config field: `sampling_options` on the reconcile Table config ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [x] manually tested — end-to-end on Databricks source (default → 50, 100000 → capped to 50000) and Redshift source (default → 50, 500 → capped to 400 with warning), covering column mapping, drop columns and schema validation - [x] added unit tests - [x] added integration tests
<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST -->
## Changes
<!-- Summary of your changes that are easy to understand. Add
screenshots when necessary, they're helpful to illustrate the before and
after state -->
### What does this PR do?
`Profiler._execute` (profiler.py:93-95) and `PipelineClass.execute()`
(pipeline.py:67,76) both correctly raise RuntimeError on step failures.
The exception gets swallowed one level up, in blueprint's `App._route`:
```
# databricks/labs/blueprint/cli.py:119-126
try:
cmd.fn(**kwargs)
except Exception as err:
logger.error(f"{err.__class__.__name__}: {err}")
```
The error is logged and the process falls off the end of main — so the Python entrypoint exits 0 on any profiler failure, and the databricks CLI in turn reports success. This affects every lakebridge command that signals failure with a plain exception, not just the profiler.
There is an escape hatch already used: SystemExit derives from BaseException, so it sails through blueprint's except Exception and terminates the process with exit code 1. That's exactly why `test-profiler-connection` raises SystemExit at cli.py:1206-1217.
### Linked issues
<!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword -->
Resolves #..
### Functionality
- [ ] added relevant user documentation
- [ ] added new CLI command
- [X] modified existing command: `databricks labs lakebridge execute-database-profiler`
### Tests
<!-- How is this tested? Please see the checklist below and also describe any other relevant tests -->
- [ ] manually tested
- [ ] added unit tests
- [ ] added integration tests
…ines schema (#2572) ## Summary Fixes two customer-reported failures when running the Synapse profiler on a serverless-only workspace. ### 1. `workspace_info` fails when there are no dedicated SQL pools **Root cause:** When `list_sql_pools()` returns an empty list, `pd.json_normalize([])` produces a DataFrame with zero rows and zero columns. `save_to_duckdb` called `conn.register()` before checking for column metadata, so DuckDB raised `Need a DataFrame with at least one column` and aborted the entire `workspace_info` step — even though an empty pool list is a valid outcome. **Fix:** Reorder `_save_overwrite` in `duckdb_helpers` to check for zero-column DataFrames before registering with DuckDB. When no schema is provided and the table already exists, truncate it (clearing stale data on re-run); otherwise warn and skip table creation. This is a systemic fix that also protects other workspace resources that may return empty lists. ### 2. `serverless_sqlpool_extract` fails on `serverless_routines` **Root cause:** PR #2289 switched serverless routines extraction to query `sys.objects` via `list_serverless_routines()` (7 columns) because `information_schema.routines` is not supported on serverless pools. When pinned schemas were added in #2431, `serverless_routines` was copied from `dedicated_routines` (23 columns, matching `information_schema.routines`). The resulting column count mismatch caused `Binder Error: table serverless_routines has 23 columns but 7 values were supplied`. **Fix:** Update `SYNAPSE_SCHEMAS["serverless_routines"]` to match the 7 columns returned by `list_serverless_routines()`: `ROUTINE_SCHEMA`, `ROUTINE_NAME`, `ROUTINE_TYPE`, `CREATED`, `LAST_ALTERED`, `ROUTINE_DEFINITION`, `POOL_NAME`. ## Test plan - [x] `uv run pytest tests/unit/assessment/test_duckdb_helpers.py tests/unit/assessment/test_synapse_schemas.py` - [x] New tests cover zero-column overwrite (skip + truncate), empty schema-backed tables, and serverless routines schema alignment with insert
…m `QUERY_TEXT` (#2567) ### Summary This pull request addresses the massive Snowflake profiler extract sizes on high-volume accounts by decoupling metrics tracking from raw SQL text query history. We preserve high-value metrics while drastically reducing the bloat. ### Key Changes - **Omit `QUERY_TEXT` from full-window `query_history`.** Prevents multi-gigabyte SQL text extraction while keeping the exact `QUERY_TYPE` and performance metrics. - **Introduce `query_samples`.** Extracts a random sample of up to 10,000 rows containing `QUERY_TEXT` and `QUERY_ID` (to join back to `query_history` for downstream migration complexity analysis). - **Drop unused `user_activity`.** Removed the `LOGIN_HISTORY`dump, which is no longer used by `lakebridge_profilers` or GVP inputs and was contributing millions of rows of secondary bloat. ### Measured Impact On a sample ~20M-row extract, omitting the raw SQL text yielded dramatic storage and network transfer savings: Metric | Before | After | Change -- | -- | -- | -- Extract Size | ~28 GiB | ~1 GiB | ~96% reduction (28× smaller) **Note.** Workload spend-mix classification relies on `QUERY_TYPE`, not `QUERY_TEXT`, meaning downstream classification accuracy remains unaffected. ### Follow-Ups - [ ] Updated `lakebridge-profilers`' `migration_complexity.sql` to read from `query_samples` instead of `query_history.QUERY_TEXT`. ### Tests - [ ] Execute `execute-database-profiler` on Snowflake staging/test accounts and verified: - [ ] `query_history` no longer contains `query_text` - [ ] `query_samples` is limited to $\le$ 10k rows and contains the text. - [ ] `user_activity` table is completely gone. - [ ] Confirm that `lakebridge-profilers` (after changes) spend-mix path still functions correctly using `QUERY_TYPE` from `query_history`. Closes #2532
Bump version to 0.14.1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
17 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
What does this PR do?
This PR aggregates critical feature additions, ecosystem expansions, and maintenance updates culminating in the v0.13.0 milestone. The primary focus of this work is expanding the capabilities of the reconciliation (recon) engine and profilers to support diverse external data sources via Lakehouse Federation, while modernizing the project's dependency baseline and CI/CD infrastructure.
Key Highlights:
Reconciliation Engine (recon) Enhancements: Migrated integration tests from local environments directly to Databricks. Added foreign connections as a reconciliation source and resolved critical schema fetch failures on Foreign Catalogs.
New Profiler Implementations: Added native support for SQL Server Profiler, Legacy SQL DW Profiler, Oracle Profiler, and initiated Redshift connection/credential support.
Ecosystem & Runtime Upgrades: Updated the project baseline to support Python 3.14 and updated critical linting/formatting tools (e.g., black bump to 26.3.1).
CLI Additions: Added the --switch-config-path parameter to the llm-transpile command and implemented DatabaseConnector#health_check().
Enterprise Features: Implemented Maven and PyPI mirror support during deployment/installation phases and fully revamped user-facing documentation.
Relevant implementation details
Testing Infrastructure: Created a dedicated, isolated test cluster for recon end-to-end integration tests (databrickslabs#2453) to replace brittle local test setups.
Dependency Pruning: Removed cryptography as an explicit dependency, relying on downstream abstractions to minimize surface vulnerability (databrickslabs#2458).
Storage Consolidation: Collapsed DuckDB write helpers and standardized shared profiler infrastructure internally (databrickslabs#2093).
CI/CD: Explicitly marked non-deployment environments in GitHub Actions to optimize pipeline runners (databrickslabs#2443).
Caveats/things to watch out for when reviewing:
Large File Diff: This PR contains changes across 220 files due to extensive lockfile linting, dependency updates, and documentation revamps. Focus reviews on the core connector logic in recon and the new profiler modules.
Python 3.14 Support: Ensure that runtime syntax additions or dependency changes do not break backward compatibility with older supported Python versions.
Database Drivers: New profiles (Oracle, SQL Server, Redshift) introduce specific connector configurations—verify that network topologies and driver handshakes behave safely during connection failures (validated by the updated non-zero exit behavior in the test profiler connection).
Linked issues
Resolves databrickslabs#2151, Resolves databrickslabs#2187, Resolves databrickslabs#2333, Resolves databrickslabs#2339, Resolves databrickslabs#2362, Resolves databrickslabs#2365, Resolves databrickslabs#2421, Resolves databrickslabs#2434, Resolves databrickslabs#2437, Resolves databrickslabs#2441, Resolves databrickslabs#2442, Resolves databrickslabs#2443, Resolves databrickslabs#2444, Resolves databrickslabs#2451, Resolves databrickslabs#2453, Resolves databrickslabs#2457, Resolves databrickslabs#2458
Functionality
[x] added relevant user documentation
[x] added new CLI command (e.g., DatabaseConnector#health_check())
[x] modified existing command: databricks labs lakebridge llm-transpile --switch-config-path
[x] Added SQL Server, Legacy SQL DW, Oracle, and Redshift Profilers.
Tests
[x] manually tested
[x] added unit tests
[x] added integration tests (Migrated to dedicated Databricks E2E cluster)