Merge branch 'main' into meetup-connector

Author: akashverma0786

.github/pull_request_template.md

@@ -5,19 +5,21 @@ Unless your change is trivial, please create an issue to discuss the change befo
 
 ### Describe your changes:
 
-Fixes <issue-number>
+Fixes #<issue-number>
+<!--
+Linking an issue is REQUIRED. Replace <issue-number> with the GitHub issue number this PR addresses
+(e.g., `Fixes #12345`). GitHub will auto-link it. If no issue exists, please open one first so the
+problem and design can be discussed before review.
+-->
 
 <!--
 Short blurb explaining:
 - What changes did you make?
 - Why did you make them?
-- How did you test your changes?
 -->
 
 I worked on ... because ...
 
-<!-- For frontend related change, please add screenshots and/or videos of your changes preview! -->
-
 #
 ### Type of change:
 <!-- You should choose 1 option and delete options that aren't relevant -->
@@ -27,13 +29,90 @@ I worked on ... because ...
 - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
 - [ ] Documentation
 
+#
+### High-level design:
+<!--
+REQUIRED for large PRs (new features, refactors, breaking changes, or anything touching >5 files).
+Skip for small bug fixes and trivial changes.
+
+Cover:
+- Architecture / approach you took and why
+- Key components or files added/changed and how they interact
+- Alternatives considered and why you rejected them
+- Any migration, backward-compatibility, or rollout concerns
+- Diagrams or links to design docs / RFCs if available
+-->
+
+N/A — small change. <!-- Or fill in the design above -->
+
+#
+### Tests:
+
+#### Use cases covered
+<!--
+List the user-visible scenarios this PR exercises. Example:
+- User with Admin role can create a Glossary Term with a parent term
+- Ingestion run for Snowflake correctly extracts row counts for partitioned tables
+-->
+
+#### Unit tests
+<!--
+- [ ] I added unit tests for the new/changed logic.
+- Files added/updated:
+- Coverage on changed classes (run `mvn jacoco:report` for backend, `yarn test:coverage` for UI,
+  `make unit_ingestion` for ingestion). Target is 90% line coverage on changed classes.
+- Coverage %: <e.g., 92% on EntityRepository.java>
+-->
+
+#### Backend integration tests
+<!--
+- [ ] I added integration tests in `openmetadata-integration-tests/` for new/changed API endpoints.
+- [ ] Not applicable (no backend API changes).
+- Files added/updated:
+-->
+
+#### Ingestion integration tests
+<!--
+- [ ] I added/updated ingestion integration tests for connector changes.
+- [ ] Not applicable (no ingestion changes).
+- Files added/updated:
+-->
+
+#### Playwright (UI) tests
+<!--
+- [ ] I added Playwright E2E tests under `openmetadata-ui/.../ui/playwright/` for UI changes.
+- [ ] Not applicable (no UI changes).
+- Files added/updated:
+-->
+
+#### Manual testing performed
+<!--
+List the manual test steps you performed before requesting review. Example:
+1. Started local stack via `./docker/run_local_docker.sh -m ui -d mysql`
+2. Logged in as admin, created entity X, verified Y appears in the UI
+3. Triggered ingestion for Snowflake source, confirmed lineage edges in the explore page
+-->
+
+#
+### UI screen recording / screenshots:
+<!--
+REQUIRED for any PR that changes the UI. Drag-and-drop a short screen recording (.mov / .mp4 / .gif)
+demonstrating the change end-to-end, plus before/after screenshots where relevant.
+Mark "Not applicable" if there are no UI changes.
+-->
+
+Not applicable. <!-- Or attach recording/screenshots above -->
+
 #
 ### Checklist:
 <!-- add an x in [] if done, don't mark items that you didn't do !-->
 - [x] I have read the [**CONTRIBUTING**](https://docs.open-metadata.org/developers/contribute) document.
 - [ ] My PR title is `Fixes <issue-number>: <short explanation>`
-- [ ] I have commented on my code, particularly in hard-to-understand areas. 
+- [ ] My PR is linked to a GitHub issue via `Fixes #<issue-number>` above.
+- [ ] I have commented on my code, particularly in hard-to-understand areas.
 - [ ] For JSON Schema changes: I updated the migration scripts or explained why it is not needed.
+- [ ] For UI changes: I attached a screen recording and/or screenshots above.
+- [ ] I have added tests (unit / integration / Playwright as applicable) and listed them above.
 
 <!-- Based on the type(s) of your change, uncomment the required checklist 👇 -->
 

.github/workflows/py-cli-e2e-tests.yml

@@ -18,7 +18,7 @@ on:
       e2e-tests:
         description: "E2E Tests to run"
         required: True
-        default: '["bigquery", "dbt_redshift", "metabase", "mssql", "mysql", "redash", "snowflake", "tableau", "python-unittests", "python-integration", "redshift", "quicksight", "datalake_s3", "postgres", "oracle", "athena", "bigquery_multiple_project"]'
+        default: '["bigquery", "dbt_redshift", "metabase", "mssql", "mysql", "redash", "snowflake", "tableau", "python-unittests", "python-integration", "redshift", "quicksight", "datalake_s3", "postgres", "oracle", "athena", "bigquery_multiple_project", "exasol"]'
       debug:
         description: "If Debugging the Pipeline, Slack and Sonar events won't be triggered [default, true or false]. Default will trigger only on main branch."
         required: False
@@ -45,7 +45,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        e2e-test: ${{ fromJSON(inputs.e2e-tests || '["bigquery", "dbt_redshift", "metabase", "mssql", "mysql", "redash", "snowflake", "tableau", "python-unittests", "python-integration", "redshift", "quicksight", "datalake_s3", "postgres", "oracle", "athena", "bigquery_multiple_project"]') }}
+        e2e-test: ${{ fromJSON(inputs.e2e-tests || '["bigquery", "dbt_redshift", "metabase", "mssql", "mysql", "redash", "snowflake", "tableau", "python-unittests", "python-integration", "redshift", "quicksight", "datalake_s3", "postgres", "oracle", "athena", "bigquery_multiple_project", "exasol"]') }}
     environment: test
 
     steps:

bootstrap/sql/migrations/native/1.12.7/mysql/postDataMigrationSQLScript.sql

@@ -0,0 +1 @@
+-- Placeholder for 1.12.7 MySQL post-data migration script

bootstrap/sql/migrations/native/1.12.7/mysql/schemaChanges.sql

@@ -0,0 +1,4 @@
+-- Placeholder for 1.12.7 MySQL schema changes
+-- The Postgres-side fix for #27158 has no MySQL counterpart: MySQL's
+-- 1.11.0 indexes were already non-partial (no partial-index syntax in
+-- MySQL), so the regression that hit Postgres did not affect MySQL.

bootstrap/sql/migrations/native/1.12.7/postgres/postDataMigrationSQLScript.sql

@@ -0,0 +1 @@
+-- Placeholder for 1.12.7 PostgreSQL post-data migration script

bootstrap/sql/migrations/native/1.12.7/postgres/schemaChanges.sql

@@ -0,0 +1,32 @@
+-- Issue #27158: tag_usage seq-scan on Postgres. #24063 dropped the
+-- `state = 1` predicate that 1.11.0's partial indexes required.
+-- Fix: add single-col indexes on the `_lower` columns, and drop the
+-- `WHERE state = 1` filter from the partials so changes can't invalidate them.
+
+DROP INDEX CONCURRENTLY IF EXISTS idx_tag_usage_targetfqnhash_lower_pattern;
+CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_tag_usage_targetfqnhash_lower_pattern
+ON tag_usage (targetfqnhash_lower text_pattern_ops);
+
+DROP INDEX CONCURRENTLY IF EXISTS idx_tag_usage_tagfqn_lower_pattern;
+CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_tag_usage_tagfqn_lower_pattern
+ON tag_usage (tagfqn_lower text_pattern_ops);
+
+DROP INDEX CONCURRENTLY IF EXISTS idx_tag_usage_target_prefix_covering;
+CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_tag_usage_target_prefix_covering
+ON tag_usage (source, targetfqnhash_lower text_pattern_ops)
+INCLUDE (tagFQN, labelType, state);
+
+DROP INDEX CONCURRENTLY IF EXISTS idx_tag_usage_tagfqn_prefix_covering;
+CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_tag_usage_tagfqn_prefix_covering
+ON tag_usage (source, tagfqn_lower text_pattern_ops)
+INCLUDE (targetFQNHash, labelType, state);
+
+DROP INDEX CONCURRENTLY IF EXISTS idx_tag_usage_join_source;
+CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_tag_usage_join_source
+ON tag_usage (tagFQNHash, source)
+INCLUDE (targetFQNHash, tagFQN, labelType, state);
+
+CREATE EXTENSION IF NOT EXISTS pg_trgm;
+DROP INDEX CONCURRENTLY IF EXISTS gin_tag_usage_targetfqn_trgm;
+CREATE INDEX CONCURRENTLY IF NOT EXISTS gin_tag_usage_targetfqn_trgm
+ON tag_usage USING GIN (targetFQNHash gin_trgm_ops);

bootstrap/sql/migrations/native/1.13.0/mysql/schemaChanges.sql

@@ -321,3 +321,9 @@ ALTER TABLE search_index_server_stats
 -- already declared `CHARACTER SET ascii COLLATE ascii_bin`, a binary
 -- collation that lets the existing unique B-tree on `fqnHash` answer LIKE
 -- prefix predicates directly. No change required on the MySQL side.
+
+-- MCP OAuth: state parameter is opaque per RFC 6749 §4.1.1 and some clients (notably the
+-- Databricks MCP Proxy) send tokens longer than 255 characters. Widen mcp_state to TEXT to
+-- avoid INSERT failures on /mcp/authorize redirects.
+ALTER TABLE mcp_pending_auth_requests
+    MODIFY COLUMN mcp_state TEXT;

bootstrap/sql/migrations/native/1.13.0/postgres/schemaChanges.sql

@@ -474,3 +474,9 @@ CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_spreadsheet_entity_fqnhash_pattern
 DROP INDEX CONCURRENTLY IF EXISTS idx_worksheet_entity_fqnhash_pattern;
 CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_worksheet_entity_fqnhash_pattern
     ON worksheet_entity (fqnHash text_pattern_ops);
+
+-- MCP OAuth: state parameter is opaque per RFC 6749 §4.1.1 and some clients (notably the
+-- Databricks MCP Proxy) send tokens longer than 255 characters. Widen mcp_state to TEXT to
+-- avoid INSERT failures on /mcp/authorize redirects.
+ALTER TABLE mcp_pending_auth_requests
+    ALTER COLUMN mcp_state TYPE TEXT;

ingestion/setup.py

@@ -282,7 +282,7 @@
     "opensearch": {VERSIONS["opensearch"]},
     "exasol": {
         "sqlalchemy_exasol>=6,<7",
-        "exasol-integration-test-docker-environment>=3.1.0,<4",
+        "exasol-integration-test-docker-environment>=6.0.0,<7",
     },
     "glue": {VERSIONS["boto3"]},
     "great-expectations": {VERSIONS["great-expectations"]},

ingestion/src/metadata/data_quality/validations/table/base/tableColumnCountToBeBetween.py

@@ -57,10 +57,11 @@ def _run_validation(self) -> TestCaseResult:
         min_bound = self.get_min_bound("minColValue")
         max_bound = self.get_max_bound("maxColValue")
 
+        column_word = "column" if count == 1 else "columns"
         return self.get_test_case_result_object(
             self.execution_date,
-            self.get_test_case_status(min_bound <= count <= max_bound),
-            f"Found columnCount={count} column vs. the expected  min={min_bound} and max={max_bound}].",
+            self.get_test_case_status(min_bound <= count <= max_bound),  # type: ignore
+            f"Found columnCount={count} {column_word} vs. the expected min={min_bound} and max={max_bound}",
             [TestResultValue(name=COLUMN_COUNT, value=str(count))],
         )