Fix: Enhance SSL support for PostgreSQL and MSSQL with mutual TLS configuration

[SumanMaharana]

Describe your changes:

Fixes #27085
This pull request improves SSL and mutual TLS (client certificate) support across multiple database and pipeline connectors. The main focus is on ensuring that all relevant SSL certificate fields (CA, client cert, client key) are correctly extracted, handled, and passed to drivers for secure connections, especially for PostgreSQL, Redshift, and MSSQL (pytds), and Kafka. The changes also add comprehensive unit tests to verify these enhancements.

SSL/mutual TLS handling improvements:

• Updated SSLManager and its usage to ensure all three SSL fields (CA certificate, client certificate, and client key) are extracted and passed to database drivers (PostgreSQL, Redshift, Greenplum, MSSQL/pytds) for proper mutual TLS authentication. Previously, only the CA certificate was handled, causing mutual TLS to silently fail. (ingestion/src/metadata/utils/ssl_manager.py [1] [2] [3]
• Refactored Hive and Kafka connection setup to use SSLManager for temporary file management of SSL certificates, avoiding direct assignment of secret values as file paths and preventing driver-level errors. (ingestion/src/metadata/ingestion/source/database/hive/connection.py [1] ingestion/src/metadata/ingestion/source/pipeline/openlineage/connection.py [2] [3]

Testing enhancements:

• Added new unit tests for PostgreSQL and Redshift to verify that all SSL parameters (CA, cert, key) are handled and set correctly for mutual TLS, and that legacy behaviors (CA-only) are preserved. (ingestion/tests/unit/test_ssl_manager.py ingestion/tests/unit/test_ssl_manager.pyR653-R802)
• Introduced a test for MSSQL/pytds mutual TLS support to ensure all three certificate files are set in connection arguments. (ingestion/tests/unit/test_ssl_manager.py ingestion/tests/unit/test_ssl_manager.pyR353-R383)

These changes collectively ensure that mutual TLS (client certificate authentication) works reliably and as expected across supported connectors, and that the codebase is better tested for these scenarios.

Type of change:

• Bug fix
• Improvement
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation

Checklist:

• I have read the CONTRIBUTING document.
• My PR title is Fixes <issue-number>: <short explanation>
• I have commented on my code, particularly in hard-to-understand areas.
• For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

[Copilot]

Pull request overview

This PR improves ingestion-side SSL handling to support mutual TLS (CA + client cert + client key) across database and pipeline connectors, and adds unit tests to prevent regressions.

Changes:

• Forward client cert + key to psycopg2 connections (Postgres/Redshift/Greenplum) and pytds (MSSQL) via SSLManager.
• Refactor Hive and OpenLineage (Kafka) connectors to use SSLManager temp files rather than passing secret contents as file paths.
• Add unit tests covering mutual TLS parameter wiring for Postgres/Redshift and MSSQL (pytds).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
ingestion/src/metadata/utils/ssl_manager.py	Extracts + forwards cert/key for psycopg2 and pytds SSL args.
ingestion/src/metadata/ingestion/source/database/hive/connection.py	Removes direct sslConfig→driver-arg mapping and relies on SSLManager temp paths.
ingestion/src/metadata/ingestion/source/pipeline/openlineage/connection.py	Uses SSLManager temp paths for Kafka SSL configuration.
ingestion/tests/unit/test_ssl_manager.py	Adds unit tests for mutual TLS wiring for Postgres/Redshift and MSSQL (pytds).

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

[github-actions]

🟡 Playwright Results — all passed (26 flaky)

✅ 3592 passed · ❌ 0 failed · 🟡 26 flaky · ⏭️ 207 skipped

Shard	Passed	Failed	Flaky	Skipped
🟡 Shard 1	456	0	1	2
🟡 Shard 2	638	0	4	32
🟡 Shard 3	647	0	4	26
🟡 Shard 4	617	0	5	47
🟡 Shard 5	605	0	2	67
🟡 Shard 6	629	0	10	33

🟡 26 flaky test(s) (passed on retry)

• Pages/UserCreationWithPersona.spec.ts › Create user with persona and verify on profile (shard 1, 1 retry)
• Features/BulkEditEntity.spec.ts › Table (shard 2, 1 retry)
• Features/BulkEditEntity.spec.ts › Glossary (shard 2, 1 retry)
• Features/BulkImport.spec.ts › Keyboard Delete selection (shard 2, 1 retry)
• Features/EntitySummaryPanel.spec.ts › should display summary panel for table (shard 2, 1 retry)
• Features/IncidentManager.spec.ts › Next, Previous and page indicator (shard 3, 1 retry)
• Features/Permissions/GlossaryPermissions.spec.ts › Team-based permissions work correctly (shard 3, 1 retry)
• Flow/CustomizeWidgets.spec.ts › Following Assets Widget (shard 3, 1 retry)
• Flow/ExploreDiscovery.spec.ts › Should display deleted assets when showDeleted is checked and deleted is not present in queryFilter (shard 3, 1 retry)
• Pages/Customproperties-part2.spec.ts › entityReferenceList shows item count, scrollable list, no expand toggle (shard 4, 1 retry)
• Pages/DataContracts.spec.ts › Create Data Contract and validate for Table (shard 4, 1 retry)
• Pages/DataContracts.spec.ts › Create Data Contract and validate for SearchIndex (shard 4, 1 retry)
• Pages/Domains.spec.ts › Rename domain with subdomains attached verifies subdomain accessibility (shard 4, 1 retry)
• Pages/Domains.spec.ts › Rename domain with tags and glossary terms preserves associations (shard 4, 1 retry)
• Pages/EntityDataConsumer.spec.ts › Tier Add, Update and Remove (shard 5, 1 retry)
• Pages/ExploreTree.spec.ts › Verify Database and Database Schema available in explore tree (shard 5, 1 retry)
• Pages/Glossary.spec.ts › Create term with related terms, tags and owners during creation (shard 6, 1 retry)
• Pages/Lineage/DataAssetLineage.spec.ts › verify create lineage for entity - Table (shard 6, 1 retry)
• Pages/Lineage/DataAssetLineage.spec.ts › Column lineage for table -> table (shard 6, 1 retry)
• Pages/Lineage/LineageFilters.spec.ts › Verify lineage schema filter selection (shard 6, 1 retry)
• Pages/Login.spec.ts › Refresh should work (shard 6, 1 retry)
• Pages/ProfilerConfigurationPage.spec.ts › Non admin user (shard 6, 1 retry)
• Pages/ServiceEntity.spec.ts › Glossary Term Add, Update and Remove (shard 6, 1 retry)
• Pages/ServiceEntity.spec.ts › Glossary Term Add, Update and Remove (shard 6, 1 retry)
• Pages/Users.spec.ts › Permissions for table details page for Data Consumer (shard 6, 1 retry)
• VersionPages/EntityVersionPages.spec.ts › Directory (shard 6, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

[gitar-bot]

Code Review ✅ Approved 1 resolved / 1 findings

Enhances SSL support for PostgreSQL and MSSQL with mutual TLS configuration, addressing the SSL temp files leak on Kafka connection failure. No issues found.

✅ 1 resolved

✅ Bug: SSL temp files leak on Kafka connection failure

📄 ingestion/src/metadata/ingestion/source/pipeline/openlineage/connection.py:70 📄 ingestion/src/metadata/ingestion/source/pipeline/openlineage/connection.py:78-88 📄 ingestion/src/metadata/ingestion/source/pipeline/openlineage/connection.py:107-109
In _get_kafka_connection, if an exception occurs after the SSLManager is created (line 78) but before the consumer is returned (e.g., KafkaConsumer(config) fails on line 102), the except block on line 107 re-raises without cleaning up the temporary certificate files written to disk by SSLManager.__init__. Since temp files are created eagerly during __init__, repeated connection failures will leave orphaned files in the system's temp directory.

The cleanup in _poll_kafka (metadata.py:743-745) only runs when the consumer is successfully created and later closed, so it cannot cover this error path.

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud