Skip to content

fix: SSL/TLS connection failures for PostgreSQL 17 with Route 53 CNAME endpoints + Python 3.12 runtime upgrade#173

Open
tejadonthu wants to merge 1 commit into
aws-samples:masterfrom
tejadonthu:feature/ssl-tls-connection-fix-for-route53-cname
Open

fix: SSL/TLS connection failures for PostgreSQL 17 with Route 53 CNAME endpoints + Python 3.12 runtime upgrade#173
tejadonthu wants to merge 1 commit into
aws-samples:masterfrom
tejadonthu:feature/ssl-tls-connection-fix-for-route53-cname

Conversation

@tejadonthu
Copy link
Copy Markdown

Problem

The SecretsManagerRDSPostgreSQLRotationSingleUser rotation Lambda fails to connect to RDS PostgreSQL 17 instances when the hostname in the secret is a Route 53 CNAME pointing to the RDS cluster endpoint. This worked on PostgreSQL 16 and earlier, but PostgreSQL 17's stricter SSL/TLS enforcement exposes three issues in the sample code:

  1. Hardcoded CA certificate pathsslrootcert='/etc/pki/tls/cert.pem' is not guaranteed to exist or contain current certificates across Lambda runtime versions.

  2. verify-full incompatible with Route 53 CNAMEssslmode='verify-full' requires the server certificate's CN/SAN to match the connection hostname. Route 53 CNAMEs (e.g. db.internal.example.com) don't match the RDS-issued certificate (e.g. mydb.abc123.us-east-1.rds.amazonaws.com). PostgreSQL 16 was lenient about this; PostgreSQL 17 rejects it.

  3. Silent SSL-to-plaintext fallback — When the ssl key is omitted from the secret JSON, the code attempts SSL first, then silently retries without encryption if SSL fails. With PostgreSQL 17 rejecting SSL handshakes more often, credential rotation traffic that was previously encrypted can silently downgrade to plaintext.

This is a common enterprise pattern — many customers upgrading from PostgreSQL 16 to 17 with DNS aliasing (Route 53 CNAMEs) in front of RDS endpoints will encounter the same failures.

Changes

Driver: PyGreSQL (pg/pgdb) → pg8000

  • Replaced C-extension-based PyGreSQL with pure-Python pg8000, simplifying Lambda packaging and enabling native Python SSL context control.

SSL/TLS: verify-ca via system trust store

  • Replaced sslrootcert='/etc/pki/tls/cert.pem', sslmode='verify-full' with ssl.create_default_context() + check_hostname = False.
  • create_default_context() loads the Lambda runtime's system CA bundle (always present and up-to-date).
  • check_hostname = False gives verify-ca semantics: the server's certificate chain is fully validated, but the certificate hostname does not need to match the connection hostname. This allows Route 53 CNAMEs that differ from the RDS certificate CN/SAN.

Removed silent SSL-to-plaintext fallback

  • get_ssl_config() now returns a single bool instead of a (use_ssl, fall_back) tuple.
  • When ssl is omitted from the secret JSON, SSL is required with no fallback. Users who need plaintext connections must explicitly set "ssl": false.

set_secret improvements

  • Confused deputy protection: validates that username and host match between AWSCURRENT and AWSPENDING before attempting any database connection.
  • Reordered logic: tries AWSCURRENT first (common case), then falls back to AWSPENDING as an idempotency check for retried Lambda invocations.
  • Removed AWSPREVIOUS fallback (not needed for single-user rotation).
  • Safer SQL escaping: replaced quote_ident() + parameterized query with PostgreSQL's format('%I ... %L', ...), which is more reliable across drivers for DDL statements.

Other improvements

  • Added aurora-postgresql to supported engine types.
  • Improved error messages — include secret ARN and actual values for easier debugging.
  • Changed connection failure log level from ERROR to WARNING (expected control flow, not terminal errors).
  • Added detection for authentication failures (28P01) and pg_hba.conf rejections.
  • Removed get_input_map_value() — printable-ASCII validation was unnecessary for trusted Secrets Manager service inputs.

Breaking changes

Change Impact Mitigation
SSL fallback removed Secrets that omit the ssl key now require SSL. If the DB doesn't support SSL, the connection will fail instead of silently downgrading to plaintext. Set "ssl": false explicitly in the secret JSON.
AWSPREVIOUS no longer attempted If a previous rotation left the secret in an inconsistent state where only AWSPREVIOUS credentials work, this version will not recover automatically. Edge case for single-user rotation; re-set the AWSCURRENT secret manually.
Driver change Lambda deployment package must include pg8000 instead of PyGreSQL. Update requirements.txt / Lambda layer.

Testing

  • RDS PostgreSQL 17 with Route 53 CNAME endpoint (the failing scenario) ✅
  • RDS PostgreSQL 16 (backward compatibility) ✅
  • Aurora PostgreSQL with aurora-postgresql engine type ✅

@tejadonthu tejadonthu changed the title fix: SSL/TLS connection failures for PostgreSQL 17 with Route 53 CNAME endpoints (Single User) fix: SSL/TLS connection failures for PostgreSQL 17 with Route 53 CNAME endpoints + Python 3.12 runtime upgrade Mar 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant