feat: add log_all_statements opt-out to silence verbose default Postgres logging by ivan-pinatti · Pull Request #92 · cloudposse-terraform-components/aws-aurora-postgres

ivan-pinatti · 2026-04-20T17:01:16Z

Summary

Add a log_all_statements bool variable (default true, preserves current behavior).
When true, src/cluster-regional.tf continues to prepend log_statement=all
and log_min_duration_statement=0 to cluster_parameters — no change for existing consumers.
When false, the hardcoded pair is skipped, so consumers can opt out and control both
parameters themselves via cluster_parameters (e.g. log_statement=none,
log_min_duration_statement=1000).

Motivation

The current defaults log every executed statement plus its duration (including
sub-millisecond ones). On an active cluster this routinely produces 20M+ log
lines per day, which can spike CloudWatch Logs and any downstream log-ingest
(Datadog, Splunk, ELK) by tens to hundreds of GB per day. We observed this
first-hand: a freshly deployed Aurora Postgres cluster running Metabase
produced ~22M postgresql log events per day and drove a ~120 GB/day
Datadog indexed-log spike until we patched the module locally.

The values are hardcoded and concat-ed before var.cluster_parameters,
so a user attempting to override them ends up with duplicate parameter names
that AWS rejects at the parameter-group level. In practice there is no
opt-out today without patching the component. This PR adds that opt-out
without changing the default behavior.

How to reproduce the problem this PR fixes

Against a freshly deployed Aurora-Postgres cluster using the current main
(no PR applied):

Deploy the component with defaults — no cluster_parameters override.
The hardcoded concat in src/cluster-regional.tf forces
log_statement=all and log_min_duration_statement=0 onto the cluster
parameter group. Verify:

aws rds describe-db-cluster-parameters \
  --db-cluster-parameter-group-name <your-pg-name> \
  --query "Parameters[?ParameterName=='log_statement' || ParameterName=='log_min_duration_statement']"
# expected: log_statement=all, log_min_duration_statement=0

Run any real workload against the cluster for 24 hours. Count the log
events shipped to CloudWatch:

aws cloudwatch get-metric-statistics \
  --namespace AWS/Logs --metric-name IncomingLogEvents \
  --dimensions Name=LogGroupName,Value=/aws/rds/cluster/<cluster>/postgresql \
  --start-time $(date -u -d '24 hours ago' +%FT%TZ) \
  --end-time $(date -u +%FT%TZ) \
  --period 86400 --statistic Sum

On our production Metabase cluster this returned ~22 million events/day
— every statement logged, plus a duration line for each. Ingested into
Datadog this drove a ~120 GB/day indexed-log spike.

Now try to turn it off from the catalog — e.g. append
{ name = "log_statement", value = "none", apply_method = "immediate" }
to var.cluster_parameters. Apply fails at the AWS parameter-group level
with a duplicate-parameter-name error, because the hardcoded values are
prepended via concat() and cannot be overridden. This is the problem:
no opt-out exists today short of patching the component.

Validating the fix in this PR

Same cluster, this PR applied:

Leave log_all_statements unset — it defaults to true. The parameter
group shows log_statement=all and log_min_duration_statement=0 exactly
as before. No behavior change for existing consumers.
Set log_all_statements = false and supply your own values:
```
log_all_statements: false
cluster_parameters:
  - { name: log_statement, value: none, apply_method: immediate }
  - { name: log_min_duration_statement, value: "1000", apply_method: immediate }
```
aws rds describe-db-cluster-parameters … now shows exactly the user's
values, no duplicates. The parameters are SIGHUP-reloadable so the
change applies live (no reboot).
After 24 hours, rerun the CloudWatch query from step 2 of reproduction.
In our production cluster the event count dropped from ~22M/day to
~200k/day — a ~99 % reduction — driven entirely by this single
parameter-group change.

Follow-up (out of scope)

Longer term, flipping the default to false in a major version bump would be
the cleanest fix, since log_statement=all + log_min_duration_statement=0
is a debug-mode combination, not a safe production default. Happy to open a
separate PR for that if maintainers agree.

Summary by CodeRabbit

New Features
- Added new configuration option to control automatic PostgreSQL statement logging for Aurora database clusters. By default, statement logging remains enabled to preserve existing behavior and maintain backward compatibility. Users can now optionally disable this automatic logging configuration if their logging requirements differ from the defaults.

…res logging Historically, src/cluster-regional.tf prepended log_statement=all and log_min_duration_statement=0 to user-supplied cluster_parameters via concat(). The combination logs every executed statement plus its duration, which on an active cluster produces tens of millions of log lines per day and can spike CloudWatch + downstream indexed-log bills by hundreds of GB/day. Because the values were hardcoded and concatenated before the user's list, a consumer attempting to override them produced duplicate parameter names that AWS rejects at the parameter-group level — effectively no opt-out existed. This PR introduces a log_all_statements bool (default true, preserves historical behavior) that gates the hardcoded pair. Consumers who want production-sane logging set it to false and control the two parameters themselves via cluster_parameters. No behavioral change for existing callers. Fixes a real-world log-volume regression observed after deploying this component with its defaults.

coderabbitai · 2026-04-20T17:01:33Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 82fab62b-f5bf-42b1-9ef6-469d648bdf83

📥 Commits

Reviewing files that changed from the base of the PR and between 36e5133 and 0e28e5d.

📒 Files selected for processing (2)

src/cluster-regional.tf
src/variables.tf

🚧 Files skipped from review as they are similar to previous changes (2)

src/variables.tf
src/cluster-regional.tf

📝 Walkthrough

Walkthrough

A new input variable log_all_statements (boolean, defaulting to true) controls whether two default PostgreSQL logging parameters are automatically added to the cluster configuration. When enabled, log_statement=all and log_min_duration_statement=0 are concatenated with user-provided parameters; when disabled, only user parameters are applied.

Changes

PostgreSQL Statement Logging Toggle

Layer / File(s)	Summary
PostgreSQL statement logging configuration `src/variables.tf`, `src/cluster-regional.tf`	New `log_all_statements` boolean variable controls conditional inclusion of default PostgreSQL logging parameters in cluster configuration using a ternary expression.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested labels

needs-test

Suggested reviewers

osterman
goruha

Poem

🐰 A toggle for logs, now fine and clear,
Let admins choose what statements to hear—
PostgreSQL chatters, or stays serene,
One variable rules the logging scene! 📋✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: introducing an opt-out mechanism for verbose default Postgres logging via a new log_all_statements variable.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

mergify · 2026-04-20T17:02:01Z

Important

Do not edit the README.md directly. It's auto-generated from the README.yaml

Please update the README.yaml file instead.

Could you fix it @ivan-pinatti? 🙏

aknysh · 2026-04-20T17:16:54Z

/terratest

Per mergify bot guidance on the PR: README.md and src/README.md are auto-generated from README.yaml + terraform-docs by the cloudposse-releaser bot after merge; human PRs should not include regen artifacts. Reverting both files to upstream main.

CI Lint was failing with conflicting constraints on cloudposse/utils: no available releases match the given constraints >= 1.7.1, >= 2.0.0, < 2.0.0, < 3.0.0 src/remote-state.tf (4 call sites) pinned stack-config/yaml v1.8.0, which requires utils >= 1.7.1, < 2.0.0. CI clones aws-account-map sibling component for ../account-map/modules/iam-roles, and iam_roles.account_map there now uses stack-config/yaml v2.0.0, which requires utils >= 2.0.0, < 3.0.0. The two ranges are incompatible. Bumping all four refs to v2.0.0 aligns with the sibling component and resolves the provider constraint. No breaking API changes between 1.8.0 and 2.0.0 in the remote-state submodule — only the utils version pin widens.

ivan-pinatti · 2026-04-20T18:16:16Z

Both issues addressed — PR now has 3 commits:

fecba0e — original log_all_statements opt-out.
db6023d — docs: revert README.md + src/README.md changes per mergify's note (those are auto-generated from README.yaml + terraform-docs by the cloudposse-releaser bot post-merge).
b6bfb7b — fix: bump cloudposse/stack-config/yaml//modules/remote-state from 1.8.0 → 2.0.0 at all 4 call sites in src/remote-state.tf, resolving the cloudposse/utils constraint conflict.

Lint root cause (from the failing job's log):

Downloading registry.terraform.io/cloudposse/stack-config/yaml 1.8.0 for dns_gbl_delegated, eks, vpc, vpc_ingress...
Downloading registry.terraform.io/cloudposse/stack-config/yaml 2.0.0 for iam_roles.account_map...

Error: Could not retrieve the list of available versions for provider cloudposse/utils:
no available releases match the given constraints >= 1.7.1, >= 2.0.0, < 2.0.0, < 3.0.0

src/remote-state.tf pinned stack-config/yaml v1.8.0 → utils >= 1.7.1, < 2.0.0.
CI clones aws-account-map HEAD to resolve ../account-map/modules/iam-roles, whose iam_roles.account_map sub-module pulls stack-config/yaml v2.0.0 → utils >= 2.0.0, < 3.0.0.
Two incompatible ranges on the same provider = unresolvable.

Main last passed lint on 2026-03-07 with resolved constraint != 1.4.0, >= 1.7.1, < 2.0.0 (only the < 2.0.0 caller existed). The >= 2.0.0 constraint appeared when aws-account-map's iam-roles sub-module bumped to stack-config v2 upstream. Bumping this repo's callers to v2.0.0 aligns both sides.

I verified the modules/remote-state interface is unchanged between v1.8.0 and v2.0.0 — the only difference is the cloudposse/utils version pin widens.

Happy to squash the 3 commits into one at merge time, or leave them separate for a clearer history — whichever you prefer.

aknysh · 2026-04-21T00:02:04Z

/terratest

mergify · 2026-04-21T00:10:52Z

Thanks @ivan-pinatti for creating this pull request!

A maintainer will review your changes shortly. Please don't be discouraged if it takes a while.

While you wait, make sure to review our contributor guidelines.

Tip

Need help or want to ask for a PR review to be expedited?

Join us on Slack in the #pr-reviews channel.

aknysh · 2026-04-21T00:57:44Z

@ivan-pinatti the tests still failed, please review

https://github.com/cloudposse-terraform-components/aws-aurora-postgres/actions/runs/24696723774/job/72231030479

let me know if you need help

Terratest was failing with the same cloudposse/utils constraint conflict that previously affected CI Lint: no available releases match the given constraints != 1.4.0, >= 1.7.1, >= 2.0.0, < 2.0.0, < 3.0.0 When atmos applies aurora-postgres/basic and aurora-postgres/serverless, src/providers.tf references ../account-map/modules/iam-roles as a local module. That module's dependency graph therefore merges into aurora-postgres's provider constraints. The test was vendoring account-map from the legacy monorepo (cloudposse/terraform-aws-components//modules/account-map at 1.520.0), whose iam-roles/main.tf still uses cloudposse/stack-config/yaml//modules/remote-state v1.5.0 — which pins cloudposse/utils >= 1.7.1, < 2.0.0. The previous commit (b6bfb7b) already bumped src/remote-state.tf to stack-config v2.0.0 (utils >= 2.0.0, < 3.0.0), so the two constraints were irreconcilable at terraform init. Switching the vendor source to the standalone cloudposse-terraform-components/aws-account-map repo at v1.537.2 brings the iam-roles sub-module up to stack-config v2.0.0, aligning all cloudposse/utils constraints in the init graph to >= 2.0.0, < 3.0.0. Validated locally: `atmos vendor pull` + `terraform init -backend=false` in a simulated aurora-postgres working dir resolves cloudposse/utils v2.5.0 cleanly with no constraint conflicts.

ivan-pinatti · 2026-04-21T13:59:23Z

Thanks for running /terratest @aknysh — the failure exposed a second constraint conflict on the test-fixture side that I've just pushed a fix for.

Root cause (same class as the lint fix, different call site)

When Terratest applies aurora-postgres/basic or aurora-postgres/serverless, src/providers.tf references ../account-map/modules/iam-roles as a local module, so that sub-module's dependency graph merges directly into aurora-postgres's provider constraints at init time.

test/fixtures/vendor.yaml was vendoring account-map from the legacy monorepo (github.com/cloudposse/terraform-aws-components//modules/account-map at 1.520.0), whose iam-roles/main.tf still uses cloudposse/stack-config/yaml//modules/remote-state v1.5.0, which pins cloudposse/utils >= 1.7.1, < 2.0.0.

My earlier commit b6bfb7b bumped src/remote-state.tf to stack-config v2.0.0 (utils >= 2.0.0, < 3.0.0) to resolve the Lint job's own conflict. Put together, the Terratest init was asked to satisfy:

cloudposse/utils  "!= 1.4.0, >= 1.7.1, >= 2.0.0, < 2.0.0, < 3.0.0"

which has no solution.

Fix — `36e5133`

Switch the account-map source in test/fixtures/vendor.yaml from the monorepo to the standalone cloudposse-terraform-components/aws-account-map repo at v1.537.2. That version's iam-roles/main.tf already uses stack-config v2.0.0, aligning every cloudposse/utils caller on >= 2.0.0, < 3.0.0.

-    source: github.com/cloudposse/terraform-aws-components.git//modules/account-map?ref={{.Version}}
-    version: 1.520.0
+    source: github.com/cloudposse-terraform-components/aws-account-map.git//src?ref={{.Version}}
+    version: v1.537.2

Also brings the test setup in line with the same repo the Lint job already clones for ../account-map, so one source of truth now governs both jobs.

Validation

Reproduced the failure locally, then reproduced the green outcome:

cd test/fixtures && atmos vendor pull
# ... vendored account-map v1.537.2 successfully

cd components/terraform && cp -r .../src/ aurora-postgres/ && cd aurora-postgres
terraform init -backend=false -lock=false -upgrade
# ...
# - Finding cloudposse/utils versions matching ">= 2.0.0, < 3.0.0"...
# - Installing cloudposse/utils v2.5.0...
# Terraform has been successfully initialized!

Single clean constraint range, no conflicts. Please kick off /terratest again when you have a minute 🙏

mergify · 2026-04-21T14:03:54Z

Thanks @ivan-pinatti for creating this pull request!

A maintainer will review your changes shortly. Please don't be discouraged if it takes a while.

While you wait, make sure to review our contributor guidelines.

Tip

Need help or want to ask for a PR review to be expedited?

Join us on Slack in the #pr-reviews channel.

aknysh · 2026-04-21T14:25:23Z

/terratest

mergify · 2026-05-20T16:59:16Z

Heads up! This pull request looks stale. It will be closed soon, if there are no new commits. ⏳

mergify · 2026-05-20T17:06:20Z

Thanks @ivan-pinatti for creating this pull request!

A maintainer will review your changes shortly. Please don't be discouraged if it takes a while.

While you wait, make sure to review our contributor guidelines.

Tip

Need help or want to ask for a PR review to be expedited?

Join us on Slack in the #pr-reviews channel.

mergify · 2026-05-27T17:01:42Z

Heads up! This pull request looks stale. It will be closed soon, if there are no new commits. ⏳

mergify · 2026-05-27T17:09:46Z

Thanks @ivan-pinatti for creating this pull request!

A maintainer will review your changes shortly. Please don't be discouraged if it takes a while.

While you wait, make sure to review our contributor guidelines.

Tip

Need help or want to ask for a PR review to be expedited?

Join us on Slack in the #pr-reviews channel.

coderabbitai · 2026-05-28T17:34:41Z

Actionable comments posted: 0

mergify · 2026-05-28T17:40:09Z

Thanks @ivan-pinatti for creating this pull request!

A maintainer will review your changes shortly. Please don't be discouraged if it takes a while.

While you wait, make sure to review our contributor guidelines.

Tip

Need help or want to ask for a PR review to be expedited?

Join us on Slack in the #pr-reviews channel.

aknysh · 2026-05-28T18:12:44Z

/terratest

mergify · 2026-05-28T19:06:23Z

Thanks @ivan-pinatti for creating this pull request!

A maintainer will review your changes shortly. Please don't be discouraged if it takes a while.

While you wait, make sure to review our contributor guidelines.

Tip

Need help or want to ask for a PR review to be expedited?

Join us on Slack in the #pr-reviews channel.

github-actions · 2026-05-28T19:44:33Z

These changes were released in v2.1.0.

mergify Bot requested review from a team April 20, 2026 17:01

mergify Bot added the triage Needs triage label Apr 20, 2026

ivan-pinatti added 2 commits April 20, 2026 13:57

aknysh previously approved these changes Apr 21, 2026

View reviewed changes

mergify Bot removed the triage Needs triage label Apr 21, 2026

aknysh added the minor New features that do not break anything label Apr 21, 2026

ivan-pinatti dismissed aknysh’s stale review via 36e5133 April 21, 2026 13:56

mergify Bot added stale This PR has gone stale and removed stale This PR has gone stale labels May 20, 2026

mergify Bot added stale This PR has gone stale and removed stale This PR has gone stale labels May 27, 2026

Merge branch 'main' into add-opt-out-verbose-logging

0e28e5d

aknysh approved these changes May 28, 2026

View reviewed changes

aknysh added this pull request to the merge queue May 28, 2026

Merged via the queue into cloudposse-terraform-components:main with commit 9135a84 May 28, 2026
19 checks passed

Uh oh!

Uh oh!

Conversation

ivan-pinatti commented Apr 20, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

How to reproduce the problem this PR fixes

Validating the fix in this PR

Follow-up (out of scope)

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Suggested labels

Suggested reviewers

Poem

Uh oh!

mergify Bot commented Apr 20, 2026

Uh oh!

aknysh commented Apr 20, 2026

Uh oh!

ivan-pinatti commented Apr 20, 2026

Uh oh!

aknysh commented Apr 21, 2026

Uh oh!

mergify Bot commented Apr 21, 2026

Need help or want to ask for a PR review to be expedited?

Uh oh!

aknysh commented Apr 21, 2026

Uh oh!

ivan-pinatti commented Apr 21, 2026

Root cause (same class as the lint fix, different call site)

Fix — 36e5133

Validation

Uh oh!

mergify Bot commented Apr 21, 2026

Need help or want to ask for a PR review to be expedited?

Uh oh!

aknysh commented Apr 21, 2026

Uh oh!

mergify Bot commented May 20, 2026

Uh oh!

mergify Bot commented May 20, 2026

Need help or want to ask for a PR review to be expedited?

Uh oh!

mergify Bot commented May 27, 2026

Uh oh!

mergify Bot commented May 27, 2026

Need help or want to ask for a PR review to be expedited?

Uh oh!

coderabbitai Bot commented May 28, 2026

Uh oh!

mergify Bot commented May 28, 2026

Need help or want to ask for a PR review to be expedited?

Uh oh!

aknysh commented May 28, 2026

Uh oh!

mergify Bot commented May 28, 2026

Need help or want to ask for a PR review to be expedited?

Uh oh!

Uh oh!

github-actions Bot commented May 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ivan-pinatti commented Apr 20, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Apr 20, 2026 •

edited

Loading

Fix — `36e5133`