v0.1.49

Highlights

New query parser

We rewrote the query parser to use our in-house library: https://github.com/pgdogdev/pg_raw_parse/. It's based on pg_query but maps Rust structs directly to the arena-allocated MemoryContext allocated by the Postgres parser. This brings our (de)ser cost for parsing queries to 0.

Early metrics:

1. 2x improvement in parsing speed vs query_parser_engine = "pg_query_raw"
2. 8x improvement in parsing speed vs query_parser_engine = "pg_query_protobuf" (used by default, so most users will see this)
3. 90% (6x) reduction in memory usage (eliminates the OOM problems we've been having with large queries e.g., #854 #1061)

It's entirely generated using Rust build scripts directly from Postgres C code and allows us to do major version upgrades in one, 5 line PR. More on this soon on our blog.

How to use

The new parser is experimental and disabled by default. We are building a separate Docker image for each release / commit in main. To use it, just append -new-parser prefix to the tag, for example:

image:
  tag: v0.1.49-new-parser

by @sgrif

New plugin interface

The plugins system was rewritten to use safer primitives and simpler API. This is a breaking change. We ported all in-tree plugins to the new interface, so no impact on any plugins we currently ship with PgDog in the Docker image. If you wrote your own plugins, reach out to us and we'll help you fix them. New documentation coming soon.

by @sgrif

Changelog

• fix: skip multi-statement SET validation in session mode by @zefixlluja
• fix: add default to engine in [[query_parsers]] by @levkk
• fix: include_primary_if_replica_banned ignored primary if it was the only candidate by @levkk @kzemek
• fix: role detector deadlocked when new databases with role = "auto" were added at runtime by @levkk
• feat: allow read_write_split when parser is disabled to direct traffic by default @levkk
• fix: query parser ran regex twice by @levkk
• test: disable copy_data retry_tests temporarily by @meskill
• fix(replication): fix wait_for_replication logic for unrelated changes by @meskill
• fix: LSN detector can now correctly identify replica-only deployments by @levkk
• fix: support psycopg2 (handle SET commands) in admin DB by @levkk
• feat(breaking): use a less horrendously unsafe FFI interface for plugins by @sgrif
• chore(config): save file content for ConfigAndUsers for diff handling in EE by @meskill
• refactor: remove fingerprinting from our codebase by @sgrif
• feat: update the parser to support Postgres 18 by @sgrif

New Contributors

• @zefixlluja made their first contribution in #1157

Full Changelog: v0.1.48...v0.1.49

v0.1.48

What's Changed

• feat: manual connection pinning using SET pgdog.pin TO true; -- false / RESET to unpin; useful to temporarily use session-level features in transaction mode
• feat: Hashicorp Vault static role credentials @larrywax
• feat: per-database query parser setting
• fix(sharding): correctly handle pipelined cross-shard queries @sgrif
• fix(replication): better cancellation handling during replication streaming @meskill
• fix: don't re-create connections when password changes, allowing for faster reloads
• chore: normalize configs for nextest and rust integration tests @meskill
• refactor: ported several functions to new, much faster parser @sgrif

Full Changelog: v0.1.47...v0.1.48

v0.1.47

What's Changed

• feat(lb): speed up failover & add standalone connection fallback for LSN check
• feat(sharding): add support for two-phase commit to COPY used for resharding @yogesh1801
• refactor(resharding): rework of async tasks used in resharding to handle cancellation better @meskill
• fix(sharding): prevent premature cutover due to lag miscalculation @meskill
• fix(sharding): update the processed LSN on keep-alive messages @meskill
• fix(lb): replica lag evaluation in load balancer used for banning replicas
• fix(lb): fix no routing error with prefer_primary when all replicas are banned @stewart-glabs
• refactor(sharding): more parser code is moving to pg_parse_raw @sgrif

New Contributors

• @yogesh1801 made their first contribution in #1088

Full Changelog: v0.1.46...v0.1.47

v0.1.46

Breaking change

PgDog now blocks UPDATE queries that mutate a sharding key in a table that has foreign key references to other tables which automatically mutate the referenced tables using:

• CASCADE
• SET NULL

Both of these were previously breaking referential integrity, since we were not moving the referenced rows. RESTRICT (which is typically default) is not affected because Postgres is able to validate ref integrity before we move the row.

Changelog

• feat(sharding): rewrite [[sharded_mappings]] config to make it easier to use @meskill (not documented yet, old config format still works)
• feat: add query_size_limit setting to block/warn on very large queries @meskill
• feat: RELOAD keeps connections and paused state if adding or removing connection databases or users @murex971
• feat(load balancer): add prefer_primary read/write separation mode to allow manual opt into load balancing @stewart-glabs
• feat: log warning when lsn_check_delay is misconfigured
• refactor: start porting the codebase to use the new, much faster SQL parser @sgrif
• feat(sharding): block cross-shard sharding key updates on tables that have foreign key constraints @murex971
• feat(auth): Hashicorp Vault server authentication integration @larrywax

New Contributors

• @stewart-glabs made their first contribution in #1090
• @larrywax made their first contribution in #958

Full Changelog: v0.1.45...v0.1.46

v0.1.45

What's Changed

Distribution

Pre-compiled binaries now ship with every release in GitHub:

• Linux x86/ARM
• Mac (Apple Silicon)
• Debian package (amd64/arm64).

Breaking change

SET LOCAL pgdog.shard and SET LOCAL pgdog.sharding_key now assume the entire transaction will be sent to that one shard only. Any query that attempts to talk to a different shard, incl. omnisharded writes, will be blocked. This is not really a breaking change, but more of a bug fix, but still worth calling this out since this changes the behavior of the sharding logic a bit.

If using sharding, please double check this release before deploying to prod just to make sure the new shard checker doesn't accidentally (and incorrectly) block your query from execution.

Changelog

• feat: check that tuples in multi-tuple INSERT statement belong to the same shard before rewriting @murex971
• feat: handle the special case of SELECT set_config(...), which we didn't track previously; this fixes the "pg_dump broke my pooler" issue @sgrif
• feat: enable parser for LISTEN/NOTIFY/UNLISTEN in pooler-only mode (`query_parser = "on" not required, makes PgDog much faster)
• feat: add LISTEN/NOTIFY stats
• fix(sharding): COPY with rows exceeding one CopyData message size were incorrectly routed to all shards @LnL7
• fix: Elixir driver sends Close for one-off statements, which we incorrectly rejected with an error
• fix: major performance improvements in pg_query, affecting long queries @sgrif
• fix: Windows build by @imor
• fix: race condition in LISTEN when registering for a channel for the first time
• fix: UnexpectedEof error log when passthrough auth is enabled
• fix: use interval instead of sleep in stats loop which may have caused our stats to stop working after 12 days of uptime (potential Tokio bug, not confirmed)

New Contributors

• @LnL7 made their first contribution in #1062
• @murex971 made their first contribution in #1065

Full Changelog: v0.1.44...v0.1.45

v0.1.44

What's Changed

• fix(security): use constant_eq for password checks @meskill
• fix(security): admin password not checked when using passthrough auth
• fix(resharding): fix issues with copy data @meskill
• feat(resharding): use binary protocol for replication (slightly faster than text) @meskill
• fix(sharding): don't track columns independently, allowing duplicate expression rewrites in cross-shard queries @sgrif
• feat(sharding): error when encountering an unsupported aggregate function in cross-shard queries @sgrif
• feat(sharding): give a more meaningful error message when a helper column is missing in cross-shard queries @sgrif
• chore: update to Rust 2024 @sgrif
• fix: refactor and improve the functionality of avg() @sgrif
• refactor: aggregate functions cleanup @sgrif
• fix: clippy fixes @imor
• fix: more clippy errors and CI enforcement @meskill
• fix: handle comments in SET in regex detector

Also introducing our company mascot, Von Count.

New Contributors

• @imor made their first contribution in #1056

Full Changelog: v0.1.43...v0.1.44

v0.1.43

What's Changed

• feat: allow disabling idle healthchecks @kitop
• feat: add query_log_stdout config for pgcat-style query logging @marcomicera
• feat: allow to bypass DNS TTL entirely with dns_ttl = 0
• fix: use smaller data structure to map clients to servers, improving connection pool performance slightly @meskill
• test: fix flaky prepared statements test @meskill
• refactor: cleanup some awkward Rust patterns @sgrif

New Contributors

• @marcomicera made their first contribution in #1015

Full Changelog: v0.1.42...v0.1.43

v0.1.42

What's Changed

• feat(logging): identify the source of auth secret for new server connections
• feat(tls): mTLS using client certificates and per-user CN/dNSName matching (this is still experimental and not final, use for testing / PoC only); can replace password auth for client -> PgDog connections
• feat: maintenance mode can now be set on a per-database basis instead of just globally
• feat(logging): log which auth error the user experienced to stderr; helps debug complex auth configurations
• fix: HTTP health check returned false negative when passthrough auth was used @sunil-2000
• fix(sharding): correct display for integers in cross-shard aggregates @sgrif
• fix(sharding): false positive cross-shard check for BEGIN in session mode with cross-shard queries disabled
• fix(sharding): multi-tuple INSERTs rewrites with go/pq (broke in the last release when we removed prepared_statements = "extended_anonymous")
• fix(metrics): false positive OpenMetrics error in logs (hyper::Error(IncompleteMessage))
• chore: update dependencies @sgrif
• test(resharding): add benchmarks for copy_data & replication by @meskill
• chore(test): fix some flaky tests

New Contributors

• @sunil-2000 made their first contribution in #997

Full Changelog: v0.1.41...v0.1.42

v0.1.41

What's Changed

• fix: read and write pool stats were incorrect, didn't account for clients waiting for a connection
• feat: add slow query parse logging @frodsan
• feat: token caching for external identity providers @magicabdel
• feat: add flattened JSON log format support @kitop
• refactor: use rust_decimal's builtin PG encode/decode @sgrif
• feat(mtls): add option to validate CN matches username (passwordless client auth)
• fix(replication): distribute destination omni among the subscribers (fixes potential deadlock) @meskill
• test(resharding): more resharding tests @meskill
• refactor: yeet (remove) unused serde impls for Numeric @sgrif
• feat(replication): add retry logic for transient errors (e.g. TPC connection reset) @meskill
• fix: maintain Numeric scale when computing avg() across shards @sgrif
• fix(prepared_statements): full rewrites extended as well (fixes prepared_statements = "full") @meskill

New Contributors

• @frodsan made their first contribution in #972
• @kitop made their first contribution in #987
• @sgrif made their first contribution in #989

Full Changelog: v0.1.40...v0.1.41

v0.1.40

What's Changed

• feat(replication): add support for replica identity full @meskill
• fix(protocol): add handling for fastpath function calls @meskill
• fix: retrying known incorrect passwords repeatedly
• fix: new clients blocked from connecting due to stale pool info
• feat: better error message for LIMIT parser
• feat(pool): jitter for server_lifetime @frangz
• feat: support for mTLS / client certificates @lars-berger
• fix: decode + as space in options startup parameter @matteosilv

New Contributors

• @frangz made their first contribution in #963
• @lars-berger made their first contribution in #969
• @matteosilv made their first contribution in #976

Full Changelog: v0.1.39...v0.1.40