Skip to content

pgwatch goes mad when the pg sink is down #1426

Description

@0xgouda

Describe the bug
After running pgwatch for a while, bring the pg sink instance down, wait a second or whatever is --batching-delay, then observe the tremendous amount of connection errors that pgwatch will generate.

Expected behavior
pgwatch shouldn't issue that many connection retrials and sleep a bit before reconnecting to the host

Screenshots

2026-06-01 19:43:49.231 [ERROR] [err:failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused] [time:188.182µs] Acquire
2026-06-01 19:43:49.231 [ERROR] [sink:postgres] [db:postgres] failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
2026-06-01 19:43:49.231 [ERROR] [database:postgres] [err:failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused] [host:localhost] [port:5432] [time:150.866µs] Connect
2026-06-01 19:43:49.231 [ERROR] [err:failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused] [time:176.781µs] Acquire
2026-06-01 19:43:49.231 [ERROR] [sink:postgres] [db:postgres] failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
2026-06-01 19:43:49.231 [ERROR] [database:postgres] [err:failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused] [host:localhost] [port:5432] [time:143.401µs] Connect
2026-06-01 19:43:49.231 [ERROR] [err:failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused] [time:177.748µs] Acquire
2026-06-01 19:43:49.231 [ERROR] [sink:postgres] [db:postgres] failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
2026-06-01 19:43:49.231 [ERROR] [database:postgres] [err:failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused] [host:localhost] [port:5432] [time:166.61µs] Connect
2026-06-01 19:43:49.231 [ERROR] [err:failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused] [time:191.416µs] Acquire
2026-06-01 19:43:49.231 [ERROR] [sink:postgres] [db:postgres] failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
2026-06-01 19:43:49.232 [ERROR] [database:postgres] [err:failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused] [host:localhost] [port:5432] [time:383.484µs] Connect
2026-06-01 19:43:49.232 [ERROR] [err:failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused] [time:437.729µs] Acquire
2026-06-01 19:43:49.232 [ERROR] [sink:postgres] [db:postgres] failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
2026-06-01 19:43:49.232 [ERROR] [database:postgres] [err:failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused] [host:localhost] [port:5432] [time:263.12µs] Connect
2026-06-01 19:43:49.232 [ERROR] [err:failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused] [time:302.132µs] Acquire
2026-06-01 19:43:49.232 [ERROR] [sink:postgres] [db:postgres] failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
2026-06-01 19:43:49.232 [ERROR] [database:postgres] [err:failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused] [host:localhost] [port:5432] [time:166.458µs] Connect
2026-06-01 19:43:49.232 [ERROR] [err:failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused] [time:199.671µs] Acquire
2026-06-01 19:43:49.232 [ERROR] [sink:postgres] [db:postgres] failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
2026-06-01 19:43:49.232 [ERROR] [database:postgres] [err:failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused] [host:localhost] [port:5432] [time:142.457µs] Connect
2026-06-01 19:43:49.233 [ERROR] [err:failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused] [time:172.078µs] Acquire
2026-06-01 19:43:49.233 [ERROR] [sink:postgres] [db:postgres] failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
2026-06-01 19:43:49.233 [ERROR] [database:postgres] [err:failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused] [host:localhost] [port:5432] [time:149.619µs] Connect
2026-06-01 19:43:49.233 [ERROR] [err:failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused] [time:173.889µs] Acquire
2026-06-01 19:43:49.233 [ERROR] [sink:postgres] [db:postgres] failed to connect to `user=postgres database=postgres`:
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused
        127.0.0.1:5432 (localhost): dial error: dial tcp 127.0.0.1:5432: connect: connection refused

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingsinksWhere and how to store monitored data

Type

Fields

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions