Overview
Same title as #4319 but this is on the pgdata volume.
The auto grow volume monitor bash script seems to have a bug.
|
TOKEN=$(cat "${SERVICEACCOUNT}/token") |
The TOKEN is only read once, outside the loop. This means that autogrow will function correctly immediately after the container is started (or restarted) but if growth occurs slowly the monitor becomes unable to act.
Checking the logs confirms this, showing a constant stream of 401 unauthorised errors.
kubectl logs hippo-main-hg94-0 -c replication-cert-copy --tail 10
Dload Upload Total Spent Left Speed
100 251 100 157 100 94 22428 13428 --:--:-- --:--:-- --:--:-- 35857
}{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "Unauthorized",
"reason": "Unauthorized",
"code": 401
This aligns with the experience we've had with CD/Snowflake support where restarting the pod "fixes it" but noone knows why.
Environment
Please provide the following details:
- Platform: EKS
- Platform Version: 1.34
- PGO Image Tag: 5.8.5
- Postgres Version: 14
- Storage: gp3
Steps to Reproduce
- Turn on autogrow
- Wait for token expiry
- Fill up disk
- See no resize
- Restart container
- See resize
Proposed Fix
Move the TOKEN read (and possibly other reads) inside the loop.
Overview
Same title as #4319 but this is on the pgdata volume.
The auto grow volume monitor bash script seems to have a bug.
postgres-operator/internal/postgres/config.go
Line 327 in 668bcae
The
TOKENis only read once, outside the loop. This means that autogrow will function correctly immediately after the container is started (or restarted) but if growth occurs slowly the monitor becomes unable to act.Checking the logs confirms this, showing a constant stream of 401 unauthorised errors.
This aligns with the experience we've had with CD/Snowflake support where restarting the pod "fixes it" but noone knows why.
Environment
Please provide the following details:
Steps to Reproduce
Proposed Fix
Move the TOKEN read (and possibly other reads) inside the loop.