Skip to content

Auto grow volume not working #4510

@gricey432

Description

@gricey432

Overview

Same title as #4319 but this is on the pgdata volume.

The auto grow volume monitor bash script seems to have a bug.

TOKEN=$(cat "${SERVICEACCOUNT}/token")

The TOKEN is only read once, outside the loop. This means that autogrow will function correctly immediately after the container is started (or restarted) but if growth occurs slowly the monitor becomes unable to act.

Checking the logs confirms this, showing a constant stream of 401 unauthorised errors.

kubectl logs hippo-main-hg94-0 -c replication-cert-copy --tail 10
                                 Dload  Upload   Total   Spent    Left  Speed
100   251  100   157  100    94  22428  13428 --:--:-- --:--:-- --:--:-- 35857
}{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401

This aligns with the experience we've had with CD/Snowflake support where restarting the pod "fixes it" but noone knows why.

Environment

Please provide the following details:

  • Platform: EKS
  • Platform Version: 1.34
  • PGO Image Tag: 5.8.5
  • Postgres Version: 14
  • Storage: gp3

Steps to Reproduce

  1. Turn on autogrow
  2. Wait for token expiry
  3. Fill up disk
  4. See no resize
  5. Restart container
  6. See resize

Proposed Fix

Move the TOKEN read (and possibly other reads) inside the loop.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions