Memory spikes in plugin-barman-cloud sidecar

[smiyc]

I have a strange memory behaviour, sometimes the memory usages spikes up to 35GB!
When the spike starts, there are no basebackups or WAL backups running.

The sidecar is running with logLevel: trace
here are the logs from the time the last spike started

plugin.log

as suggested by @mnencia the kube admin had a look into the cgroup of the pod.

These are the commands he used to

cd /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/
crictl --runtime-endpoint unix:///run/containerd/containerd.sock pods

...
POD ID              CREATED             STATE               NAME                                                 NAMESPACE           ATTEMPT             RUNTIME
ae2b9d6dca19a       2 hours ago         Ready               wldiva4t-1                                           cnpg-system         0                   (default)
...
...

crictl --runtime-endpoint unix:///run/containerd/containerd.sock inspectp ae2b9d6dca19a

    "createdAt": "2026-03-30T17:16:07.401632246Z",
    "id": "ae2b9d6dca19a6934a90a7a1506d89628eaaf704c82d3b1b894e265f9709c9e5",
    "labels": {
      "app.kubernetes.io/component": "database",
      "app.kubernetes.io/instance": "wldiva4t",
      "app.kubernetes.io/managed-by": "cloudnative-pg",
      "app.kubernetes.io/name": "postgresql",
      "app.kubernetes.io/version": "15",
      "cnpg.io/cluster": "wldiva4t",
      "cnpg.io/instanceName": "wldiva4t-1",
      "cnpg.io/podRole": "instance",
      "io.kubernetes.pod.name": "wldiva4t-1",
      "io.kubernetes.pod.namespace": "cnpg-system",
      "io.kubernetes.pod.uid": "bc27d9b9-03ac-4c40-8dd3-a79b753293dd",
      "release": "prometheus-operator",
      "topology.kubernetes.io/zone": "domain-c1011"

ls -tlraR|grep ae2b9d6dca19a6934a90a7a1506d89628eaaf704c82d3b1b894e265f9709c9e5

root@cl-postgres-test-worker-nodepool-a3-2gxzx-g7x7n-9k5f2:/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice# ls -tlraR|grep ae2b9d6dca19a6934a90a7a1506d89628eaaf704c82d3b1b894e265f9709c9e5
drwxr-xr-x  2 root root 0 Mar 30 17:16 cri-containerd-ae2b9d6dca19a6934a90a7a1506d89628eaaf704c82d3b1b894e265f9709c9e5.scope
./kubepods-burstable-podbc27d9b9_03ac_4c40_8dd3_a79b753293dd.slice/cri-containerd-ae2b9d6dca19a6934a90a7a1506d89628eaaf704c82d3b1b894e265f9709c9e5.scope:

/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc27d9b9_03ac_4c40_8dd3_a79b753293dd.slice/cri-containerd-ae2b9d6dca19a6934a90a7a1506d89628eaaf704c82d3b1b894e265f9709c9e5.scope

ls |while read a; do echo "=================="; echo "file: $a"; cat $a; echo "================="; done

Here is the output
cgroup.txt

[dosubot]

This is a documented issue area for plugin-barman-cloud. Memory growth in the sidecar has been reported and partially addressed across several versions [1] [2]. That said, 35GB spikes are significantly higher than previously reported cases (typically 10-12GB range), so there may be an additional factor at play here.

Key known causes of memory growth:

1. Python bytecode recompilation on read-only filesystem — this was the primary memory leak, fixed in v0.11.0 (PR fix: resolve WAL archiving performance and memory issues #746). Each barman-cloud invocation recompiled bytecode because .pyc files couldn't be written. If you're on an older version, upgrading should help significantly [2].

2. OS page cache accumulation — Kubernetes counts OS-level page cache as container memory (container_memory_working_set_bytes). The cache drop fix in v0.10.0 (PR feat: remove wal from cache after archiving #659) only runs when a WAL is archived, so during idle periods the cache isn't actively reclaimed [1].

3. ExtendedClient cache unbounded growth — expired cache entries for Secrets/ObjectStore objects are never cleaned up in long-running sidecars [3]. A fix (PR fix: prevent memory leak by periodically cleaning up expired cache entries #671) exists but hasn't been merged yet.

Recommendations:

• Check your plugin version — if you're not on v0.11.0+, upgrade first, as that fixes the biggest leak.
• Set memory limits on the sidecar via instanceSidecarConfiguration.resources.limits.memory in your ObjectStore spec. Even ~500Mi can work for moderate databases — setting a limit forces the kernel to reclaim cached memory more aggressively. Users confirmed this resolves the issue without triggering OOMKilled [4].
• Monitor container_memory_rss instead of container_memory_working_set_bytes to distinguish actual process memory from OS cache.
• Reduce log level from trace to info or debug — while trace logging alone is unlikely to cause 35GB spikes, it adds overhead at every RPC boundary (health checks, metrics scrapes, cache operations). Note: changing log level requires a pod restart.
• Enable pprof profiling on the sidecar to get actual Go heap usage and identify whether this is a real application leak vs. OS-level cache inflation.

Given the 35GB magnitude, it would also be helpful to share:

• Which plugin-barman-cloud version you're running
• The container_memory_rss metric alongside container_memory_working_set_bytes
• Your database size and WAL generation rate
• Whether memory limits are currently set on the sidecar

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}
   

[smiyc]

• The plugin version is 0.11.0
• sum by (pod) (container_memory_rss{pod="wldiva4t-1"}) = 192315392
• sum by (pod) (container_memory_working_set_bytes{pod="wldiva4t-1"}) = 75497467904
• no WAL's generated at spike time
• no sidecar requests / limits set

[dosubot]

This is very telling — your RSS is only ~183MB while working_set_bytes is ~70GB. That massive gap means this is almost entirely OS page cache (file-backed memory), not an application memory leak [1].

The core issue: without memory limits set on the sidecar, the Linux kernel has no pressure signal to reclaim cached pages from the cgroup, so page cache from file I/O accumulates indefinitely [2]. The retention policy loop, which runs every 30 minutes by default and downloads the full backup catalog from object storage each cycle [3], is likely the main contributor to this cache accumulation during idle periods.

The fix is to set memory limits on the sidecar. This gives the kernel the pressure it needs to reclaim cached pages. Users have confirmed this resolves the issue without triggering OOMKilled [1]. In your ObjectStore spec:

instanceSidecarConfiguration:
  resources:
    requests:
      memory: "500Mi"
      cpu: "250m"
    limits:
      memory: "500Mi"
      cpu: "250m"

One caveat: be aware that Kubernetes evaluates OOM kills based on working_set_bytes, not RSS [1]. So when you add limits, start with something reasonable (350-500Mi is the recommendation for databases in the 1-32GB range) and monitor — the kernel should start actively reclaiming page cache once it approaches the limit.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}
   

[smiyc]

requests / limits set
logLevel removed
will monitor the situation