pre-release WLO 1.5.0 - pods not scaling down when memory usage is less than target memory

[tam512]

Test pre-release WebSphere Liberty operator 1.5.0, test with Daytrader10 application, deploy the application image to Amazon EKS with Kubernetes v1.33

% oc get nodes -o wide
NAME                                           STATUS   ROLES    AGE   VERSION               INTERNAL-IP      EXTERNAL-IP    OS-IMAGE                       KERNEL-VERSION                   CONTAINER-RUNTIME
ip-192-168-15-251.us-east-2.compute.internal   Ready    <none>   20h   v1.33.3-eks-3abbec1   192.168.15.251   52.15.125.74   Amazon Linux 2023.8.20250818   6.12.40-63.114.amzn2023.x86_64   containerd://1.7.27
ip-192-168-66-240.us-east-2.compute.internal   Ready    <none>   20h   v1.33.3-eks-3abbec1   192.168.66.240   3.15.204.211   Amazon Linux 2023.8.20250818   6.12.40-63.114.amzn2023.x86_64   containerd://1.7.27

Deployed app with the following settings

spec:
  autoscaling:
    minReplicas: 2
    maxReplicas: 10
    targetMemoryUtilizationPercentage: 80
    behavior:
      scaleUp:
        tolerance: 0.02
      scaleDown:
        tolerance: 0.02
  resources:
    requests:
      cpu: 500m
      memory: 500Mi
    limits:
      cpu: 500m
      memory: 1200Mi

Run 10 minutes stress test and monitor hpa during the test run

% oc -n daytrader10 get hpa -w
NAME          REFERENCE                 TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
daytrader10   StatefulSet/daytrader10   memory: 51%/80%   2         10        2          2m19s
daytrader10   StatefulSet/daytrader10   memory: 51%/80%   2         10        2          2m30s
daytrader10   StatefulSet/daytrader10   memory: 57%/80%   2         10        2          2m45s
daytrader10   StatefulSet/daytrader10   memory: 55%/80%   2         10        2          3m
daytrader10   StatefulSet/daytrader10   memory: 56%/80%   2         10        2          3m15s
daytrader10   StatefulSet/daytrader10   memory: 56%/80%   2         10        2          3m30s
daytrader10   StatefulSet/daytrader10   memory: 56%/80%   2         10        2          3m45s
daytrader10   StatefulSet/daytrader10   memory: 56%/80%   2         10        2          4m
daytrader10   StatefulSet/daytrader10   memory: 56%/80%   2         10        2          4m15s
daytrader10   StatefulSet/daytrader10   memory: 57%/80%   2         10        2          4m30s
daytrader10   StatefulSet/daytrader10   memory: 57%/80%   2         10        2          4m45s
daytrader10   StatefulSet/daytrader10   memory: 62%/80%   2         10        2          5m
daytrader10   StatefulSet/daytrader10   memory: 66%/80%   2         10        2          5m15s
daytrader10   StatefulSet/daytrader10   memory: 66%/80%   2         10        2          5m30s
daytrader10   StatefulSet/daytrader10   memory: 69%/80%   2         10        2          5m45s
daytrader10   StatefulSet/daytrader10   memory: 68%/80%   2         10        2          6m
daytrader10   StatefulSet/daytrader10   memory: 71%/80%   2         10        2          6m16s
daytrader10   StatefulSet/daytrader10   memory: 73%/80%   2         10        2          6m31s
daytrader10   StatefulSet/daytrader10   memory: 74%/80%   2         10        2          6m46s
daytrader10   StatefulSet/daytrader10   memory: 74%/80%   2         10        2          7m1s
daytrader10   StatefulSet/daytrader10   memory: 74%/80%   2         10        2          7m16s
daytrader10   StatefulSet/daytrader10   memory: 76%/80%   2         10        2          7m31s
daytrader10   StatefulSet/daytrader10   memory: 77%/80%   2         10        2          7m46s
daytrader10   StatefulSet/daytrader10   memory: 77%/80%   2         10        2          8m1s
daytrader10   StatefulSet/daytrader10   memory: 78%/80%   2         10        2          8m16s
daytrader10   StatefulSet/daytrader10   memory: 79%/80%   2         10        2          8m31s
daytrader10   StatefulSet/daytrader10   memory: 79%/80%   2         10        2          8m46s
daytrader10   StatefulSet/daytrader10   memory: 80%/80%   2         10        2          9m1s
daytrader10   StatefulSet/daytrader10   memory: 81%/80%   2         10        2          9m16s
daytrader10   StatefulSet/daytrader10   memory: 82%/80%   2         10        2          9m31s
daytrader10   StatefulSet/daytrader10   memory: 83%/80%   2         10        2          9m46s
daytrader10   StatefulSet/daytrader10   memory: 83%/80%   2         10        2          10m
daytrader10   StatefulSet/daytrader10   memory: 85%/80%   2         10        2          10m
daytrader10   StatefulSet/daytrader10   memory: 85%/80%   2         10        2          10m
daytrader10   StatefulSet/daytrader10   memory: 86%/80%   2         10        2          10m
daytrader10   StatefulSet/daytrader10   memory: 87%/80%   2         10        2          11m
daytrader10   StatefulSet/daytrader10   memory: 87%/80%   2         10        2          11m
daytrader10   StatefulSet/daytrader10   memory: 90%/80%   2         10        2          11m
daytrader10   StatefulSet/daytrader10   memory: 88%/80%   2         10        3          11m
daytrader10   StatefulSet/daytrader10   memory: 67%/80%   2         10        3          12m
daytrader10   StatefulSet/daytrader10   memory: 73%/80%   2         10        3          12m
daytrader10   StatefulSet/daytrader10   memory: 76%/80%   2         10        3          12m
daytrader10   StatefulSet/daytrader10   memory: 78%/80%   2         10        3          12m
daytrader10   StatefulSet/daytrader10   memory: 78%/80%   2         10        3          13m
daytrader10   StatefulSet/daytrader10   memory: 82%/80%   2         10        3          13m
daytrader10   StatefulSet/daytrader10   memory: 83%/80%   2         10        3          13m
daytrader10   StatefulSet/daytrader10   memory: 86%/80%   2         10        3          13m
daytrader10   StatefulSet/daytrader10   memory: 85%/80%   2         10        3          14m
daytrader10   StatefulSet/daytrader10   memory: 86%/80%   2         10        3          14m
daytrader10   StatefulSet/daytrader10   memory: 86%/80%   2         10        3          14m
daytrader10   StatefulSet/daytrader10   memory: 87%/80%   2         10        3          14m
daytrader10   StatefulSet/daytrader10   memory: 87%/80%   2         10        3          15m
daytrader10   StatefulSet/daytrader10   memory: 89%/80%   2         10        3          15m
daytrader10   StatefulSet/daytrader10   memory: 89%/80%   2         10        4          15m
daytrader10   StatefulSet/daytrader10   memory: 73%/80%   2         10        4          15m
daytrader10   StatefulSet/daytrader10   memory: 75%/80%   2         10        4          16m
daytrader10   StatefulSet/daytrader10   memory: 77%/80%   2         10        4          16m
daytrader10   StatefulSet/daytrader10   memory: 78%/80%   2         10        4          16m
daytrader10   StatefulSet/daytrader10   memory: 83%/80%   2         10        4          16m

After the test done running and the memory usage is at 69%/80%, app pods is still at 4 and not scaling down to 2 (minReplicas)

daytrader10   StatefulSet/daytrader10   memory: 69%/80%   2         10        4          54m
daytrader10   StatefulSet/daytrader10   memory: 69%/80%   2         10        4          54m
daytrader10   StatefulSet/daytrader10   memory: 69%/80%   2         10        4          54m
daytrader10   StatefulSet/daytrader10   memory: 69%/80%   2         10        4          55m
..........
daytrader10   StatefulSet/daytrader10   memory: 68%/80%   2         10        4          65m
daytrader10   StatefulSet/daytrader10   memory: 68%/80%   2         10        4          66m

[leochr]

@tam512 The average memory usage is still high: 68%. Given that there are 4 pods, it wouldn't be possible for HPA to scale down as that'll result in exceeding the target of 80% ((68 * 4) / 3 = 90%).

Also, tolerance within scaleDown and scaleUp is an alpha feature in Kubernetes 1.33+, hence, as Melissa mentioned in Slack, it's not enabled by default. HPAConfigurableTolerance feature gate must be enabled. Not sure if it was enabled in this cluster, but wanted to point it out in case not. More info is here. Thank you.

[tam512]

I didn't have HPAConfigurableTolerance feature gate enable in my cluster.