|
| 1 | +--- |
| 2 | +title: "Tuning JVM memory settings for a Mendix app running in Mendix on Kubernetes" |
| 3 | +linktitle: "JVM memory tuning" |
| 4 | +url: /developerportal/deploy/private-cloud-jvm-memory-tuning/ |
| 5 | +description: "Describes how to tune a Mendix app's JVM memory settings in Mendix on Kubernetes" |
| 6 | +weight: 10 |
| 7 | +--- |
| 8 | +## Introduction |
| 9 | + |
| 10 | +Mendix Runtime and any Mendix apps run in a Java Virtual Machine. Most Java Virtual Machines (including the one used in Mendix on Kubernetes) are based on the HotSpot Java VM. The HotSpot VM uses a garbage collector to free up unused memory. However, running the garbage collector causes a negative performance impact. The Hotspot VM tries to only run garbage collection when absolutely necessary (for example, when memory usage reaches a certain threshold). |
| 11 | + |
| 12 | +{{% alert color="info" %}} |
| 13 | +For a comprehensive guide explaining how the HotSpot VM performs garbage collection, see the [official Java documentation](https://docs.oracle.com/en/java/javase/21/gctuning/index.html). |
| 14 | +{{% /alert %}} |
| 15 | + |
| 16 | +## Memory Tuning |
| 17 | + |
| 18 | +In most cases, Mendix on Kubernetes does not need advanced memory tuning. Starting with Operator version 2.26, the default memory settings have been aligned with Mendix Cloud and are based on typical Mendix app usage patterns. |
| 19 | + |
| 20 | +{{% alert color="warning" %}} |
| 21 | +Manual adjusting JVM memory usage is only needed in some cases, when the default memory settings leave unused memory. |
| 22 | +{{% /alert %}} |
| 23 | + |
| 24 | +### Types of Memory |
| 25 | + |
| 26 | +In the HotSpot VM, memory is split into two regions, **Heap** and **Non-Heap**. |
| 27 | + |
| 28 | +* **Heap** memory is used to store temporary data, including microflow variables, entities, and internal states of the Mendix Runtime and third-party libraries. |
| 29 | +* **Non-Heap** memory stores code, internal Java VM data and data from non-Java components, such as the [BAPI Connector](/appstore/modules/sap/sap-bapi-connector/#setup-environment) or machine learning modules from [ML Kit](/refguide/machine-learning-kit/). |
| 30 | + |
| 31 | +There is a fixed boundary between **Heap** and **Non-Heap** memory. Any memory reserved for the heap cannot be used for any other purpose. All remaining memory is assigned to the **Non-Heap** region. |
| 32 | + |
| 33 | +{{< figure src="/attachments/deployment/private-cloud/private-cloud-deploy/jvm-memory-overview.png" class="no-border" >}} |
| 34 | + |
| 35 | +It is not possible to dynamically adjust the amount of **Heap** and **Non-Heap** regions of a running app. This needs to be specified before starting an app. |
| 36 | + |
| 37 | +### Default Memory Allocation {#default-memory-allocation} |
| 38 | + |
| 39 | +Starting with Operator version 2.26, the default memory settings use the following rules: |
| 40 | + |
| 41 | +| Container memory limit | JVM non-heap size | |
| 42 | +| --- | --- | |
| 43 | +| Less than 2 GB | Half of the limit (but at least 300 MB) | |
| 44 | +| Between 2 and 4 GB | 1 GB | |
| 45 | +| Between 4 and 8 GB | 1.5 GB | |
| 46 | +| Between 8 and 16 GB | 2 GB | |
| 47 | +| Between 16 and 32 GB | 3 GB | |
| 48 | +| 32 GB or more | 4 GB | |
| 49 | + |
| 50 | +All other memory is allocated to the heap. |
| 51 | + |
| 52 | +{{% alert color="info" %}} |
| 53 | +Older Operator versions do not set any memory configuration. In this case, the HotSpot VM allocates 25% of the memory limit to heap memory, and the rest is assigned to non-heap memory. This assignment is not optimal, as typical Mendix apps primarily benefit from heap memory. |
| 54 | +{{% /alert %}} |
| 55 | + |
| 56 | +### Metrics-Based Tuning |
| 57 | + |
| 58 | +{{% alert color="info" %}} |
| 59 | +Instead of freeing memory, the HotSpot VM prefers to recycle/reuse it. When looking at total memory usage in a container (for example using `kubectl top pods` or data collected by `kube-state-metrics`), the total memory usage will not reflect internal details. For example, the JVM **Heap** might be underutilized, but will show as *used memory* in the container memory usage. To have a clear picture of how memory is allocated and used, use Prometheus metrics. |
| 60 | +{{% /alert %}} |
| 61 | + |
| 62 | +For examples how to read JVM memory usage graphs, see [Java Memory Usage](/refguide/java-memory-usage/). |
| 63 | + |
| 64 | +### Adjusting Memory in Mendix on Kubernetes |
| 65 | + |
| 66 | +In a [Standalone environment](/developerportal/deploy/private-cloud-operator/#edit-cr), the JVM heap size can be adjusted in the `jvmMemorySettings` section of the `MendixApp` CR. |
| 67 | + |
| 68 | +For example: |
| 69 | + |
| 70 | +```yaml |
| 71 | +apiVersion: privatecloud.mendix.com/v1alpha1 |
| 72 | +kind: MendixApp |
| 73 | +metadata: |
| 74 | +# ... |
| 75 | +# omitted lines for brevity |
| 76 | +# ... |
| 77 | +spec: |
| 78 | + # ... |
| 79 | + # omitted lines for brevity |
| 80 | + # ... |
| 81 | + # Add or update this section: |
| 82 | + jvmMemorySettings: |
| 83 | + heapLimit: 1700Mi # example: set heap limit to 1700 megabytes |
| 84 | +``` |
| 85 | +
|
| 86 | +Removing the `jvmMemorySettings` section will switch to the [default memory allocation](#default-memory-allocation) settings. |
| 87 | + |
| 88 | +Starting with Operator version 2.26, the JVM heap size should be adjusted as documented in this section. If you have an environment using any of the following custom JVM options, they will not be applied to the app configuration and will instead be removed: |
| 89 | + |
| 90 | +* `-Xms` - minimal heap size |
| 91 | +* `-Xmx` - heap size limit |
| 92 | +* `-XX:MinRAMPercentage` - heap size limit, in percentages for environments with less than 250MB of container memory |
| 93 | +* `-XX:MaxRAMPercentage` - heap size limit, in percentages for environments with more than 250MB of container memory |
| 94 | +* `-XX:InitialRAMPercentage` - minimal heap size, in percentages |
| 95 | + |
| 96 | +## Addressing Memory Issues |
| 97 | + |
| 98 | +On a typical desktop system and many server environments, reaching or exceeding the memory limit is not a problem. The operating system will move some of the memory contents to a storage device such as an SSD or hard drive. However, this will likely cause a major performance penalty, and in Kubernetes is [disabled by default](https://kubernetes.io/docs/concepts/cluster-administration/swap-memory-management/). |
| 99 | + |
| 100 | +When an app tries to use more memory than its container memory limit, it will be terminated by Kubernetes (OOMKilled) and will effectively crash. To prevent performance degradation and the app from crashing, it might need its memory to be adjusted. |
| 101 | + |
| 102 | +### Addressing Out Of Memory Crashes |
| 103 | + |
| 104 | +An *OOMKilled* (Out Of Memory-killed) event happens when an app attempts to use more memory than specified in its container limit: |
| 105 | + |
| 106 | +{{< figure src="/attachments/deployment/private-cloud/private-cloud-deploy/jvm-nonheap-full.png" class="no-border" >}} |
| 107 | + |
| 108 | +This shows as a container restarting with an `OOMKilled` exit code: |
| 109 | + |
| 110 | +{{< figure src="/attachments/deployment/private-cloud/private-cloud-deploy/jvm-memory-oomkilled.png" class="no-border" >}} |
| 111 | + |
| 112 | +To address this issue, choose one of the following options: |
| 113 | + |
| 114 | +* Recommended fix: increase the core resource memory (**requests** and **limits**). |
| 115 | +* Alternatively, if the Prometheus metrics show that the heap memory usage is low, decrease the heap memory size. |
| 116 | + |
| 117 | +{{% alert color="info" %}} |
| 118 | +This can also happen if a node is overcommitted (not enough available memory for app pods). To prevent overcommitment, adjust the memory **requests** to match the container **limits**. |
| 119 | +{{% /alert %}} |
| 120 | + |
| 121 | +### Addressing Java.lang.OutOfMemoryError Exceptions |
| 122 | + |
| 123 | +A `java.lang.OutOfMemoryError` exception happens when the JVM heap is full and does not have enough available memory to perform an action (such as a Microflow or a Java Custom Action). |
| 124 | + |
| 125 | +{{< figure src="/attachments/deployment/private-cloud/private-cloud-deploy/jvm-heap-full.png" class="no-border" >}} |
| 126 | + |
| 127 | +This shows as a `java.lang.OutOfMemoryError: Java heap space` error message in the logs. |
| 128 | + |
| 129 | +To address this issue, choose one of the following options: |
| 130 | + |
| 131 | +* Recommended fix: increase the core resource memory (**requests** and **limits**). |
| 132 | +* Alternatively, if the container has a lot of free, unused memory, increase the heap memory size. |
0 commit comments