DES-7259 Documented how to tune JVM memory.

Explained why JVM memory tuning might be needed, what parameters are
user-adjustable and how to deal with memory managemend problems.

Author: zlogic

…nt/private-cloud/private-cloud-deploy.md …ate-cloud/private-cloud-deploy/_index.mdcontent/en/docs/deployment/private-cloud/private-cloud-deploy.md renamed to content/en/docs/deployment/private-cloud/private-cloud-deploy/_index.md content/en/docs/deployment/private-cloud/private-cloud-deploy.md renamed to content/en/docs/deployment/private-cloud/private-cloud-deploy/_index.md

@@ -0,0 +1,154 @@
+---
+title: "Tuning JVM memory settings for a Mendix app running in Mendix on Kubernetes"
+linktitle: "JVM memory tuning"
+url: /developerportal/deploy/private-cloud-jvm-memory-tuning/
+description: "Describes how to tune a Mendix app's JVM memory settings in Mendix on Kubernetes"
+weight: 10
+---
+## Introduction
+
+Mendix Runtime and any Mendix apps run in a Java Virtual Machine.
+
+Most Java Virtual Machines (including the one used in Mendix on Kubernetes) are based on the HotSpot Java VM.
+The HotSpot VM uses a garbage collector to free up unused memory.o
+
+Running the garbage collector causes a negative performance impact; the Hotspot VM tries to only run garbage collection when absolutely necessary (for example, when memory usage reaches a certain threshold).
+
+{{% alert color="info" %}}
+For a comprehensive guide explaining how the HotSpot VM performs garbage collection, see the [official Java documentation](https://docs.oracle.com/en/java/javase/21/gctuning/index.html).
+{{% /alert %}}
+
+## Memory tuning
+
+In most cases, Mendix on Kubernetes doesn't need advanced memory tuning.
+Starting with Operator version 2.26, the default memory settings have been aligned with Mendix Cloud and are based on typical Mendix app usage patterns.
+
+Manual adjusting JVM memory usage is only needed in some cases - when the default memory settings leave unused memory.
+
+### Types of memory
+
+In the HotSpot VM, memory is split into two regions, **Heap** and **Non-Heap**.
+
+* **Heap** memory is used to store temporary _data_, including microflow variables, entities and internal state of the Mendix Runtime and third-party libraries.
+* **Non-Heap** memory stores _code_, internal Java VM data and data from non-Java components, such as the [BAPI Connector](/appstore/modules/sap/sap-bapi-connector/#setup-environment) or machine learning modules from [ML Kit](/refguide/machine-learning-kit/).
+
+There's a fixed boundary between **Heap** and **Non-Heap** memory: any memory reserved for the **Heap** cannot be used for any other purpose; all remaining memory will be assigned to the **Non-Heap** region.
+
+{{< figure src="/attachments/deployment/private-cloud/private-cloud-deploy/jvm-memory-overview.png" class="no-border" >}}
+
+It's not possible to dynamically adjust the amount of **Heap** and **Non-Heap** regions of a running app, this needs to be specified before starting an app.
+
+### Default memory allocation{#default-memory-allocation}
+
+Starting with Operator version 2.26, the default memory settings use the following rules:
+
+| Container memory limit | JVM non-heap size |
+|------------------------|-------------------|
+| Less than 2 GB         | Half of the limit (but at least 300 MB) |
+| Between 2 and 4 GB     | 1 GB              |
+| Between 4 and 8 GB     | 1.5 GB            |
+| Between 8 and 16 GB    | 2 GB              |
+| Between 16 and 32 GB   | 3 GB              |
+| 32 GB or more          | 4 GB              |
+
+All other memory is allocated to the heap.
+
+{{% alert color="info" %}}
+Older Operator versions don't set any memory configuration.
+In this case, the HotSpot VM allocates 25% of the memory limit to heap memory, and the rest is assigned to non-heap memory.
+This assignment is not optimal, as typical Mendix apps primarily benefit from heap memory.
+{{% /alert %}}
+
+### Metrics-based tuning
+
+{{% alert color="info" %}}
+Instead of freeing memory, the HotSpot VM prefers to recycle/reuse it.
+When looking at total memory usage in a container (for example using `kubectl top pods` or data collected by `kube-state-metrics`),
+the total memory usage will not reflect internal details.
+
+For example, the JVM **Heap** might be underutilized, but will show as "used memory" in the container memory usage.
+
+To have a clear picture of how memory is allocated and used, use Prometheus metrics.
+{{% /alert %}}
+
+For examples how to read JVM memory usage graphs, see the [Java Memory Usage](/refguide/java-memory-usage/) document.
+
+### Adjusting memory in Mendix on Kubernetes
+
+In a [Standalone environment](/developerportal/deploy/private-cloud-operator/#edit-cr), the JVM heap size can be adjusted in the `jvmMemorySettings` section of the `MendixApp` CR.
+
+For example:
+
+```yaml
+apiVersion: privatecloud.mendix.com/v1alpha1
+kind: MendixApp
+metadata:
+# ...
+# omitted lines for brevity
+# ...
+spec:
+  # ...
+  # omitted lines for brevity
+  # ...
+  # Add or update this section:
+  jvmMemorySettings:
+    heapLimit: 1700Mi # example: set heap limit to 1700 megabytes
+```
+
+Removing the `jvmMemorySettings` section will switch to the [default memory allocation](#default-memory-allocation) settings.
+
+<!-- TODO: show how to adjust memory in Portunus when DES-7177 is merged -->
+
+Starting with Operator version 2.26, the JVM heap size should be adjusted as documented in this section.
+If you have an environment using any of the following custom JVM options, they _will not be applied_ to the app configuration (will be removed):
+
+* `-Xms` - minimal heap size
+* `-Xmx` - heap size limit
+* `-XX:MinRAMPercentage` - heap size limit, in percentages for environments with less than 250MB of container memory
+* `-XX:MaxRAMPercentage` - heap size limit, in percentages for environments with more than 250MB of container memory
+* `-XX:InitialRAMPercentage` - minimal heap size, in percentages
+
+## Addressing memory issues
+
+On a typical desktop system and many server environments, reaching or exceeding the memory limit is not a problem.
+
+The operating system will move some of the memory contents to a storage device such as an SSD or hard drive.
+However, this will likely cause a major performance penalty, and in Kubernetes is [disabled by default](https://kubernetes.io/docs/concepts/cluster-administration/swap-memory-management/).
+
+When an app tries to use more memory than it's container memory limit, it will be terminated by Kubernetes (OOMKilled) and will effectively crash.
+
+To prevent performance degradation and the app from crashing, it might need its memory to be adjusted.
+
+### Addressing OOMKilled crashes
+
+An OOMKilled (Out Of Memory-killed) event happens when an app attempts to use more memory than specified in its container limit:
+
+{{< figure src="/attachments/deployment/private-cloud/private-cloud-deploy/jvm-nonheap-full.png" class="no-border" >}}
+
+This will show as a container restarting with an `OOMKilled` exit code:
+
+{{< figure src="/attachments/deployment/private-cloud/private-cloud-deploy/jvm-memory-oomkilled.png" class="no-border" >}}
+
+To address this issue:
+
+* Recommended fix: increase the core resource memory (_requests_ and _limits_).
+* Alternatively, if the Prometheus metrics show that the heap memory usage is low: decrease the heap memory size.
+
+{{% alert color="info" %}}
+This can also happen if a node is overcommitted (not enough available memory for app pods).
+
+To prevent overcommitment, adjust the memory _requests_ to match the container _limits_.
+{{% /alert %}}
+
+### Addressing java.lang.OutOfMemoryError exceptions
+
+A `java.lang.OutOfMemoryError` exception happens when the JVM **Heap** is full and doesn't have enough available memory to perform an action (such as a Microflow or a Java Custom Action).
+
+{{< figure src="/attachments/deployment/private-cloud/private-cloud-deploy/jvm-heap-full.png" class="no-border" >}}
+
+This will show as a `java.lang.OutOfMemoryError: Java heap space` error message in the logs.
+
+To address this issue:
+
+* Recommended fix: increase the core resource memory (_requests_ and _limits_).
+* Alternatively, if the container has a lot of free, unused memory: increase the heap memory size.

content/en/docs/deployment/private-cloud/private-cloud-deploy/jvm-memory-tuning.md

@@ -84,6 +84,8 @@ spec:
       cpu: 250m
       memory: 256Mi
       ephemeral-storage: 256Mi
+  jvmMemorySettings: # Optional, can be omitted: set custom Java VM memory configuration
+    heapLimit: 100Mi # set heap limit to 100 megabytes
   runtimeDeploymentPodAnnotations: # Optional, can be omitted : set custom annotations for Mendix Runtime Pods
     # example: inject the Linkerd proxy sidecar
     linkerd.io/inject: enabled
@@ -195,6 +197,7 @@ You must make the following changes:
 * **certificate** and **key** - Not recommended for production environments. Provide the `tls.crt` and `tls.key` values directly. Cannot be used together with **secretName**.
 * **replicas** - By default, one replica is started when you deploy your app.
 * **resources** - Change the minimum and maximum container resources your app requires.
+* **jvmMemorySettings** - Change the JVM memory configuration. For more information, see [JVM memory tuning](/developerportal/deploy/private-cloud-jvm-memory-tuning/).
 * **serviceAnnotations** - Set custom annotations for network Services. These annotations are applied on top of [default annotations](/developerportal/deploy/private-cloud-cluster/#advanced-network-settings) from `OperatorConfiguration`.
 * **endpointAnnotations** - Set custom annotations for Ingress (or OpenShift Route) objects. These annotations are applied on top of [default annotations](/developerportal/deploy/private-cloud-cluster/#advanced-network-settings) from `OperatorConfiguration`.
 * **ingressPath** - Specify a custom Ingress path. This overrides the [default ingress path](/developerportal/deploy/private-cloud-cluster/#advanced-network-settings) from `OperatorConfiguration`.