Retain large allocations on the heap by default (#1213)

* Modify default policy to keep large allocations on the heap and never auto-trim

* update Best_Performance.md

* Improve explanation of memory allocation policy

Clarify memory allocation behavior and performance impact.

---------

Co-authored-by: yspolyakov <89226542+yspolyakov@users.noreply.github.com>

Author: pascoec

docs/static_docs/Best_Performance.md

@@ -45,6 +45,24 @@ Within OpenFHE, the use of hyperthreading can lead to decreased performance so t
 
 If an alternative parallelization mechanism is used, e.g., pthreads, C++11 threads, or multiprocessing, OpenMP should be turned off by setting the `WITH_OPENMP` CMake flag to OFF.
 
+# Memory Allocation Policy
+
+OpenFHE allocates and frees many large polynomial buffers (~0.5–30 MB each) throughout normal operation. Under the default allocator policy, freed buffers of this size are returned to the OS as soon as they are released, resulting in many page faults when new buffers are requested in subsequent operations.
+When operations run back to back, this allocate / return / re-fault churn adds substantial page-fault and kernel overhead, resulting in reduced performance.
+
+To combat this, OpenFHE modifies the default policy at library load to keep large allocations on the heap and never return them to the OS. Freed buffers stay resident and are reused by subsequent allocation requests instead of being re-acquired from the OS (similar behaviour to a memory pool), which minimizes page faults and the associated performance penalty. This is applied automatically and requires no user action.
+
+The trade-off is that a process's resident set size (RSS) reflects its **peak** transient allocation rather than its steady-state working set — e.g., a process that runs `EvalBootstrap` and then idles keeps that peak footprint rather than releasing it.
+
+**If reducing RSS matters for your application, you must explicitly release the retained memory at the application layer** via:
+
+```cpp
+#include "utils/memory.h"
+lbcrypto::AllocTrim();   // return free heap memory to the OS
+```
+
+Call `AllocTrim()` at a quiescent point after large, transient-footprint work has completed and its buffers have been freed (e.g., after a loop over `EvalBootstrap`, or once a batch of operations finishes) and before the process idles; calling it mid-computation is wasteful, since the heap will simply re-grow on the next allocation. OpenFHE invokes `AllocTrim()` automatically on context teardown (`ReleaseAllContexts()`, `ClearStaticMapsAndVectors()`) but deliberately **not** after individual operations, since only the application knows when it has reached a genuinely low-footprint point.
+
 # Accelerating OpenFHE using Specialized Hardware Backends #
 
 OpenFHE supports multiple hardware acceleration backends. Currently, one such backend has been released based on the Intel HEXL library for Intel processors with AVX-512 support.

src/core/include/utils/memory.h

@@ -57,12 +57,24 @@ void MoveAppend(std::vector<X>& dst, std::vector<X>& src) {
 }
 
 /**
- * @brief secure_memset() is a function with the same functionality which is provided by std::memset.
+ * @brief secure_memset() is a function with the same functionality provided by std::memset.
  *        Usually, the compiler optimizes a call to std::memset out if it is called for a memory which goes out of scope.
  *        This function is never optimized out and used to re-initialize a memory for security reasons.
  */
 void secure_memset(volatile void* mem, uint8_t c, size_t len);
 
+/**
+ * @brief AllocTrim() returns free (unused) heap memory to the operating system.
+ *        It is the companion to OpenFHE's retain-by-default malloc tuning: call it at a quiescent point after a large
+ *        transient-footprint operation (e.g., EvalBootstrap) to reclaim peak scratch memory.
+ *
+ *        This is a performance/RSS utility, NOT a security primitive: it does not erase data. Use
+ *        secure_memset() to wipe sensitive memory before freeing. It is not async-signal-safe.
+ *
+ * @return true if a trim was attempted, false on unsupported platforms.
+ */
+bool AllocTrim();
+
 }  // namespace lbcrypto
 
 #endif  // LBCRYPTO_UTILS_MEMORY_H

src/core/lib/utils/memory.cpp

@@ -30,8 +30,51 @@
 //==================================================================================
 #include "utils/memory.h"
 
+#if defined(__GLIBC__)
+    #include <malloc.h>
+#endif
+
 namespace lbcrypto {
 
+#if defined(__GLIBC__)
+// Tune glibc malloc for FHE allocation patterns. Each homomorphic operation allocates and
+// frees many large (~0.5-30 MB) polynomial coefficient buffers. By default glibc returns
+// these freed buffers to the OS (brk heap-trim / munmap) and re-faults the pages on the next
+// operation; under a single thread this page-fault / kernel-time overhead can dominate
+// runtime. Keeping large allocations on the heap (M_MMAP_MAX=0) and never trimming it back to
+// the OS (M_TRIM_THRESHOLD=-1) lets the buffers be reused, eliminating the churn. Runs once,
+// process-wide, at library load. NOTE: this retains freed memory in-process, so workloads
+// with large transient allocation peaks will show a higher resident set size. Use AllocTrim()
+// after large allocations are freed if this becomes an issue.
+namespace {
+[[maybe_unused]] const bool ofheGlibcMallocTuned = []() noexcept {
+    mallopt(M_MMAP_MAX, 0);
+    mallopt(M_TRIM_THRESHOLD, -1);
+    return true;
+}();
+}  // namespace
+#endif
+
+// AllocTrim() asks the allocator to return free (unused) heap memory to the OS. It is the
+// companion to the retain-by-default malloc tuning above: call it at a quiescent point after a
+// large transient-footprint operation (e.g. EvalBootstrap) to reclaim peak scratch memory.
+//
+// WARNING: this is a performance / RSS tool, NOT a security primitive. It does not erase data --
+// it only releases already-free chunks (the bytes are not scrubbed; the kernel zero-fills pages
+// only when they are next handed out). To wipe sensitive data (keys, plaintext, noise) use
+// secure_memset() before freeing.
+//
+// Not async-signal-safe and acquires allocator locks; call only from normal execution context,
+// never from a signal handler. Returns true if a trim was attempted, false on unsupported builds.
+bool AllocTrim() {
+#if defined(__GLIBC__)
+    malloc_trim(0);
+    return true;
+#else
+    return false;
+#endif
+}
+
 void secure_memset(volatile void* mem, uint8_t c, size_t len) {
     volatile uint8_t* ptr = (volatile uint8_t*)mem;
     for (size_t i = 0; i < len; ++i)

src/pke/include/cryptocontextfactory.h

@@ -35,6 +35,7 @@
 #include "cryptocontext-fwd.h"
 #include "lattice/lat-hal.h"
 #include "scheme/scheme-id.h"
+#include "utils/memory.h"
 
 #include <memory>
 #include <string>
@@ -65,8 +66,8 @@ class CryptoContextFactory {
     static void ReleaseAllContexts() {
         if (!AllContexts.empty())
             AllContexts[0]->ClearStaticMapsAndVectors();
-
         AllContexts.clear();
+        AllocTrim();
     }
 
     static int GetContextCount() {

src/pke/lib/cryptocontext.cpp

@@ -39,6 +39,7 @@
 #include "math/chebyshev.h"
 #include "scheme/ckksrns/ckksrns-cryptoparameters.h"
 #include "schemerns/rns-scheme.h"
+#include "utils/memory.h"
 
 namespace lbcrypto {
 
@@ -63,6 +64,7 @@ void CryptoContextImpl<Element>::ClearStaticMapsAndVectors() {
 #ifdef WITH_NTL
     NTL::ChineseRemainderTransformFTTNtl<M6Vector>().Reset();
 #endif
+    AllocTrim();
 }
 
 template <typename Element>