CS-1187 add systemd and cgroups integration

CMakeLists.txt

@@ -88,6 +88,7 @@ option(WITH_LCOV "Enable code coverage analysis with lcov" OFF)
 option(WITH_PYTHON "Enable Python external bindings" OFF)
 option(WITH_BOOST "Enable Boost framework" OFF)
 option(WITH_MUNGE "Enable Munge authentication" ON)
+option(WITH_SYSTEMD "Enable systemd support" OFF)
 
 # private extensions
 set(PROJECT_EXTENSIONS "None" CACHE STRING "directory of private extensions")
@@ -188,6 +189,10 @@ if (WITH_MUNGE)
    add_compile_definitions("OCS_WITH_MUNGE")
 endif()
 
+if (WITH_SYSTEMD)
+   add_compile_definitions("OCS_WITH_SYSTEMD")
+endif()
+
 #if (SGE_ARCH MATCHES "darwin-arm64" OR SGE_ARCH MATCHES "fbsd-amd64")
 if (NOT WITH_SPOOL_BERKELEYDB AND NOT WITH_SPOOL_DYNAMIC)
    set(SPOOLING_LIBS spoolloader spoolc_static spool)

cmake/ArchitectureSpecificSettings.cmake

@@ -61,25 +61,7 @@ function(architecture_specific_settings)
       message("Build with extensions is enabled")
    endif()
 
-   if (SGE_ARCH MATCHES "lx-riscv64")
-      # Linux RiscV
-      message(STATUS "We are on Linux: ${SGE_ARCH}")
-      set(CMAKE_C_FLAGS "-Wall -Werror -pedantic" CACHE STRING "" FORCE)
-      set(CMAKE_CXX_FLAGS "-Wall -Werror -pedantic" CACHE STRING "" FORCE)
-
-      add_compile_definitions(LINUX _GNU_SOURCE GETHOSTBYNAME_R6 GETHOSTBYADDR_R8 HAS_IN_PORT_T SPOOLING_dynamic __SGE_COMPILE_WITH_GETTEXT__)
-      add_compile_options(-fPIC)
-      add_compile_options(-pthread)
-      add_link_options(-pthread -rdynamic)
-
-      set(TIRPC_INCLUDES /usr/include/tirpc PARENT_SCOPE)
-      set(TIRPC_LIB tirpc PARENT_SCOPE)
-      message(STATUS "using libtirpc")
-
-      set(WITH_JEMALLOC OFF PARENT_SCOPE)
-      set(WITH_MTMALLOC OFF PARENT_SCOPE)
-      set(JNI_ARCH "linux" PARENT_SCOPE)
-   elseif (SGE_ARCH MATCHES "lx-.*" OR SGE_ARCH MATCHES "ulx-.*" OR SGE_ARCH MATCHES "xlx-.*")
+   if (SGE_ARCH MATCHES "lx-.*" OR SGE_ARCH MATCHES "ulx-.*" OR SGE_ARCH MATCHES "xlx-.*")
       # master is not supported on CentOS 6. Execd is deprecated and will be removed in the future.
       if (SGE_ARCH STREQUAL "xlx-.*")
          set(INSTALL_SGE_BIN_MASTER OFF CACHE BOOL "Install master daemon binaries" FORCE)
@@ -166,6 +148,23 @@ function(architecture_specific_settings)
          message(STATUS "no libtirpc or libntirpc found")
       endif ()
 
+      # build with systemd?
+      # @todo we might want to check the api version, we need at least
+      #       - 235: here FreezeUnit and ThawUnit were added (not required, we work around this not being available)
+      #       - 231: 240? here sd_bus_process() was added (not required, we work around this)
+      #       - 221: here StopUnit was added
+      #       Our build hosts are OK as it is (RHEL-8 compatible for lx-* has a recent enough version,
+      #       RHEL-7 compatible for ulx-* does not have it at all)
+      if (EXISTS /usr/include/systemd/sd-bus.h)
+         set(WITH_SYSTEMD ON PARENT_SCOPE CACHE STRING "" FORCE)
+         message(STATUS "systemd development files found")
+      endif()
+
+      if (SGE_ARCH MATCHES "lx-riscv64")
+         # Linux RiscV
+         add_compile_options(-fPIC)
+         set(WITH_JEMALLOC OFF PARENT_SCOPE)
+      endif()
       if (SGE_ARCH STREQUAL "lx-x86" OR SGE_ARCH STREQUAL "ulx-x86" OR SGE_ARCH STREQUAL "xlx-x86")
          # we need patchelf for setting the run path in the db_* tools
          # but patchelf is not available on CentOS 7 x86

doc/markdown/man/man5/sge_conf.md

@@ -1044,11 +1044,19 @@ completely.
 
 ***ENABLE_BINDING***
 
-If this parameter is set then xxQS_NAMExx enables the core binding module within the execution daemon to apply 
-binding parameters that are specified during submission time of a job. This parameter is not set per default and 
+If this parameter is set, then xxQS_NAMExx enables the core binding module within the execution daemon to apply 
+binding parameters that are specified during submission time of a job. This parameter is not set per default, and 
 therefore all binding related information will be ignored. Find more information for job to core binding in the 
 section `-binding` of qsub(1).
 
+***ENABLE_SYSTEMD***
+
+If this parameter is set,
+and an execution hosts supports systemd, then jobs will be started in a systemd scope. This allows the execution daemon to
+manage the job's processes as a group, which is useful for resource management and job control.
+
+This parameter is set to true by default, meaning that on hosts that support systemd, jobs will be started in a systemd scope. If a host does not support systemd, then this parameter will be ignored.
+
 ***SCRIPT_TIMEOUT***
 
 This parameter allows to configure the allowed runtime of execution side scripts like prolog, epilog, and the PE 
@@ -1060,6 +1068,15 @@ in one load report interval. The default for *execd_params* is none.
 
 The global configuration entry for this value may be overwritten by the execution host local configuration.
 
+***USAGE_COLLECTION***
+
+This parameter controls how xxqs_name_sxx_execd collects the online usage information of jobs. The following values are recognized:
+
+- *FALSE* : No online usage information is collected. Use with care, this also disables limit enforcement for *s_cpu*, *h_cpu*, *s_rss*, *h_rss*, *s_vmem*, and *h_vmem*.
+- *PDC* : Online usage information is collected by the PDC (Portable Data Collector) mode, even if Systemd is available.
+- *HYBRID* : Hybrid mode, where online usage information is both gathered via Systemd (if available) and the PDC. Use this mode, when your jobs are controlled by systemd, but you also want to collect usage information for jobs that is not available via Systemd, e.g., vmem, maxvmem, io, and iow.
+- *TRUE* : This is the default mode. Online usage information is collected via Systemd if the host supports Systemd and *ENABLE_SYSTEMD* is set to *TRUE* (which is the default). It is collected by the PDC (Portable Data Collector) if the host does not support Systemd or if *ENABLE_SYSTEMD* is set to *FALSE*.
+
 ## gdi_request_limits
 
 This value is a global configuration parameter only, and is used to prevent denial-of-service attacks on the xxqs_name_sxx_qmaster(8) process.
@@ -1375,3 +1392,4 @@ xxqs_name_sxx_shepherd*(8), cron(8),
 # COPYRIGHT
 
 See xxqs_name_sxx_intro(1) for a full statement of rights and permissions.
+

doc/markdown/manual/development-guide/00_overview.md

@@ -34,7 +34,12 @@ Tags and branches before `V9` will also not be described here.
 |              | V900p1\_TAG | patch to the 9.0.0 making it work on GCP (CS-663) |
 |              |             |                                                   |
 | V90\_BRANCH  |             | maintenance of 9.0                                |
+|              | V903\_TAG   | third 9.0 patch                                   |
+|              | V904\_TAG   | fourth 9.0 patch                                  |
+|              | V905\_TAG   | fifth 9.0 patch                                   |
+|              | V906\_TAG   | sixth 9.0 patch                                   |
+|              | V907\_TAG   | seventh 9.0 patch                                 |
 |              |             |                                                   |
 
-[//]: # (Each file has to end with two emty lines)
+[//]: # (Each file has to end with two empty lines)
 

doc/markdown/manual/development-guide/01_prepare_dev_env.md

@@ -267,5 +267,5 @@ git clone https://github.com/hpc-gridware/gcs-extensions
 git clone https://github.com/hpc-gridware/gcs-testsuite
 ```
 
-[//]: # (Each file has to end with two emty lines)
+[//]: # (Each file has to end with two empty lines)
 

doc/markdown/manual/development-guide/02_build_configuration.md

@@ -188,5 +188,5 @@ Here we use *CLion* as example because it provides full integration with CMake t
 
 Next step is to build and install xxQS_NAMExx.
 
-[//]: # (Eeach file has to end with two emty lines)
+[//]: # (Eeach file has to end with two empty lines)
 

doc/markdown/manual/development-guide/03_build_and_installation.md

@@ -21,5 +21,5 @@ make install
 You can now either install the product (follow the instructions in the *Installation Guide*) or you can continue to 
 setup the automated test environment as described in the next chapter.
 
-[//]: # (Eeach file has to end with two emty lines)
+[//]: # (Eeach file has to end with two empty lines)
 

doc/markdown/manual/development-guide/05_simulating_hosts_and_execution.md

@@ -101,5 +101,5 @@ Instead, the job execution is just simulated.
 
 @todo add details
 
-[//]: # (Eeach file has to end with two emty lines)
+[//]: # (Eeach file has to end with two empty lines)
 

doc/markdown/manual/development-guide/25_scheduler_thread.md

@@ -296,5 +296,5 @@ we can switch off a few potentially expensive features and just rely on scheduli
 * do not configure queue load_thresholds and suspend_thresholds
 * do not use load adjustments (in the scheduler config)
 
-[//]: # (Eeach file has to end with two emty lines)
+[//]: # (Eeach file has to end with two empty lines)
 

doc/markdown/manual/installation-guide/01_planning_the_installation.md

@@ -280,7 +280,7 @@ Please refer to the next question for more information.
 
 ### Where is the spooling area for the master service located?
 
-For HA-setups, it must be a shared network location; otherwise, it can be the local filesystem of the host 
+For HA setups, it must be a shared network location; otherwise, it can be the local filesystem of the host 
 running the master service.
 
 Ensure that the spooling location meets the requirements of the spooling mechanism. Classic spooling can be done on 
@@ -359,4 +359,5 @@ If this is your first time installing xxQS_NAMExx, we suggest a manual installat
 Automatic installation is recommended if you need to install or reinstall a cluster multiple times or if you plan 
 to install multiple clusters with slightly different settings.
 
-[//]: # (Eeach file has to end with two emty lines)
+[//]: # (Eeach file has to end with two empty lines)
+

doc/markdown/manual/installation-guide/02_download.md

@@ -54,5 +54,5 @@ Once you have downloaded all packages, you can test and install them at the desi
 
 4. If your `<install-dir>` is located on a shared filesystem available on all hosts in the cluster then you can start the installation process.
 
-[//]: # (Eeach file has to end with two emty lines)
+[//]: # (Eeach file has to end with two empty lines)