CS-1187 add systemd and cgroups integration (#60)

* EH: CS-1188 control daemons with systemd

* avoid endless loop in case an invalid slice is given in the autoinstall template
// BelongsTo: CS-1188

* EH: CS-1192 at startup of daemons output the cgroups slice the service is running in

* fixed type "deamon"

* EH: CS-1223 with systemd integration, move sge_shepherd processes out of the sge_execd service cgroup

* sd_bus method StartTransientUnit does only start a job creating the unit
and returns before the action has actually finished.
Need to wait for the job to be finished.
// BelongsTo: CS-1123

* - do not report systemd as init system on ulx-* as we cannot build systemd support in sge_execd, libsystemd.so is too old
- fixed broken build on CentOS 8

* * sd_bus error was not reported to caller
* error messages were truncated at 100 characters, introduced SFN4 macro for 400 character strings

* fixed non-unique message ids

* EH: CS-1291 move shepherd child to its own scope

* shepherd tried to use systemd on host having systemd library but not having systemd as init system (Antix Linux)

* EH: CS-1292 get job online usage information via systemd

* tried to connect to systemd on host not having systemd

* errors in StartTransientUnit were not always propagated to caller

* EH: CS-1294 set job limits via systemd

* EH: CS-1315 set binding via systemd

* cleanup

* EH CS-1295 set device isolation via systemd

* EH: CS-1241 add profiling information for systemd operations

* - execd profiling could not be disabled again
- cleanup, moved code to own module
// BelongsTo: CS-1241

* EH: CS-1318 allow to run jobs under systemd control even if sge_execd itself is not started as systemd service

* EH: CS-1319 make running jobs under systemd control configurable

* added ENABLE_SYSTEMD to sge_conf.5 man page
// BelongsTo: CS-1319

* EH: CS-1322 the job specific scopes need to contain the toplevel slice name to be unique

* EH: CS-1300 do not add and handle the additional group id for jobs running under systemd

* BF: CS-1325 possible race condition between calling StartTransientUnit and waiting for the corresponding job to finish

* EH: CS-1296 kill jobs via systemd

* EH: CS-1321 allow to configure a hybrid usage data collection (both via systemd and the pdc)

* fixed memory leaks

* BF: CS-1335 need special handling for interrupted system call

* EH: CS-1342 add systemd specific settings (toplevel slice name) to the installation guide

* cleanup and added systemd integration to the release notes

* cleanup

* - addressed review comments
- fixed a race condition leading to multiple execd children trying to create the shepherds.scope

* added more details of the systemd integration to the release notes

* addressed review comments

* refactoring and documentation with Doxygen headers

* EH: CS-1408 USAGE_COLLECTION mode must be kept consistent for running jobs

* EH: CS-1419 disable systemd integration if sge_execd is started as non privileged user

* with HYBRID usage collection non systemd hosts didn't report cpu and rss

* reprioritization code was broken by systemd integration
// SeeAlso: CS-1421

* - improved diagnostics when ptf job / osjob cannot be found
- enforce cleanup in execd only when KEEP_ACTIVE is changed to FALSE

* BF: CS-1019 sge_execd logs errors when running tightly integrated parallel jobs

* BF: CS-1425 backup/restore does not handle $SGE_ROOT/$SGE_CELL/slice_name

* BF: CS-1429 sge_qmaster can segfault on qdel -f

* BF: CS-1019 sge_execd logs errors when running tightly integrated parallel jobs

* BF: CS-1430 running tightly integrated parallel jobs leaves systemd slices
// + additional cleanup

* fix to the fix for CS-1019

* added missing files

---------

Co-authored-by: Joachim Gabler <joga.oge@gabler-net.de>

Author: jgabler-hpc

CMakeLists.txt

@@ -88,6 +88,7 @@ option(WITH_LCOV "Enable code coverage analysis with lcov" OFF)
 option(WITH_PYTHON "Enable Python external bindings" OFF)
 option(WITH_BOOST "Enable Boost framework" OFF)
 option(WITH_MUNGE "Enable Munge authentication" ON)
+option(WITH_SYSTEMD "Enable systemd support" OFF)
 
 # private extensions
 set(PROJECT_EXTENSIONS "None" CACHE STRING "directory of private extensions")
@@ -188,6 +189,10 @@ if (WITH_MUNGE)
    add_compile_definitions("OCS_WITH_MUNGE")
 endif()
 
+if (WITH_SYSTEMD)
+   add_compile_definitions("OCS_WITH_SYSTEMD")
+endif()
+
 #if (SGE_ARCH MATCHES "darwin-arm64" OR SGE_ARCH MATCHES "fbsd-amd64")
 if (NOT WITH_SPOOL_BERKELEYDB AND NOT WITH_SPOOL_DYNAMIC)
    set(SPOOLING_LIBS spoolloader spoolc_static spool)

cmake/ArchitectureSpecificSettings.cmake

@@ -61,25 +61,7 @@ function(architecture_specific_settings)
       message("Build with extensions is enabled")
    endif()
 
-   if (SGE_ARCH MATCHES "lx-riscv64")
-      # Linux RiscV
-      message(STATUS "We are on Linux: ${SGE_ARCH}")
-      set(CMAKE_C_FLAGS "-Wall -Werror -pedantic" CACHE STRING "" FORCE)
-      set(CMAKE_CXX_FLAGS "-Wall -Werror -pedantic" CACHE STRING "" FORCE)
-
-      add_compile_definitions(LINUX _GNU_SOURCE GETHOSTBYNAME_R6 GETHOSTBYADDR_R8 HAS_IN_PORT_T SPOOLING_dynamic __SGE_COMPILE_WITH_GETTEXT__)
-      add_compile_options(-fPIC)
-      add_compile_options(-pthread)
-      add_link_options(-pthread -rdynamic)
-
-      set(TIRPC_INCLUDES /usr/include/tirpc PARENT_SCOPE)
-      set(TIRPC_LIB tirpc PARENT_SCOPE)
-      message(STATUS "using libtirpc")
-
-      set(WITH_JEMALLOC OFF PARENT_SCOPE)
-      set(WITH_MTMALLOC OFF PARENT_SCOPE)
-      set(JNI_ARCH "linux" PARENT_SCOPE)
-   elseif (SGE_ARCH MATCHES "lx-.*" OR SGE_ARCH MATCHES "ulx-.*" OR SGE_ARCH MATCHES "xlx-.*")
+   if (SGE_ARCH MATCHES "lx-.*" OR SGE_ARCH MATCHES "ulx-.*" OR SGE_ARCH MATCHES "xlx-.*")
       # master is not supported on CentOS 6. Execd is deprecated and will be removed in the future.
       if (SGE_ARCH STREQUAL "xlx-.*")
          set(INSTALL_SGE_BIN_MASTER OFF CACHE BOOL "Install master daemon binaries" FORCE)
@@ -166,6 +148,23 @@ function(architecture_specific_settings)
          message(STATUS "no libtirpc or libntirpc found")
       endif ()
 
+      # build with systemd?
+      # @todo we might want to check the api version, we need at least
+      #       - 235: here FreezeUnit and ThawUnit were added (not required, we work around this not being available)
+      #       - 231: 240? here sd_bus_process() was added (not required, we work around this)
+      #       - 221: here StopUnit was added
+      #       Our build hosts are OK as it is (RHEL-8 compatible for lx-* has a recent enough version,
+      #       RHEL-7 compatible for ulx-* does not have it at all)
+      if (EXISTS /usr/include/systemd/sd-bus.h)
+         set(WITH_SYSTEMD ON PARENT_SCOPE CACHE STRING "" FORCE)
+         message(STATUS "systemd development files found")
+      endif()
+
+      if (SGE_ARCH MATCHES "lx-riscv64")
+         # Linux RiscV
+         add_compile_options(-fPIC)
+         set(WITH_JEMALLOC OFF PARENT_SCOPE)
+      endif()
       if (SGE_ARCH STREQUAL "lx-x86" OR SGE_ARCH STREQUAL "ulx-x86" OR SGE_ARCH STREQUAL "xlx-x86")
          # we need patchelf for setting the run path in the db_* tools
          # but patchelf is not available on CentOS 7 x86

doc/markdown/man/man5/sge_conf.md

@@ -1044,11 +1044,19 @@ completely.
 
 ***ENABLE_BINDING***
 
-If this parameter is set then xxQS_NAMExx enables the core binding module within the execution daemon to apply 
-binding parameters that are specified during submission time of a job. This parameter is not set per default and 
+If this parameter is set, then xxQS_NAMExx enables the core binding module within the execution daemon to apply 
+binding parameters that are specified during submission time of a job. This parameter is not set per default, and 
 therefore all binding related information will be ignored. Find more information for job to core binding in the 
 section `-binding` of qsub(1).
 
+***ENABLE_SYSTEMD***
+
+If this parameter is set,
+and an execution hosts supports systemd, then jobs will be started in a systemd scope. This allows the execution daemon to
+manage the job's processes as a group, which is useful for resource management and job control.
+
+This parameter is set to true by default, meaning that on hosts that support systemd, jobs will be started in a systemd scope. If a host does not support systemd, then this parameter will be ignored.
+
 ***SCRIPT_TIMEOUT***
 
 This parameter allows to configure the allowed runtime of execution side scripts like prolog, epilog, and the PE 
@@ -1060,6 +1068,15 @@ in one load report interval. The default for *execd_params* is none.
 
 The global configuration entry for this value may be overwritten by the execution host local configuration.
 
+***USAGE_COLLECTION***
+
+This parameter controls how xxqs_name_sxx_execd collects the online usage information of jobs. The following values are recognized:
+
+- *FALSE* : No online usage information is collected. Use with care, this also disables limit enforcement for *s_cpu*, *h_cpu*, *s_rss*, *h_rss*, *s_vmem*, and *h_vmem*.
+- *PDC* : Online usage information is collected by the PDC (Portable Data Collector) mode, even if Systemd is available.
+- *HYBRID* : Hybrid mode, where online usage information is both gathered via Systemd (if available) and the PDC. Use this mode, when your jobs are controlled by systemd, but you also want to collect usage information for jobs that is not available via Systemd, e.g., vmem, maxvmem, io, and iow.
+- *TRUE* : This is the default mode. Online usage information is collected via Systemd if the host supports Systemd and *ENABLE_SYSTEMD* is set to *TRUE* (which is the default). It is collected by the PDC (Portable Data Collector) if the host does not support Systemd or if *ENABLE_SYSTEMD* is set to *FALSE*.
+
 ## gdi_request_limits
 
 This value is a global configuration parameter only, and is used to prevent denial-of-service attacks on the xxqs_name_sxx_qmaster(8) process.
@@ -1375,3 +1392,4 @@ xxqs_name_sxx_shepherd*(8), cron(8),
 # COPYRIGHT
 
 See xxqs_name_sxx_intro(1) for a full statement of rights and permissions.
+

doc/markdown/manual/development-guide/00_overview.md

@@ -34,7 +34,12 @@ Tags and branches before `V9` will also not be described here.
 |              | V900p1\_TAG | patch to the 9.0.0 making it work on GCP (CS-663) |
 |              |             |                                                   |
 | V90\_BRANCH  |             | maintenance of 9.0                                |
+|              | V903\_TAG   | third 9.0 patch                                   |
+|              | V904\_TAG   | fourth 9.0 patch                                  |
+|              | V905\_TAG   | fifth 9.0 patch                                   |
+|              | V906\_TAG   | sixth 9.0 patch                                   |
+|              | V907\_TAG   | seventh 9.0 patch                                 |
 |              |             |                                                   |
 
-[//]: # (Each file has to end with two emty lines)
+[//]: # (Each file has to end with two empty lines)
 

doc/markdown/manual/development-guide/01_prepare_dev_env.md

@@ -267,5 +267,5 @@ git clone https://github.com/hpc-gridware/gcs-extensions
 git clone https://github.com/hpc-gridware/gcs-testsuite
 ```
 
-[//]: # (Each file has to end with two emty lines)
+[//]: # (Each file has to end with two empty lines)
 

doc/markdown/manual/development-guide/02_build_configuration.md

@@ -188,5 +188,5 @@ Here we use *CLion* as example because it provides full integration with CMake t
 
 Next step is to build and install xxQS_NAMExx.
 
-[//]: # (Eeach file has to end with two emty lines)
+[//]: # (Eeach file has to end with two empty lines)
 

doc/markdown/manual/development-guide/03_build_and_installation.md

@@ -21,5 +21,5 @@ make install
 You can now either install the product (follow the instructions in the *Installation Guide*) or you can continue to 
 setup the automated test environment as described in the next chapter.
 
-[//]: # (Eeach file has to end with two emty lines)
+[//]: # (Eeach file has to end with two empty lines)
 

doc/markdown/manual/development-guide/05_simulating_hosts_and_execution.md

@@ -101,5 +101,5 @@ Instead, the job execution is just simulated.
 
 @todo add details
 
-[//]: # (Eeach file has to end with two emty lines)
+[//]: # (Eeach file has to end with two empty lines)
 

doc/markdown/manual/development-guide/25_scheduler_thread.md

@@ -296,5 +296,5 @@ we can switch off a few potentially expensive features and just rely on scheduli
 * do not configure queue load_thresholds and suspend_thresholds
 * do not use load adjustments (in the scheduler config)
 
-[//]: # (Eeach file has to end with two emty lines)
+[//]: # (Eeach file has to end with two empty lines)
 

doc/markdown/manual/installation-guide/01_planning_the_installation.md

@@ -280,7 +280,7 @@ Please refer to the next question for more information.
 
 ### Where is the spooling area for the master service located?
 
-For HA-setups, it must be a shared network location; otherwise, it can be the local filesystem of the host 
+For HA setups, it must be a shared network location; otherwise, it can be the local filesystem of the host 
 running the master service.
 
 Ensure that the spooling location meets the requirements of the spooling mechanism. Classic spooling can be done on 
@@ -359,4 +359,5 @@ If this is your first time installing xxQS_NAMExx, we suggest a manual installat
 Automatic installation is recommended if you need to install or reinstall a cluster multiple times or if you plan 
 to install multiple clusters with slightly different settings.
 
-[//]: # (Eeach file has to end with two emty lines)
+[//]: # (Eeach file has to end with two empty lines)
+