Merging Orchestrator with RHDH chart (#109)

* Orchestrator merged with backstage chart

* Readme changes

* changes to chart version and value lint issue

* Changes after PR review

* chart bump

* updated plugins to correct versions

* correct orchestrator plugins - official

* network policy pod selection + servicebinding default

* chart version bump

* orchestrator value flattened

* values json update

* Exist check for sonataflowplatform

* sonataflow job working for local psql

* Integrates sonataflowplatform with external DB + readme instructions

* making job image configurable

* moved wait to init container

* configurable images and comments

* chart bump

* chart bump to 3.5.0

* lint issues

* bumped version to 4.0.0

* conditionals for templates and error messages

Author: elai-shalev

charts/backstage/Chart.yaml

@@ -44,4 +44,4 @@ sources:
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 3.4.0
+version: 4.0.0

charts/backstage/README.md

@@ -2,7 +2,7 @@
 # RHDH Backstage Helm Chart for OpenShift (Community Version)
 
 [![Artifact Hub](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/rhdh-chart&style=flat-square)](https://artifacthub.io/packages/search?repo=rhdh-chart)
-![Version: 3.4.0](https://img.shields.io/badge/Version-3.4.0-informational?style=flat-square)
+![Version: 4.0.0](https://img.shields.io/badge/Version-4.0.0-informational?style=flat-square)
 ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square)
 
 A Helm chart for deploying Red Hat Developer Hub.
@@ -191,6 +191,20 @@ Kubernetes: `>= 1.25.0-0`
 | global.dynamic.includes[0] | List of dynamic plugins included inside the `janus-idp/backstage-showcase` container image, some of which are disabled by default. This file ONLY works with the `janus-idp/backstage-showcase` container image. | string | `"dynamic-plugins.default.yaml"` |
 | global.dynamic.plugins | List of dynamic plugins, possibly overriding the plugins listed in `includes` files. Every item defines the plugin `package` as a [NPM package spec](https://docs.npmjs.com/cli/v10/using-npm/package-spec), an optional `pluginConfig` with plugin-specific backstage configuration, and an optional `disabled` flag to disable/enable a plugin listed in `includes` files. It also includes an `integrity` field that is used to verify the plugin package [integrity](https://w3c.github.io/webappsec-subresource-integrity/#integrity-metadata-description). | list | `[]` |
 | global.host | Custom hostname shorthand, overrides `global.clusterRouterBase`, `upstream.ingress.host`, `route.host`, and url values in `upstream.backstage.appConfig`. | string | `""` |
+| orchestrator.enabled |  | bool | `false` |
+| orchestrator.serverlessLogicOperator.enabled |  | bool | `false` |
+| orchestrator.serverlessOperator.enabled |  | bool | `false` |
+| orchestrator.sonataflowPlatform.createDBJobImage | Image for the container used by the create-db job | string | `"postgres:15"` |
+| orchestrator.sonataflowPlatform.eventing.broker.name |  | string | `""` |
+| orchestrator.sonataflowPlatform.eventing.broker.namespace |  | string | `""` |
+| orchestrator.sonataflowPlatform.externalDBName | Name for the user-configured external Database | string | `""` |
+| orchestrator.sonataflowPlatform.externalDBsecretRef | Secret name for the user-created secret to connect an external DB | string | `""` |
+| orchestrator.sonataflowPlatform.initContainerImage | Image for the init container used by the create-db job | string | `"busybox"` |
+| orchestrator.sonataflowPlatform.monitoring.enabled |  | bool | `true` |
+| orchestrator.sonataflowPlatform.resources.limits.cpu |  | string | `"500m"` |
+| orchestrator.sonataflowPlatform.resources.limits.memory |  | string | `"1Gi"` |
+| orchestrator.sonataflowPlatform.resources.requests.cpu |  | string | `"250m"` |
+| orchestrator.sonataflowPlatform.resources.requests.memory |  | string | `"64Mi"` |
 | route | OpenShift Route parameters | object | `{"annotations":{},"enabled":true,"host":"{{ .Values.global.host }}","path":"/","tls":{"caCertificate":"","certificate":"","destinationCACertificate":"","enabled":true,"insecureEdgeTerminationPolicy":"Redirect","key":"","termination":"edge"},"wildcardPolicy":"None"}` |
 | route.annotations | Route specific annotations | object | `{}` |
 | route.enabled | Enable the creation of the route resource | bool | `true` |
@@ -320,3 +334,30 @@ upstream:
     volumePermissions:
       enabled: true
 ```
+
+## Installing RHDH with Orchestrator
+
+Orchestrator brings serverless workflows into Backstage, focusing on the journey for application migration to the cloud, on boarding developers ,and user-made workflows of Backstage actions or external systems.
+Orchestrator is a flavor of RHDH, and can be installed alongside the RHDH in the same namespace and in the folloing way:
+
+1. Have an admin install the orchestrator-infra helm chart, which will install the pre-requisites required to install RHDH flavored Orchestrator. This proccess will include installing cluster-wide resources, so should be done with admin privileges
+```
+helm install <release_name> charts/orchestrator-infra
+```
+2. Manually approve the Install Plans created by the chart, and wait for the Openshift Serverless and Openshift Serverless Logic Operators to be deployed.
+3. Install backstage chart with helm, setting orchestrator to be enabled.
+4. Enable serverlessLogicOperator and serverlessOperator in the backstage values.
+
+To use orchestrator with an external DB, please follow the instructions in [our documentation](https://github.com/redhat-developer/rhdh-chart/blob/main/docs/external-db.md)
+and populate the following values in the values.yaml:
+```bash
+    externalDBsecretRef: <cred-secret>
+    externalDBName: ""
+```
+Please note that externalDBName is the name of the user-configured existing database, not the database that orchestrator and sonataflow resources will use.
+
+Finally, install the helm chart:
+```
+helm install <release_name> charts/backstage --set orchestrator.enabled=true --set orchestrator.serverlessLogicOperator.enabled=true --set orchestrator.serverlessOperator.enabled=true \
+--set externalDBsecretRef=<cred-secret> --set externalDBName=example
+```

charts/backstage/README.md.gotmpl

@@ -269,3 +269,30 @@ upstream:
     volumePermissions:
       enabled: true
 ```
+
+## Installing RHDH with Orchestrator
+
+Orchestrator brings serverless workflows into Backstage, focusing on the journey for application migration to the cloud, on boarding developers ,and user-made workflows of Backstage actions or external systems.
+Orchestrator is a flavor of RHDH, and can be installed alongside the RHDH in the same namespace and in the folloing way: 
+
+1. Have an admin install the orchestrator-infra helm chart, which will install the pre-requisites required to install RHDH flavored Orchestrator. This proccess will include installing cluster-wide resources, so should be done with admin privileges
+```
+helm install <release_name> charts/orchestrator-infra
+```
+2. Manually approve the Install Plans created by the chart, and wait for the Openshift Serverless and Openshift Serverless Logic Operators to be deployed. 
+3. Install backstage chart with helm, setting orchestrator to be enabled.
+4. Enable serverlessLogicOperator and serverlessOperator in the backstage values.
+
+To use orchestrator with an external DB, please follow the instructions in [our documentation](https://github.com/redhat-developer/rhdh-chart/blob/main/docs/external-db.md)
+and populate the following values in the values.yaml:
+```bash
+    externalDBsecretRef: <cred-secret>
+    externalDBName: ""
+```
+Please note that externalDBName is the name of the user-configured existing database, not the database that orchestrator and sonataflow resources will use.
+
+Finally, install the helm chart:
+```
+helm install <release_name> charts/backstage --set orchestrator.enabled=true --set orchestrator.serverlessLogicOperator.enabled=true --set orchestrator.serverlessOperator.enabled=true \
+--set externalDBsecretRef=<cred-secret> --set externalDBName=example
+```

charts/backstage/templates/_helpers.tpl

@@ -47,4 +47,4 @@ Referenced from: https://github.com/bitnami/charts/blob/main/bitnami/postgresql/
     {{- else -}}
         {{- printf "%s" (include "common.names.fullname" .) -}}
     {{- end -}}
-{{- end -}}
+{{- end -}}

charts/backstage/templates/_partials.tpl

@@ -0,0 +1,49 @@
+{{- define "orchestrator.plugins" }}
+{{- $config := include "orchestrator.plugins.config" . | fromYaml  }}
+plugins:
+  - disabled: false
+    package: "{{ $config.orchestratorPlugins.scope }}/{{ $config.orchestratorPlugins.orchestratorBackend.package }}"
+    integrity: "{{ $config.orchestratorPlugins.orchestratorBackend.integrity }}"
+    pluginConfig:
+      orchestrator:
+        dataIndexService:
+          url: http://sonataflow-platform-data-index-service.{{ .Release.Namespace }}
+  - disabled: false
+    package: "{{ $config.orchestratorPlugins.scope }}/{{ $config.orchestratorPlugins.orchestrator.package }}"
+    integrity: "{{ $config.orchestratorPlugins.orchestrator.integrity }}"
+    pluginConfig:
+      dynamicPlugins:
+        frontend:
+          red-hat-developer-hub.backstage-plugin-orchestrator:
+            appIcons:
+              - importName: OrchestratorIcon
+                module: OrchestratorPlugin
+                name: orchestratorIcon
+            dynamicRoutes:
+              - importName: OrchestratorPage
+                menuItem:
+                  icon: orchestratorIcon
+                  text: Orchestrator
+                module: OrchestratorPlugin
+                path: /orchestrator
+  - disabled: true
+    package: "{{ $config.orchestratorPlugins.scope }}/{{ $config.orchestratorPlugins.scaffolderBackendOrchestrator.package }}"
+    integrity: "{{ $config.orchestratorPlugins.scaffolderBackendOrchestrator.integrity }}"{{- end }}
+    pluginConfig:
+      orchestrator:
+        dataIndexService:
+          url: http://sonataflow-platform-data-index-service.{{ .Release.Namespace }}
+
+{{- define "orchestrator.plugins.config" }}
+orchestratorPlugins: 
+    scope: "@redhat"
+    orchestrator:
+      package: "backstage-plugin-orchestrator@1.5.1"
+      integrity: sha512-7VOe+XGTUzrdO/av0DNHbydOjB3Lo+XdCs6fj3JVODLP7Ypd3GXHf/nssYxG5ZYC9F1t9MNeguE2bZOB6ckqTA==
+    orchestratorBackend:
+      package: "backstage-plugin-orchestrator-backend-dynamic@1.5.1"
+      integrity: sha512-VIenFStdq9QvvmgmEMG8O7b2wqIebvEcqNeJ9SWZ8jen9t+efTK6D3Rde74LQ1no1QaHLx8RoxNCOuTUEF8O/g==
+    scaffolderBackendOrchestrator:
+      package: "backstage-plugin-scaffolder-backend-module-orchestrator-dynamic@1.5.1"
+      integrity: sha512-bnVQjVsUZ470Vgm2kd5Lo/bVa2fF0q4GufBDc/8oTQsnP3zZJQqKFvFElBTCjY76RqkECydlvZ1UFybSzvockQ==
+{{- end }}

charts/backstage/templates/dynamic-plugins-configmap.yaml

@@ -4,5 +4,33 @@ metadata:
   name: {{ printf "%s-dynamic-plugins" .Release.Name }}
 data:
   dynamic-plugins.yaml: |
-    {{- include "common.tplvalues.render" ( dict "value"
-    .Values.global.dynamic "context" $) | nindent 4 }}
+    {{- $dynamic := deepCopy .Values.global.dynamic }}
+    {{- $plugins := list }}
+
+    {{- range .Values.global.dynamic.plugins }}
+      {{- $plugins = append $plugins . }}
+    {{- end }}
+
+    {{- if .Values.orchestrator.enabled }}
+      {{- $orchestratorPlugins := include "orchestrator.plugins" . | fromYaml }}
+      {{- range $orchestratorPlugins.plugins }}
+        {{- $plugins = append $plugins . }}
+      {{- end }}
+    {{- end }}
+
+    {{- $_ := set $dynamic "plugins" $plugins }}
+
+    {{- include "common.tplvalues.render" (dict "value" $dynamic "context" $) | nindent 4 }}
+  
+---
+{{- if .Values.orchestrator.enabled }}
+apiVersion: v1
+kind: Secret
+metadata:
+  name: {{ .Release.Name}}-dynamic-plugins-npmrc
+  namespace: {{ .Release.Namespace }}
+type: Opaque
+stringData:
+  .npmrc: |
+    @redhat:registry=https://npm.registry.redhat.com
+{{- end }}

charts/backstage/templates/network-policies.yaml

@@ -0,0 +1,74 @@
+{{- if and .Values.orchestrator.enabled .Values.orchestrator.serverlessLogicOperator.enabled }}
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: {{ .Release.Name }}-allow-knative-to-sonataflow-and-workflows
+  # Sonataflow and Workflows are using the RHDH target namespace.
+  namespace: {{ .Release.Namespace | quote }}
+spec:
+  podSelector: {}
+  ingress:
+    - from:
+      - namespaceSelector:
+          matchLabels:
+            # Allow knative events to be delivered to workflows.
+            kubernetes.io/metadata.name: knative-eventing
+      - namespaceSelector:
+          matchLabels:
+            # Allow auxiliary knative function for workflow (such as m2k-save-transformation)
+            kubernetes.io/metadata.name: knative-serving
+---
+# NetworkPolicy to unblock incoming traffic to the namespace
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: {{ .Release.Name }}-allow-external-communication
+  namespace: {{ .Release.Namespace | quote }}
+spec:
+  podSelector: {}
+  ingress:
+    - from:
+      - namespaceSelector:
+          matchLabels:
+            # Allow knative events to be delivered to workflows.
+            kubernetes.io/metadata.name: openshift-ingress
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: {{ .Release.Name }}-allow-intra-network
+  namespace: {{ .Release.Namespace | quote }}
+spec:
+  # Apply this policy to all pods in the namespace
+  podSelector: {}
+  # Specify policy type as 'Ingress' to control incoming traffic rules
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+      # Allow ingress from any pod within the same namespace
+      - podSelector: {}
+
+
+{{- end }}
+---
+{{- if and .Values.orchestrator.enabled .Values.orchestrator.sonataflowPlatform.monitoring.enabled }}
+# NetworkPolicy to allow openshift-user-workload-monitoring pods to access all pods within the workflow's namespace
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: {{ .Release.Name }}-allow-monitoring-to-sonataflow-and-workflows
+  namespace: {{ .Release.Namespace | quote }}
+spec:
+  # Apply this policy to all pods in the namespace
+  podSelector: {}
+  # Specify policy type as 'Ingress' to control incoming traffic rules
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+      - namespaceSelector:
+          matchLabels:
+            # Allow openshift-user-workload-monitoring pods to access the workflow.
+            kubernetes.io/metadata.name: openshift-user-workload-monitoring
+{{- end }}