Skip to content

Commit 11fcf99

Browse files
jmagakGitHub Actionsclaude
authored
[RHIDP-9494]: Add CloudEvents and Kafka workflow integration documentation (redhat-developer#2137)
* [RHIDP-9494]: Add CloudEvents and Kafka workflow integration documentation Add comprehensive documentation for triggering Orchestrator workflows using CloudEvents via Apache Kafka integration. New content: - Assembly: Integrate CloudEvents and Kafka with Orchestrator workflows - Concept: CloudEvents and event-driven workflow execution - Procedure: Configure Kafka connection for CloudEvent workflows - Procedure: Trigger workflows as CloudEvents from the RHDH UI - Reference: CloudEvent structure and Kafka topic mapping Target release: RHDH 1.10.0 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Fix CQA compliance issues CQA-05: Fix non-standard block title - Change ".CloudEvent attributes" to ".CloudEvent attribute descriptions" CQA-09: Fix short description length violations - con-cloudevents: Reduce abstract from 494 to 235 chars - proc-configure-kafka: Reduce abstract from 331 to 215 chars - Move additional context to separate paragraphs CQA-10: Rename files and update IDs for title compliance - assembly-integrate-cloudevents-kafka-workflows.adoc → assembly-integrate-cloudevents-and-kafka-with-orchestrator-workflows.adoc - con-cloudevents-event-driven-workflows.adoc → con-cloudevents-and-event-driven-workflow-execution.adoc - proc-configure-kafka-cloudevent-workflows.adoc → proc-configure-kafka-connection-for-cloudevent-workflows.adoc - proc-trigger-workflows-cloudevents-ui.adoc → proc-trigger-workflows-as-cloudevents-from-the-rhdh-ui.adoc - ref-cloudevent-structure-kafka-mapping.adoc → ref-cloudevent-structure-and-kafka-topic-mapping.adoc Update all cross-references and includes to use new filenames and IDs. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Add CloudEvents and Kafka workflow integration documentation * Add CloudEvents and Kafka workflow integration documentation * Add CloudEvents and Kafka workflow integration documentation * Apply technical review suggestions * Apply Tim's suggestion --------- Co-authored-by: GitHub Actions <github-actions@github.com> Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
1 parent 0adb51a commit 11fcf99

6 files changed

Lines changed: 399 additions & 0 deletions

‎assemblies/extend_orchestrator-in-rhdh/assembly-trigger-workflows-from-event-driven-systems-with-cloudevents.adoc‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
ifdef::context[:parent-context: {context}]
3+
4+
[id="trigger-workflows-from-event-driven-systems-with-cloudevents_{context}"]
5+
= Trigger workflows from event-driven systems with CloudEvents
6+
7+
:context: trigger-workflows-from-event-driven-systems-with-cloudevents
8+
9+
[role="_abstract"]
10+
Connect your Apache Kafka infrastructure to {product-short} Orchestrator to trigger workflows asynchronously from existing event-driven systems. This integration lets you maintain your messaging architecture while adding workflow orchestration capabilities.
11+
12+
include::../modules/extend_orchestrator-in-rhdh/con-event-driven-workflow-execution-for-enterprise-messaging-integration.adoc[leveloffset=+1]
13+
14+
include::../modules/extend_orchestrator-in-rhdh/proc-enable-event-driven-workflows-by-configuring-kafka-connectivity.adoc[leveloffset=+1]
15+
16+
include::../modules/extend_orchestrator-in-rhdh/proc-run-workflows-asynchronously-through-the-ui-with-cloudevents.adoc[leveloffset=+1]
17+
18+
include::../modules/extend_orchestrator-in-rhdh/ref-cloudevent-structure-reference-for-workflow-design-and-troubleshooting.adoc[leveloffset=+1]
19+
20+
ifdef::parent-context[:context: {parent-context}]
21+
ifndef::parent-context[:!context:]

‎modules/extend_orchestrator-in-rhdh/con-event-driven-workflow-execution-for-enterprise-messaging-integration.adoc‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
:_mod-docs-content-type: CONCEPT
2+
[id="event-driven-workflow-execution-for-enterprise-messaging-integration_{context}"]
3+
= Event-driven workflow execution for enterprise messaging integration
4+
5+
[role="_abstract"]
6+
Event-driven workflows enable {product-very-short} Orchestrator to respond to business events from existing messaging systems. This architecture maintains loose coupling and integrates workflows into established enterprise event streams.
7+
8+
== Why CloudEvents matter for workflow integration
9+
10+
When you integrate workflows with message-driven systems, you need a common event format that works across different platforms and services without custom adapters for each system.
11+
12+
CloudEvents is a Cloud Native Computing Foundation (CNCF) specification that standardizes how to describe event data across services, platforms, and systems. This common envelope format ensures that systems can produce and consume events without custom integration code for each platform.
13+
14+
A CloudEvent includes required metadata fields such as the event type, source, unique identifier, and specification version.
15+
Optional fields provide additional context such as content type, data schema, subject, and timestamp.
16+
The `data` field contains the event payload itself, which can include structured data relevant to the event type.
17+
18+
For example, a CloudEvent might describe a deployment request, a compliance check trigger, or a customer order placement.
19+
Because CloudEvents standardizes the event format, multiple systems can process these events by using common libraries and tools, reducing integration complexity.
20+
21+
== How your workflows respond to CloudEvents
22+
23+
When you configure a workflow for event-driven execution, {product-very-short} Orchestrator processes incoming CloudEvents sourced from Apache Kafka that match specific event types.
24+
The underlying SonataFlow engine natively supports CloudEvents, which means your workflows can consume events without additional transformation or middleware.
25+
26+
When a CloudEvent arrives:
27+
28+
. The workflow engine validates the CloudEvent structure and extracts the event metadata.
29+
. The engine matches the event type to registered workflows configured to handle that event type.
30+
. The workflow instance starts automatically, with the CloudEvent data available as workflow input.
31+
. The workflow runs its defined steps, which can include calling APIs, orchestrating services, or emitting additional CloudEvents.
32+
33+
This event-driven model differs from HTTP-triggered workflows, where you explicitly call a workflow endpoint to start the workflow.
34+
35+
== Event-driven versus HTTP-triggered workflow execution
36+
37+
Event-driven and HTTP-triggered workflows serve different integration patterns:
38+
39+
HTTP-triggered workflows::
40+
This approach is appropriate for synchronous operations where the caller needs immediate feedback or must wait for workflow completion.
41+
The caller sends an HTTP request to a specific workflow endpoint and receives a response indicating the workflow status.
42+
This pattern works well for user-initiated actions in web applications or API integrations that require request-response semantics.
43+
44+
Event-driven workflows::
45+
Use this approach when your event producers need to continue working immediately without waiting for workflow completion.
46+
Your producers publish CloudEvents to Apache Kafka, and workflows start automatically when events arrive, without blocking the producer.
47+
This pattern supports fire-and-forget semantics, enabling the producer to continue processing without blocking on the workflow.
48+
Event-driven workflows also provide better scalability for high-volume workloads and support complex event routing and filtering capabilities that the message broker provides.
49+
50+
== Benefits of event-driven workflows
51+
52+
Integrating workflows with CloudEvents and message brokers provides several architectural advantages:
53+
54+
Loose coupling::
55+
Event producers do not need direct knowledge of workflow endpoints or {product-very-short} infrastructure.
56+
They publish standardized CloudEvents to a message broker, and the workflow engine consumes events independently.
57+
This separation allows services to evolve independently without tight coupling.
58+
59+
Asynchronous execution::
60+
Event producers do not block waiting for workflows to complete.
61+
This improves system responsiveness and allows workflows to handle long-running operations without impacting the producing system.
62+
63+
Architectural consistency::
64+
Organizations that have standardized on message-oriented middleware can integrate {product-very-short} workflows into existing event-driven architectures without creating HTTP-based exceptions.
65+
This maintains architectural consistency across the enterprise.
66+
67+
Enterprise messaging standards::
68+
CloudEvents specification provides a vendor-neutral event format that major cloud providers and messaging platforms support.
69+
This ensures portability and reduces vendor lock-in.
70+
71+
== CloudEvent type to Kafka topic mapping
72+
73+
When you configure workflows to consume CloudEvents from Kafka, you decide how CloudEvent types map to Kafka topics based on your workflow design and organizational standards.
74+
75+
A common pattern maps each workflow to a specific Kafka topic, where the topic name corresponds to the CloudEvent type that triggers the workflow.
76+
For example, a workflow that processes deployment requests might subscribe to a `deployment.request` topic, and producers would publish CloudEvents with `type: deployment.request` to that topic.
77+
78+
You can organize workflows by business capability and route events to the appropriate workflow by using Kafka topic-based routing.
79+
You can also use Kafka consumer groups to scale workflow processing across multiple {product-very-short} instances.
80+
81+
== When to use CloudEvent triggering
82+
83+
Choose CloudEvent-based workflow triggering when:
84+
85+
* Your organization has standardized on message brokers such as Apache Kafka for system-to-system communication.
86+
* Your workflows respond to business events produced by other systems in your architecture.
87+
* You need asynchronous, fire-and-forget workflow execution.
88+
* You want to decouple workflow callers from {product-very-short} infrastructure.
89+
* Your workflows integrate into existing event-driven architectural patterns.
90+
91+
Use HTTP-triggered workflows when:
92+
93+
* Your users need immediate workflow feedback or synchronous responses.
94+
* Your workflows serve as APIs for external systems that expect request-response patterns.
95+
* Your organization has not deployed message broker infrastructure.
96+
* Your workflow execution is strictly on-demand rather than event-driven.

‎modules/extend_orchestrator-in-rhdh/proc-enable-event-driven-workflows-by-configuring-kafka-connectivity.adoc‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
:_mod-docs-content-type: PROCEDURE
2+
[id="enable-event-driven-workflows-by-configuring-kafka-connectivity_{context}"]
3+
= Enable event-driven workflows by configuring Kafka connectivity
4+
5+
[role="_abstract"]
6+
Configure Apache Kafka connectivity in the Orchestrator backend to enable workflows triggered by CloudEvents. This configuration allows workflows to respond asynchronously to business events from your messaging infrastructure.
7+
8+
.Prerequisites
9+
10+
* You have enabled Orchestrator plugins.
11+
* You have deployed Apache Kafka broker infrastructure and ensured it is accessible from {product-very-short}.
12+
* You have Kafka broker URLs and connection credentials.
13+
* You have verified network connectivity between {product-very-short} and Kafka brokers.
14+
15+
.Procedure
16+
17+
. Locate your {product-short} application configuration file.
18+
+
19+
The location depends on your deployment method:
20+
+
21+
--
22+
* For Operator deployments: The configuration is in a ConfigMap, typically named `{my-app-config-config-map}`.
23+
* For Helm deployments: The configuration is in the `values.yaml` file or a custom configuration file referenced in your Helm values.
24+
--
25+
26+
. Add the `orchestrator.kafka` configuration section to your `{my-app-config-file}` file.
27+
+
28+
[source,yaml,subs="+attributes"]
29+
----
30+
orchestrator:
31+
kafka:
32+
clientId: my-rhdh-orchestrator
33+
brokers:
34+
- kafka-broker-1.example.com:9092
35+
- kafka-broker-2.example.com:9092
36+
- kafka-broker-3.example.com:9092
37+
# logLevel override for the orchestrator kafka services. Defaults to INFO which is 4
38+
# logLevel values based on KafkaJS values https://kafka.js.org/docs/configuration#logging
39+
# logLevel: 5 // DEBUG
40+
logLevel: 4
41+
----
42+
+
43+
where:
44+
+
45+
`clientId`:: Unique identifier for the {product-very-short} Kafka client. This identifier is displayed in Kafka broker logs and metrics.
46+
`brokers`:: Array of Kafka broker URLs. Include multiple brokers for high availability.
47+
`logLevel`:: Optional. Kafka client logging level. Valid numeric values based on link:https://kafka.js.org/docs/configuration#logging[KafkaJS values] are `0` (NOTHING), `1` (ERROR), `2` (WARN), `4` (INFO), or `5` (DEBUG). Default is `4` (INFO).
48+
49+
. Apply the configuration changes.
50+
+
51+
--
52+
* For Operator deployments: Update the ConfigMap and restart the {product-very-short} instance by scaling the deployment to zero and back to the target replica count, or by deleting the pod to trigger a restart.
53+
+
54+
Replace `<my_deployment_name>` with the name of your deployment:
55+
+
56+
[source,terminal,subs="+attributes"]
57+
----
58+
$ oc rollout restart deployment/<my_deployment_name>
59+
----
60+
61+
* For Helm deployments: Upgrade the Helm release with the updated configuration.
62+
+
63+
[source,terminal,subs="+attributes"]
64+
----
65+
$ helm upgrade {my-product-cr-name} redhat-developer/backstage -f values.yaml -n {my-product-namespace}
66+
----
67+
--
68+
69+
.Verification
70+
71+
. Check the orchestrator-backend plugin logs for Kafka connection messages.
72+
+
73+
Replace `<my_deployment_name>` with the name of your deployment:
74+
+
75+
[source,terminal,subs="+attributes"]
76+
----
77+
$ oc logs deployment/<my_deployment_name> | grep -i kafka
78+
----
79+
+
80+
Successful connection logs include messages indicating the Kafka client has connected to the broker cluster.
81+
82+
. Navigate to the Orchestrator plugin in the {product-very-short} UI.
83+
84+
. Verify that the *Run as Event* button is displayed next to workflows.
85+
+
86+
The button is only visible when Kafka connectivity is successfully configured.
87+
88+
.Troubleshooting
89+
90+
If the *Run as Event* button does not appear:
91+
92+
* Verify that the Kafka broker URLs are correct and accessible from the {product-very-short} pod.
93+
* Check the orchestrator-backend logs for connection errors or authentication failures.
94+
* Confirm that network policies allow traffic between {product-very-short} and the Kafka brokers.
95+
* Verify that the `orchestrator.kafka` configuration section is correctly formatted in the configuration file.

‎modules/extend_orchestrator-in-rhdh/proc-run-workflows-asynchronously-through-the-ui-with-cloudevents.adoc‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
:_mod-docs-content-type: PROCEDURE
2+
[id="run-workflows-asynchronously-through-the-ui-with-cloudevents_{context}"]
3+
= Run workflows asynchronously through the UI with CloudEvents
4+
5+
[role="_abstract"]
6+
Publish CloudEvents to Apache Kafka from the {product-very-short} UI to trigger workflows asynchronously. This method enables fire-and-forget operation without blocking on workflow completion.
7+
8+
.Prerequisites
9+
10+
* You have configured Kafka connectivity for the Orchestrator.
11+
* You have deployed an event-type workflow that appears in the Orchestrator plugin.
12+
13+
.Procedure
14+
15+
. In the {product-very-short} UI, navigate to the Orchestrator plugin.
16+
17+
. In the workflows list, locate the workflow you want to trigger.
18+
19+
. Click the *Run as Event* button next to the workflow.
20+
+
21+
[NOTE]
22+
====
23+
The *Run as Event* button appears only when you have configured Kafka connectivity and the workflow supports event-driven execution.
24+
====
25+
26+
. If the workflow requires input data, complete the workflow input form.
27+
+
28+
The form fields correspond to the workflow's input schema.
29+
The CloudEvent data payload includes the values you provide.
30+
31+
. Click *Submit* to send the CloudEvent to Kafka.
32+
+
33+
The {product-very-short} UI transmits a CloudEvent to the configured Kafka broker with the workflow input data.
34+
The workflow starts when the Kafka broker delivers the event to the SonataFlow engine.
35+
36+
. Monitor the workflow status.
37+
+
38+
After submitting the CloudEvent, one of two outcomes occurs:
39+
+
40+
--
41+
Immediate start:: If the workflow starts before the UI timeout period, the UI navigates to the workflow instance detail page, where you can monitor progress.
42+
43+
Delayed start:: If the workflow has not started when the UI timeout expires, the UI displays an informational message indicating that it sent the event to Kafka as a `kafkaEvent`.
44+
The UI navigates to the workflow runs list, where the workflow instance appears when the workflow starts.
45+
--
46+
47+
. If the workflow does not start immediately, locate the workflow in the workflow runs list.
48+
+
49+
The workflow instance appears in the list when the Kafka broker delivers the CloudEvent and the workflow engine starts the workflow.
50+
Depending on Kafka broker latency and workflow engine processing time, this can take several seconds.
51+
52+
.Troubleshooting
53+
54+
If the workflow does not appear in the workflow runs list after several minutes:
55+
56+
* Verify that the Kafka broker is running and accessible.
57+
* Check the orchestrator-backend logs for errors related to Kafka message publishing.
58+
* Confirm that you configured the workflow to consume CloudEvents from the correct Kafka topic.
59+
* Verify that the CloudEvent type matches the workflow's event type configuration.

0 commit comments

Comments
 (0)