Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions docs/concepts/event-bus.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
.. include:: ../common_refs.rst

.. _Event Bus:

Open edX Event Bus
####################

Expand Down Expand Up @@ -29,7 +31,7 @@ The :term:`Event Bus` can help us achieve loose coupling between services, repla
How Does the Open edX Event Bus Work?
***************************************

The Open edX platform uses the ``OpenEdxPublicSignals`` (Open edX-specific Django Signals) to send events within a service. The event bus extends these signals, allowing them to be broadcasted and handled across multiple services. That's how Open edX Events are used for internal and external communication. For more details on how these Open edX-specific Django Signals are used by the event bus, refer to the :doc:`../decisions/0004-external-event-bus-and-django-signal-events` Architectural Decision Record (ADR).
The Open edX platform uses the ``OpenEdxPublicSignals`` (Open edX-specific Django Signals) to send events within a service. The event bus extends these signals, allowing them to be broadcasted and handled across multiple services. That's how Open edX Events are used for internal and external communication. For more details on how these Open edX-specific Django Signals are used by the event bus, refer to the :ref:`ADR-4` Architectural Decision Record (ADR).

Open edX Events provides an abstract implementation of the `publish/subscribe messaging pattern`_ (pub/sub), which is the chosen pattern for the event bus implementation, as explained in :doc:`openedx-proposals:architectural-decisions/oep-0052-arch-event-bus-architecture`. It implements two abstract classes, `EventProducer`_ and `EventConsumer`_, which allow concrete implementations of the event bus based on different message brokers, such as Pulsar.

Expand Down Expand Up @@ -78,7 +80,7 @@ From Service B (Consumer)
2. When a new message is found, the ``EventConsumer`` deserializes the message and re-emits it as an event with the data that was transmitted.
3. The event sending and processing workflow repeats in Service B.

This approach of producing events via settings with the generic handler was chosen to allow for flexibility in the event bus implementation. It allows developers to choose the event bus implementation that best fits their needs, and easily switch between implementations if necessary. See more details in the :doc:`../decisions/0012-producing-to-event-bus-via-settings` Architectural Decision Record (ADR).
This approach of producing events via settings with the generic handler was chosen to allow for flexibility in the event bus implementation. It allows developers to choose the event bus implementation that best fits their needs, and easily switch between implementations if necessary. See more details in the :ref:`ADR-12` Architectural Decision Record (ADR).

Event Bus vs Asynchronous Tasks
********************************
Expand Down Expand Up @@ -123,7 +125,7 @@ How is the Open edX Event Bus Used?

The event bus is used to broadcast Open edX Events to multiple services, allowing them to react to changes or actions in the system.

We encourage you to review the :doc:`../reference/real-life-use-cases` page for examples of how the community uses the event bus in the Open edX ecosystem. Also, see the :doc:`../how-tos/use-the-event-bus-to-broadcast-and-consume-events` guide to start sending events to the event bus.
We encourage you to review the :ref:`Real-Life Use Cases` page for examples of how the community uses the event bus in the Open edX ecosystem. Also, see the :ref:`Use the Open edX Event Bus to Broadcast and Consume Events` guide to start sending events to the event bus.

.. _general_signal_handler: https://github.com/openedx/openedx-events/blob/main/openedx_events/apps.py#L16-L44
.. _EventProducer: https://github.com/openedx/openedx-events/blob/main/openedx_events/event_bus/__init__.py#L71-L91
Expand Down
2 changes: 2 additions & 0 deletions docs/concepts/index.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _Concepts:

Concepts
========

Expand Down
8 changes: 4 additions & 4 deletions docs/concepts/openedx-events.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,12 @@ These events are built on top of Django signals, inheriting their behavior while

.. note:: Django includes a "signal dispatcher", which helps decoupled applications get notified when actions occur elsewhere in the framework. In a nutshell, signals allow certain senders to notify a set of receivers that some action has taken place. They're especially useful when many pieces of code may be interested in the same events.

Events are primarily used as a communication method between internal services by leveraging Django Signals and external services using the :doc:`../concepts/event-bus`, making them the standard communication mechanism within the Open edX ecosystem.
Events are primarily used as a communication method between internal services by leveraging Django Signals and external services using the :ref:`Event Bus`, making them the standard communication mechanism within the Open edX ecosystem.

How Do Open edX Events Work?
****************************

Open edX Events are implemented by a class called `OpenEdxPublicSignal`_, which inherits from `Django's Signals class` and adds behaviors specific to the Open edX ecosystem. Thanks to this design, ``OpenEdxPublicSignal`` leverages the functionality of Django signals, allowing developers to apply their existing knowledge of the Django framework. You can review the :doc:`Open edX Events Tooling <../reference/events-tooling>` documentation for more information on the tooling available for working with Open edX events.
Open edX Events are implemented by a class called `OpenEdxPublicSignal`_, which inherits from `Django's Signals class` and adds behaviors specific to the Open edX ecosystem. Thanks to this design, ``OpenEdxPublicSignal`` leverages the functionality of Django signals, allowing developers to apply their existing knowledge of the Django framework. You can review the :ref:`Events Tooling` documentation for more information on the tooling available for working with Open edX events.

.. _events architecture:

Expand Down Expand Up @@ -79,9 +79,9 @@ The `Django Signals Documentation`_ provides a more detailed explanation of how
How Are Open edX Events Used?
*****************************

As mentioned previously, developers can listen to Open edX Events by registering signal receivers from their Open edX Django plugins that respond to the emitted events or by using the :doc:`../concepts/event-bus` to send events to external services.
As mentioned previously, developers can listen to Open edX Events by registering signal receivers from their Open edX Django plugins that respond to the emitted events or by using the :ref:`Event Bus` to send events to external services.

For more information on using Open edX Events, refer to the :doc:`../how-tos/create-a-new-event` how-to guide. We also encourage you to explore the :doc:`../reference/real-life-use-cases` section for real-life examples of how Open edX Events are used by the community.
For more information on using Open edX Events, refer to the :ref:`Create a New Open edX Event with Long-Term Support` how-to guide. We also encourage you to explore the :ref:`Real-Life Use Cases` section for real-life examples of how Open edX Events are used by the community.

.. _triggering the COURSE_ENROLLMENT_CREATED event: https://github.com/openedx/edx-platform/blob/master/common/djangoapps/student/models/course_enrollment.py#L777-L795
.. _course_enrollment_post_save receiver: https://github.com/openedx/edx-platform/blob/master/openedx/core/djangoapps/notifications/handlers.py#L38-L53
Expand Down
2 changes: 2 additions & 0 deletions docs/decisions/0002-events-naming-and-versioning.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _ADR-2:

2. Open edX events naming and versioning
========================================

Expand Down
2 changes: 2 additions & 0 deletions docs/decisions/0003-events-payload.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _ADR-3:

3. Open edX events payload conventions
======================================

Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _ADR-4:

4. External event bus and Django Signal events
==============================================

Expand All @@ -17,7 +19,7 @@ Decision
--------
Note that for purposes of this ADR, ``event`` refers to a dictionary of data that is emitted by a Django signal, specifically by an instance of ``OpenEdxPublicSignal``

- Event definitions will be shared between internal and external events. The event definitions will continue to be in the form of ``OpenEdxPublicSignal``, as decided in ADR ":doc:`0003-events-payload`".
- Event definitions will be shared between internal and external events. The event definitions will continue to be in the form of ``OpenEdxPublicSignal``, as decided in ADR ":ref:`ADR-3`".

- At this time, events will be published to the event bus from a Django signal handler. That is, when an internal event is sent via a Django signal, one of the handlers will take the event and publish it to the event bus.

Expand All @@ -32,7 +34,7 @@ Consequences

- An external event will never be sent without a corresponding internal event (at this time, based on the current design).

- The external event bus handler will listen for Django signals (``OpenEdxPublicSignals``) that have been configured to be sent as external events. These external events will be serialized using utilities in this library, as decided in ADR ":doc:`0005-external-event-schema-format`", before sending the messages over the wire.
- The external event bus handler will listen for Django signals (``OpenEdxPublicSignals``) that have been configured to be sent as external events. These external events will be serialized using utilities in this library, as decided in ADR ":ref:`ADR-5`", before sending the messages over the wire.

- The use of the ``OpenEdxPublicSignal`` on both the event producing and event consuming sides for external events should hopefully provide a consistent mechanism to plug in for events.

Expand Down
4 changes: 3 additions & 1 deletion docs/decisions/0005-external-event-schema-format.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _ADR-5:

5. External Event Schema Format
===============================

Expand All @@ -12,7 +14,7 @@ Context

* It is a best practice to use an explicit schema definition. Avro is the recommended serialization format for Kafka.

* The attrs objects that we currently have for signal-based events don't easily inter-operate with event bus client objects. Source ADR ":doc:`0003-events-payload`".
* The attrs objects that we currently have for signal-based events don't easily inter-operate with event bus client objects. Source ADR ":ref:`ADR-3`".

* Industry best practices seem to suggest using a binary encoding of messages.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _ADR-6:

6. Event schema serialization and evolution
===========================================

Expand All @@ -17,9 +19,9 @@ Currently, the specification for events are written as OpenEdxPublicSignal insta

Over time, as needs change, applications will need the schema for an event to change.

For internal events, there is some discussion of versioning events related to how they might change in ADR ":doc:`0002-events-naming-and-versioning`". For external events to be sent using an event bus, the same OpenEdxPublicSignal definitions will provide the schema.
For internal events, there is some discussion of versioning events related to how they might change in ADR ":ref:`ADR-2`". For external events to be sent using an event bus, the same OpenEdxPublicSignal definitions will provide the schema.

However, for the event bus, we will be introducing more rigorous event evolution rules using an explicit schema and schema registry. We had decided to use Avro in ":doc:`0005-external-event-schema-format`".
However, for the event bus, we will be introducing more rigorous event evolution rules using an explicit schema and schema registry. We had decided to use Avro in ":ref:`ADR-5`".

Event schemas can evolve in various ways and each implementation of an event bus should choose a schema evolution configuration. The choices include how data can change, as well as the order in which producer and consumer needs to be deployed with new changes.

Expand Down
2 changes: 1 addition & 1 deletion docs/decisions/0007-optional-fields.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,5 @@ Consequences
Deferred/Rejected Decisions
---------------------------

- Top-level fields will continue to be required, as stated in ":doc:`0006-event-schema-serialization-and-evolution`". We continue to defer the possibility of making these optional.
- Top-level fields will continue to be required, as stated in ":ref:`ADR-6`". We continue to defer the possibility of making these optional.
- We are also deferring any work around making an explicit ``Optional`` keyword for attrs fields
2 changes: 2 additions & 0 deletions docs/decisions/0010-multiple-event-types-per-topic.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _ADR-10:

10. Multiple event types per topic
##################################

Expand Down
2 changes: 2 additions & 0 deletions docs/decisions/0012-producing-to-event-bus-via-settings.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _ADR-12:

12. Enable producing to event bus via settings
##############################################

Expand Down
4 changes: 3 additions & 1 deletion docs/decisions/0016-event-design-practices.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _ADR-16:

16. Event Design Best Practices
###############################

Expand Down Expand Up @@ -57,7 +59,7 @@ Responsibility and Granularity

- Design events with a single responsibility in mind. Each event should represent a single action or fact that happened in the system. If an event contains multiple actions, consider splitting it into multiple events. For instance, if the course grade is updated to pass or fail, there should be two events: one for the pass action and another for the fail action.

.. note:: For the :doc:`Event Bus <../concepts/event-bus>`, events that are split across multiple actions are an exceptional case where the same event :term:`Topic` should be used to help maintain order across these events.
.. note:: For the :ref:`Event Bus <Event Bus>`, events that are split across multiple actions are an exceptional case where the same event :term:`Topic` should be used to help maintain order across these events.

- Manage the granularity of the event so it is not too coarse (generic with too much information) or too fine-grained (specific with too little information). When making a decision on the granularity of the event, start with the minimum required information for consumers to react to the event and add more information as needed with enough justification. If necessary, leverage API calls from the consumer side to retrieve additional information but always consider the trade-offs of adding dependencies with other services.
- Ensure that the triggering logic is consistent and narrow. For instance, if an event is triggered when a user enrolls in a course, it should be triggered when the user enrolls in a course in all ways possible to enroll in a course. If the event is triggered when a user enrolls in a course through the API, it should also be triggered when the user enrolls in a course through the UI.
Expand Down
2 changes: 2 additions & 0 deletions docs/decisions/index.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _ADRs:

Architectural Decision Records (ADRs)
#####################################

Expand Down
12 changes: 7 additions & 5 deletions docs/how-tos/add-event-bus-support-to-an-event.rst
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
.. include:: ../common_refs.rst

.. _Add Event Bus Support:

Add Event Bus Support to an Open edX Event
############################################

Before sending an event across services, you need to ensure that the event is compatible with the Open edX Event Bus. This involves ensuring that the event, with its corresponding payload, can be emitted by a service through the event bus and that other services can consume it. This guide will walk you through the process of adding event bus support to an Open edX event.

For more details on how the :term:`Event Payload` is structured, refer to the :doc:`../decisions/0003-events-payload` decision record.
For more details on how the :term:`Event Payload` is structured, refer to the :ref:`ADR-3` decision record.

.. note::
This guide assumes that you have already created an Open edX event. If you have not, refer to the :doc:`../how-tos/create-a-new-event` how-to guide.
This guide assumes that you have already created an Open edX event. If you have not, refer to the :ref:`Create a New Open edX Event with Long-Term Support` how-to guide.

Step 1: Does my Event Need Event Bus Support?
===============================================
Expand All @@ -32,7 +34,7 @@ Step 2: Define the Event Payload

An Open edX Event is compatible with the event bus when its payload can be serialized, sent, and deserialized by other services. The payload, structured as `attrs data classes`_, must align with the event bus schema format, which in this case is the :term:`Avro Schema`. This schema is used to serialize and deserialize the :term:`Event Payload` when sending it across services.

This ensures the event can be sent by the producer and then re-emitted by the same instance of |OpenEdxPublicSignal| on the consumer side, guaranteeing that the data sent and received is identical. Serializing this way should prevent data inconsistencies between services, e.g., timezone issues and precision loss. For more information on the event bus schema format, refer to the :doc:`../decisions/0004-external-event-bus-and-django-signal-events` and :doc:`../decisions/0005-external-event-schema-format` decision records.
This ensures the event can be sent by the producer and then re-emitted by the same instance of |OpenEdxPublicSignal| on the consumer side, guaranteeing that the data sent and received is identical. Serializing this way should prevent data inconsistencies between services, e.g., timezone issues and precision loss. For more information on the event bus schema format, refer to the :ref:`ADR-4` and :ref:`ADR-5` decision records.

The data types used in the attrs classes that the current Open edX Event Bus with the chosen schema are:

Expand All @@ -59,7 +61,7 @@ In the ``data.py`` files within each architectural subdomain, you can find examp
Step 3: Ensure Serialization and Deserialization
==================================================

Before sending the event across services, you need to ensure that the :term:`Event Payload` can be serialized and deserialized correctly. The event bus concrete implementations use the :term:`Avro Schema` to serialize and deserialize the :term:`Event Payload` as mentioned in the :doc:`../decisions/0005-external-event-schema-format` decision record. The concrete implementation of the event bus handles serialization and deserialization with the help of methods implemented by this library.
Before sending the event across services, you need to ensure that the :term:`Event Payload` can be serialized and deserialized correctly. The event bus concrete implementations use the :term:`Avro Schema` to serialize and deserialize the :term:`Event Payload` as mentioned in the :ref:`ADR-5` decision record. The concrete implementation of the event bus handles serialization and deserialization with the help of methods implemented by this library.

If you are interested in how the serialization and deserialization of the :term:`Event Payload` are handled by the event bus, you can refer to the concrete event bus implementation in the Open edX Event Bus repository. For example, here's how the Redis event bus handles `serialization`_ and `deserialization`_ when sending and receiving events.

Expand Down Expand Up @@ -126,7 +128,7 @@ Run ``python manage.py generate_avro_schemas --help`` to see the available optio
Step 5: Send the Event Across Services with the Event Bus
==========================================================

To validate that you can consume the event emitted by a service through the event bus, you can send the event across services. Here is an example of how you can send the event across services using the Redis event bus implementation following the `setup instructions in a Tutor environment`_. We recommend also following :doc:`../how-tos/use-the-event-bus-to-broadcast-and-consume-events` to understand how to use the event bus in your environment.
To validate that you can consume the event emitted by a service through the event bus, you can send the event across services. Here is an example of how you can send the event across services using the Redis event bus implementation following the `setup instructions in a Tutor environment`_. We recommend also following :ref:`Use the Open edX Event Bus to Broadcast and Consume Events` to understand how to use the event bus in your environment.

.. note:: If you implemented a custom serializer for a type in the :term:`Event Payload`, the custom serializer support must be included in both the producer and consumer sides before it can be used.

Expand Down
2 changes: 2 additions & 0 deletions docs/how-tos/add-new-event-bus-concrete-implementation.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _Add New Implementation of the Event Bus:

Add a New Concrete Implementation of the Event Bus
###################################################

Expand Down
Loading