Use observer pattern for sending alerts

[Poofjunior]

Currently, we add alert (email) functionality to the brainslosher by polling for state changes in RouterServer and then sending an email from RouterServer. This adds business logic to the RouterServer, making it more than just a message-dispatcher to one-or-more RouterClients. Here are two alternate ways of adding alert functionality that respect our "instrument/infrastructure-agnostic" architecture diagram.

Option 1: Observer Pattern

This is the most straightforward option.
In the instrument base class we add two functions: add_alert_handler and send_alert.

The add_alert_handler(fn) adds a function to a list of callback functions to be called when we call send_alert. The send_alert function just iterates through the list of callback functions and calls them with any input data that comes with the alert (maybe a string?). Something like this:

# in instrument base class

def add_alert_handler(func: callable):
  self.alert_handlers.append(func)

def send_alert(message: str):
    for alert_func in self.alert_handlers:
        alert_func(message)  # call the function

In the Brainslosher business logic, we just call send_alert and trust that the attached callback function knows how to do this.

try:
    # do brainslosher steps
except Exception as e:
    self.send_alert("Brainsloshing failed to complete job!")

In the above case, there's no need to incorporate RouterServer at all, we just attach a callback function, and the Brainslosher instrument calls it.

(More deets on this pattern here: https://refactoring.guru/design-patterns/observer)

Option 2: Observer Pattern + one-liner to external system

This is more of a "microservices" approach. In the case where Option 1 is not sufficient or you truly want decoupled behavior to silo failures, I would suggest this option.

Here, I would do the same as above: create add_alert_handler and send_alert functions. Then I would generate a custom stream function in one-liner using get_stream_fn. (Example of how to generate this callback function. )

Finally, I would create a one-liner RouterClient to listen to received alerts from this stream and invoke the attached alert system. The benefit to this approach is that the separate alert system can be a monolithic, cumbersome, or slow process to deal with, but it wont slow down the instrument operation. The instrument just needs to send the alert, and the one-liner Server/Client connection sends it to this system separately. Finally, it can also be a separate process (or PC!) so it can fail separately.

[patricklatimer]

https://github.com/AllenNeuralDynamics/SIPE-Admin/issues/159