Monitoring and Audit Trail

[iaktern]

Goal: every executed process within a company space needs to be logged for verification, auditing, monitoring, export, and improvement

• this should be user-friendly, meaning BPMN and process-specific words (like "instance") should be avoided

Engine Instance Object Extension:

• inside each token, add a property engineCrashCounter. Set the value to 0 each time the token enters a new BPMN FlowNode. Then, archive the instance afterwards.
  • After a (re)start of an Engine, the Engine should check if there are open/running process instances (instance that is not in en End State). Restart these instances and for each token inside an instance, add +1 to engineCrashCounter.
  • After adding +1, check each token of each running instance if the value is >=5. If true, instead of restarting the instance (where the token belongs to), put the whole instance into the state INTERRUPTED and pause the execution of this instance.
    • Show the INTERRUPTED state in the MS and implement the possibility to continue the instance with the Play button.
  • Update wiki
• rename processInitiator, spaceIdOfProcessInitiator to processInstanceInitiator, processInstanceInitiatorSpaceId
• inside the "log"-array-objects of "variables" add a new entry "newValue" and store the new value, not just the "oldValue". (Current Problem: if a variable was changed, the new value is only stored inside "value". But this always represents the last state of the variable, so if a variable is changed multiple times within a process, it is not easy to follow the complete audit trail of value changes)
  • Update wiki
• (low prio) default values: inside the "log"-array-objects of "variables" add setting the default value of variables as the first entry. Set "changedBy": to "Process Start: Set Default Value"
  • Update wiki
  • (low prio) set the default values of variables at process start for None Start Events and Typed Start Events (like Timers)
  • (low prio) Show the default value in the start form if a new Instance is manually started
• Log of Variables at Process Start: inside the "log"-array-objects of "variables" add "changedBy": "Process Start: Set by Form for None Start Events" also for variables set inside the process start form (background: we can not use the ID of a start event because there can be multiple empty start events).
  • Update wiki
• (low prio) add the changed variables to the "log" entry for each bpmn element. same structure as under "variables"
  • Update wiki

---

Instance Documentation Page (and Process Documentation Page:

• Have the following structure: Front Page (with Title and some general infos), Table of Contents, Process Overview - children: [ Summary, Process Diagram], Execution Overview - children: [ Summary, End States of Process Variables (plus description: "The following table lists the final states of the process variables for the executed process." - Table - "To view the complete history of changes to the variables throughout the process, please refer to the detailed list of process elements below.")], Detailed Execution Log - children: [ BPMN Elements: <Type: Name> (if no Name, then use Element ID) - children: [ Diagram Element, Description (if available), Execution Log, Variable Changes (if it happened)]]
• Have all headlines and paragraph in the same consistent size if they are on the same level
• Order the Elements correctly: Start with Start Events

---

CSV Export:

• Export Data of one Instance into a CSV file. Use defined example.
• Export all Instances of a Process at once into a CSV
• Export all Instances of all executed Processes
• Add a filter button in the Execution View to "Show Past Executions" -> see next section.

---

DB: Process, Instance and Log storage within the MS DB

• Create DB schema and functionality to support the following:

  • the MS is the single source of truth: process, deployment and instance data is first stored in the DB of the MS. There is no sync of the Engine to the MS, because processes are unique and always belong to a specific Space (a Space should not see processes/deployments/instances of other Spaces). -> of course the instance data is continuously requested from the Engines and stored in the MS (later, we will probably also push the instance data to the MS).

  • Store the instance data of every process execution in the MS to have an offline view of the last synced process state. Each time, override the old instance object with the new one. (The schema needs to support: an instance belongs to a specific process version which belongs to a specific Space, the instance is running on one Engine, the process version can be deployed to multiple Engines)

  • if a Process is "deleted" by the user, do not delete it completely from the MS because we need it for creating the audit trail (= process + instance data). Just don't show the process in the Process List, Process Editor and Execution views anymore.

    • In the Execution view: add a radio button to "Show Past Executions". (Question Mark: This option displays all processes that have already been executed in the past, even if a process has already been deleted from the Management System.)
    • If there is no running instance, delete the process deployment from all Engines.
    • If there are running instances, show a hint to the User: "The process can currently not be deleted because it has running executions. Please, stop all executions if you want to delete the process."
    • If the Engines are not reachable and there are no running instances, mark the process as deleted and automatically remove the deployments if the Engines get available again.

  • the process deployment should usually be invisible to the user (exception: direct deployment in the Execution View). That means, a process is automatically deployed in the background to an Engine if the user starts a new process instance via the Process List View (or the read-only Process Editor of the Process List). The word "Deployment" should not be used in the UI because this is not user-friendly.

    • Deployment algorithm = random deployment algroithm
      • if a process (all versions) is not yet deployed to any Engine, deploy the newest version randomly to one available Engine. Store selected Engine in DB (this need to be an array, because a process version could be deployed to multiple Engines).
      • if a previous version of a process is already deployed to an Engine, try to deploy to the same Engine. If the specific Engine is not available, deploy the version randomly to one available Engine. Store selected Engine in DB array.
      • if no Engine is currently available, don't deploy. Show the user an error with the reason "No process execution could be started because there is not Process Engine available."
    • Execution View table: show all stored deployments of processes belonging to the Space even if the Engines are currently not available.
    • Execution View: manual deployment of the latest process version to a specific or random Engine should still be possible (use the defined algorithm for random). But check if automation = true before the deployment. Also, deployment deletion should still be possible with the same rules as for deletion of a process (see above).
      • (low prio) deployment delete button: show a dialog to select if the process has to be deleted from all Engines or just a specific Engine

  • Instance Start (either in Instance view or through Process List view): if the newest process version is deployed on multiple Engines, select one Engine randomly. Store the selected Engine together with the instance data in the DB.

    • (low prio) new setting: let the user select an instance distribution algorithm: random, last engine, minimal load (the last setting probably also requires an automatic deployment)

  • if a new process version is created with automation = false but the previous version had automation = true:

    • If there is no running instance, delete the process deployment from all Engines.
    • If the Engines are not reachable, mark the process as deleted and automatically remove the deployments if the Engines get available again
    • If there are running process instances of the old version, mark the process as deleted and automatically remove the deployments if all instance have ended
    • Show a hint to the user: "Attention: currently, the automation feature has been disabled, even though the previous process version supported automation. This means if you create a new version of the current process, you will no longer be able to start new executions of this process."

• (low prio) Option to "archive" a space: currently it is only possible to delete a space. This will also delete all audit data. To prevent the deletion of the audit data, there should be an archive button, which disables the space but does not delete all data.