|
| 1 | +# Logging (user facing) |
| 2 | + |
| 3 | +This should entirely focus on the user side. How logigng works, and then how to use the stuff in daqptyools |
| 4 | + |
| 5 | +--- |
| 6 | +Updates as of mid February |
| 7 | + |
| 8 | +# Logging for Python in DUNE-DAQ |
| 9 | + |
| 10 | +Welcome, fellow logging enthusiast! This page provides a user's guide to how logging is done in Python in the context of DUNE-DAQ. |
| 11 | + |
| 12 | +## Basics |
| 13 | + |
| 14 | +The bulk of the loggign functionality in drunc and other Python applications is built off the cool [Python logging framework](https://docs.python.org/3/library/logging.html), with its mission defined below: |
| 15 | + |
| 16 | +> This module defines functions and classes which implement a flexible event logging system for applications and libraries. |
| 17 | +
|
| 18 | +It is worth a read to understand how logging works in Python, however the salient points are covered below. |
| 19 | + |
| 20 | +In general, the built in logging module allows for producing severity-classified diagnostic events, which can be filtered, formatted, or routed as necessary. These logs automatically contain useful information including the timestamp, module, and context of the message. |
| 21 | + |
| 22 | +The core object of the logging functionality in Python is the logger. A logging instance, `log`, can be initialised as follows. The phrase "Hello, World!" is used as an input to the logger, with this message being bundled up by other useful information, including the severity level, to form whats known as a LogRecord. This record will then be transmitted as required. |
| 23 | + |
| 24 | +```python |
| 25 | +import logging |
| 26 | +log = logging.getLogger("Demo") |
| 27 | +log.warning("Hello, world!") |
| 28 | + |
| 29 | +>> Hello, World! |
| 30 | +``` |
| 31 | + |
| 32 | +### Severity levels |
| 33 | + |
| 34 | +Every record has an attached severity level, which can be used to flag how important a log record is. By default, Python has 5 main levels and one 'notset' level as shown in the image below[^1]: |
| 35 | + |
| 36 | + |
| 37 | + |
| 38 | +[^1]: more can be defined as required, see Python's logging manual. |
| 39 | + |
| 40 | +Each logging instance can have an attached severity level. If it has one, then only records that have the same severity level or higher will be transmitted. |
| 41 | + |
| 42 | +```python |
| 43 | +import logging |
| 44 | +log = logging.getLogger("Demo", level = logging.WARNING) |
| 45 | + |
| 46 | +log.info("This will not print") |
| 47 | +log.warning("This will print") |
| 48 | + |
| 49 | +>> This will print |
| 50 | +``` |
| 51 | + |
| 52 | +### Handlers |
| 53 | + |
| 54 | +Handlers are a key concept of logging in Python, as they control how the records are processed and formatted. There are several default one that the DAQ uses, and there are also several ones that are custom defined for the purposes of the DAQ. |
| 55 | + |
| 56 | +The example below shows an example of a file handler, a stream handler, and a webhook handler. As can be seen, each of the records are processed and formatted by each of the handlers and transmitted in each of their respective ways. |
| 57 | + |
| 58 | + |
| 59 | + |
| 60 | +Importantly, each handler can have its own associated severity level! In the example above, it is certainly possible to have the WebHookHandler to only transmit if a record is of the level Warning or higher. |
| 61 | + |
| 62 | + |
| 63 | + |
| 64 | +### Filters |
| 65 | +A sidegrade and important add on for the loggers is the filters, whos primary purpose is to decide if an error should be transmitted or not. Filters can be attached to both the logger instance as well as any handlers attached to the logger instance itself. |
| 66 | + |
| 67 | +When a log record arrives, it will first be processed by the filters attached to the loggers first. Should they pass, the record is then passed onto each handler as shown before, where they are then further processed by each handler's attached filters. Only when they pass will a log be record be transmitted. |
| 68 | + |
| 69 | + |
| 70 | + |
| 71 | +### Inheritance |
| 72 | + |
| 73 | +Another key part of logging in Python is the inheritance feature. Loggers are organised in a heirarchical fashion and so it is possible to initialise descendant loggers by chaining the names together with a period, such as "root.parent.child". |
| 74 | + |
| 75 | +By default, loggers will inheret certain properties of the parent: |
| 76 | +- severity level of the logger |
| 77 | +- handlers (and all attached properties, including severity level and filters on handlers) |
| 78 | + |
| 79 | + |
| 80 | + |
| 81 | + |
| 82 | + |
| 83 | +Note that a particular exceptoin is that they _don't_ inheret any filters attached to the logger itself. |
| 84 | + |
| 85 | +A useful diagram to peruse is the [logging flow in the official docs](https://docs.python.org/2/howto/logging.html#logging-flow). |
| 86 | + |
| 87 | + |
| 88 | +# Daqpytools |
| 89 | + |
| 90 | +## Initialising a handler |
| 91 | + |
| 92 | +## Subsec of existing handlers, filters, and walkthrough |
| 93 | + |
| 94 | + |
| 95 | + |
| 96 | +### Advanced logging |
| 97 | + |
| 98 | + |
| 99 | +#### Context |
| 100 | + |
| 101 | +#### Handler streams |
| 102 | + |
| 103 | + |
| 104 | +#### Log Handler Conf |
| 105 | + |
| 106 | + |
| 107 | +#### Using ERS |
| 108 | + |
| 109 | + |
| 110 | + |
| 111 | + |
| 112 | + |
| 113 | + |
| 114 | + |
| 115 | + |
| 116 | +## Basic info |
| 117 | +basically see your presentation |
| 118 | + |
| 119 | +- the basics of python logging |
| 120 | +- handlers |
| 121 | +- inheritance |
| 122 | +- log levels |
| 123 | + |
| 124 | +## User story |
| 125 | +And then talk about the features in daqpytools as a user |
| 126 | + |
| 127 | +- Best practices on initialising loggers |
| 128 | +- How to deal with handlertypes |
| 129 | +- How to deal with handlerconf |
| 130 | +- integratoin with ers |
| 131 | + |
| 132 | +## Advanced handlers (WIP as of Dec 2025) |
| 133 | + |
| 134 | +Aside from the core set of loggers and handlers defined natively in drunc, there are several other configurations that interplay with drunc. These can be best thought of as 'streams' which in interact with several other handlers. |
| 135 | + |
| 136 | +<img width="510" height="561" alt="streams" src="https://github.com/user-attachments/assets/de95f47c-fe30-48f3-ac96-0f7c091bbc17" /> |
| 137 | + |
| 138 | +The native implementation in drunc is referred to as the 'base' stream and interacts with the three core handlers already discussed. |
| 139 | +Other streams include the 'Opmon' stream interacting with its own set of handlers, and the 'ERS' stream that interacts with different handlers based on the log message's severity level. |
| 140 | + |
| 141 | +The 'ERS' configuration is configured in OKS ([for example here](https://github.com/DUNE-DAQ/daqsystemtest/blob/974965be6e96aff969c69a380ed34aa96705e802/config/daqsystemtest/ccm.data.xml#L189)), and are automatically parsed by drunc and daqpytools as they get used. |
| 142 | + |
| 143 | +To deal with the constraints set above, a handler configuration dataclass is constructed to define the relevant configurations and a system of filters is initialised with every handler. When passing a log record that needs to be processed via a specific stream, the log record and the handler configuration is passed to the relevant logger, after which it is processed. |
| 144 | + |
| 145 | + |
| 146 | +## Dev story |
| 147 | + |
| 148 | + |
| 149 | + |
0 commit comments