Skip to content

Commit ffdbf04

Browse files
committed
Wrote basics of logging docs
1 parent 0e66e9b commit ffdbf04

5 files changed

Lines changed: 149 additions & 0 deletions

File tree

docs/Logging.md

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
# Logging (user facing)
2+
3+
This should entirely focus on the user side. How logigng works, and then how to use the stuff in daqptyools
4+
5+
---
6+
Updates as of mid February
7+
8+
# Logging for Python in DUNE-DAQ
9+
10+
Welcome, fellow logging enthusiast! This page provides a user's guide to how logging is done in Python in the context of DUNE-DAQ.
11+
12+
## Basics
13+
14+
The bulk of the loggign functionality in drunc and other Python applications is built off the cool [Python logging framework](https://docs.python.org/3/library/logging.html), with its mission defined below:
15+
16+
> This module defines functions and classes which implement a flexible event logging system for applications and libraries.
17+
18+
It is worth a read to understand how logging works in Python, however the salient points are covered below.
19+
20+
In general, the built in logging module allows for producing severity-classified diagnostic events, which can be filtered, formatted, or routed as necessary. These logs automatically contain useful information including the timestamp, module, and context of the message.
21+
22+
The core object of the logging functionality in Python is the logger. A logging instance, `log`, can be initialised as follows. The phrase "Hello, World!" is used as an input to the logger, with this message being bundled up by other useful information, including the severity level, to form whats known as a LogRecord. This record will then be transmitted as required.
23+
24+
```python
25+
import logging
26+
log = logging.getLogger("Demo")
27+
log.warning("Hello, world!")
28+
29+
>> Hello, World!
30+
```
31+
32+
### Severity levels
33+
34+
Every record has an attached severity level, which can be used to flag how important a log record is. By default, Python has 5 main levels and one 'notset' level as shown in the image below[^1]:
35+
36+
![log_level_overview](img/loglevels.png)
37+
38+
[^1]: more can be defined as required, see Python's logging manual.
39+
40+
Each logging instance can have an attached severity level. If it has one, then only records that have the same severity level or higher will be transmitted.
41+
42+
```python
43+
import logging
44+
log = logging.getLogger("Demo", level = logging.WARNING)
45+
46+
log.info("This will not print")
47+
log.warning("This will print")
48+
49+
>> This will print
50+
```
51+
52+
### Handlers
53+
54+
Handlers are a key concept of logging in Python, as they control how the records are processed and formatted. There are several default one that the DAQ uses, and there are also several ones that are custom defined for the purposes of the DAQ.
55+
56+
The example below shows an example of a file handler, a stream handler, and a webhook handler. As can be seen, each of the records are processed and formatted by each of the handlers and transmitted in each of their respective ways.
57+
58+
![drunc_overview](img/handlers.png)
59+
60+
Importantly, each handler can have its own associated severity level! In the example above, it is certainly possible to have the WebHookHandler to only transmit if a record is of the level Warning or higher.
61+
62+
63+
64+
### Filters
65+
A sidegrade and important add on for the loggers is the filters, whos primary purpose is to decide if an error should be transmitted or not. Filters can be attached to both the logger instance as well as any handlers attached to the logger instance itself.
66+
67+
When a log record arrives, it will first be processed by the filters attached to the loggers first. Should they pass, the record is then passed onto each handler as shown before, where they are then further processed by each handler's attached filters. Only when they pass will a log be record be transmitted.
68+
69+
![filters](img/filters.png)
70+
71+
### Inheritance
72+
73+
Another key part of logging in Python is the inheritance feature. Loggers are organised in a heirarchical fashion and so it is possible to initialise descendant loggers by chaining the names together with a period, such as "root.parent.child".
74+
75+
By default, loggers will inheret certain properties of the parent:
76+
- severity level of the logger
77+
- handlers (and all attached properties, including severity level and filters on handlers)
78+
79+
![inheritance](img/inheritance.png)
80+
81+
82+
83+
Note that a particular exceptoin is that they _don't_ inheret any filters attached to the logger itself.
84+
85+
A useful diagram to peruse is the [logging flow in the official docs](https://docs.python.org/2/howto/logging.html#logging-flow).
86+
87+
88+
# Daqpytools
89+
90+
## Initialising a handler
91+
92+
## Subsec of existing handlers, filters, and walkthrough
93+
94+
95+
96+
### Advanced logging
97+
98+
99+
#### Context
100+
101+
#### Handler streams
102+
103+
104+
#### Log Handler Conf
105+
106+
107+
#### Using ERS
108+
109+
110+
111+
112+
113+
114+
115+
116+
## Basic info
117+
basically see your presentation
118+
119+
- the basics of python logging
120+
- handlers
121+
- inheritance
122+
- log levels
123+
124+
## User story
125+
And then talk about the features in daqpytools as a user
126+
127+
- Best practices on initialising loggers
128+
- How to deal with handlertypes
129+
- How to deal with handlerconf
130+
- integratoin with ers
131+
132+
## Advanced handlers (WIP as of Dec 2025)
133+
134+
Aside from the core set of loggers and handlers defined natively in drunc, there are several other configurations that interplay with drunc. These can be best thought of as 'streams' which in interact with several other handlers.
135+
136+
<img width="510" height="561" alt="streams" src="https://github.com/user-attachments/assets/de95f47c-fe30-48f3-ac96-0f7c091bbc17" />
137+
138+
The native implementation in drunc is referred to as the 'base' stream and interacts with the three core handlers already discussed.
139+
Other streams include the 'Opmon' stream interacting with its own set of handlers, and the 'ERS' stream that interacts with different handlers based on the log message's severity level.
140+
141+
The 'ERS' configuration is configured in OKS ([for example here](https://github.com/DUNE-DAQ/daqsystemtest/blob/974965be6e96aff969c69a380ed34aa96705e802/config/daqsystemtest/ccm.data.xml#L189)), and are automatically parsed by drunc and daqpytools as they get used.
142+
143+
To deal with the constraints set above, a handler configuration dataclass is constructed to define the relevant configurations and a system of filters is initialised with every handler. When passing a log record that needs to be processed via a specific stream, the log record and the handler configuration is passed to the relevant logger, after which it is processed.
144+
145+
146+
## Dev story
147+
148+
149+

docs/img/filters.png

167 KB
Loading

docs/img/handlers.png

325 KB
Loading

docs/img/inheritance.png

92.1 KB
Loading

docs/img/loglevels.png

25.4 KB
Loading

0 commit comments

Comments
 (0)