Skip to content

Commit 6878015

Browse files
Benedikt Volkelchiarazampolli
Benedikt Volkel
authored and
chiarazampolli
committed
Update documentation
1 parent 538e4fa commit 6878015

2 files changed

Lines changed: 107 additions & 101 deletions

File tree

‎RelVal/README.md‎

Lines changed: 40 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,11 @@ There are 2 ROOT macros which can in principle be used as-is. Their functionalit
1212

1313
This macro allows to compare 2 ROOT files that contain `TH1` objects. Objects are considered to correspond to each other if they have the same name.
1414
At the moment, 3 different comparisons are implemented:
15-
1. relative difference of bin contents,
16-
1. Chi2 test,
17-
1. simple comparison of number of entries
15+
1. `chi2`: Chi2 test of compared histograms,
16+
1. `bin_cont`: relative difference of normalised bin content of both histograms,
17+
1. `num_entries`: relative difference in the number of entries.
1818

19-
The first 2 tests are considered critical, hence if the threshold is exceeded, the comparison result is named `BAD`.
19+
The first 2 tests are considered critical, hence if the threshold is exceeded, the comparison result is named `BAD`. Also the third test is considered critical in case efficiencies are compared coming from `TEfficiency` objects.
2020

2121
There are 5 different test severities per test:
2222
1. `GOOD` if the threshold was not exceeded,
@@ -49,52 +49,40 @@ The wrapper includes 3 different sub-commands for now
4949
1. `compare` to compare the results of 2 RelVal runs,
5050
1. `influx` to convert the summary into a format that can be understood by and sent to an InfluxDB instance.
5151

52+
Each sub-command can be run with `--help` to see all options/flags.
53+
5254
### Basic usage
5355

5456
If you would like to compare 2 files, simply run
5557
```bash
56-
python o2dpg_release_validation.py rel-val -i <list-of-first-files> -j <list-of-second-files> [-o <output/dir>] [--include-dirs <list-of-directories>]
58+
python o2dpg_release_validation.py rel-val -i <first-list-of-files> -j <second-list-of-files> [-o <output/dir>] [--include-dirs <list-of-directories>]
5759
```
5860
This performs all of the above mentioned tests. If only certain tests should be performed, this can be achieved with the flags `--with-test-<which-test>` where `<which-test>` is one of
5961
1. `chi2`,
60-
1. `bincont`,
61-
1. `numentries`.
62+
1. `bin-cont`,
63+
1. `num-entries`.
6264

6365
By default, all of them are switched on.
6466

65-
If `--include-dirs` is specified, only objects under those directories are taken into account. Note that this is not a patter matching but it needs to start with the top directory. Thus, if for instance `--include-dirs /path/to/interesting`, everything below that path will be considered. However, something placed in `/another/path/to/interesting` will not be considered.
66-
Note that `o2::quality_control::core::MonitorObjectCollection` is treated as a directory in this respect.
67-
68-
### Apply to entire simulation outcome
69-
70-
In addition to simply comparing 2 ROOT files, the script offers the possibility of comparing 2 corresponding directories that contain simulation artifacts (and potentially QC and analysis results). It is not foreseen to run over everything inside those directories but the files must be specifiec via a small config file. See [this example](config/rel_val_sim_dirs_default.json). It is passed via the option `--dirs-config`. In addition, top-level keys can be enabled(disabled) with `--dirs-config-enable <keys>`(`dirs-config-disable <keys>`) where disabling takes precedence.
71-
72-
**NOTE** That each single one of the comparisons is only done if mutual files were found in the 2 corresponding directories. As an example, one could do
73-
```bash
74-
cd ${DIR1}
75-
python o2dpg_workflow_runner.py -f <workflow-json1>
76-
cd ${DIR2}
77-
# potentially something has changed in the software or the simulation/reconstruction parameters
78-
python o2dpg_workflow_runner.py -f <workflow-json2>
79-
python ${O2DPG_ROOT}/ReleaseValidation/o2dpg_release_validation.py rel-val -i ${DIR1} -j ${DIR2} --dirs-config ${O2DPG_ROOT}/RelVal/config/rel_val_sim_dirs_default.json --dirs-config-enable QC [-o <output/dir>] [<test-flags>]
80-
```
81-
This would run the RelVal von everything specified under the top key `QC`.
67+
If `--include-dirs` is specified, only objects under those directories inside the given ROOT files are taken into account. Note that this is not a pattern matching but it needs to start with the top directory. Thus, if for instance `--include-dirs /path/to/interesting`, everything below that path will be considered. However, something placed in `/another/path/to/interesting` will not be considered.
68+
**Note** that `o2::quality_control::core::MonitorObjectCollection` is treated as a directory in this respect.
8269

8370
### Inspection and re-plotting summary grid
8471

8572
This is done via
8673
```bash
87-
python ${O2DPG_ROOT}/ReleaseValidation/o2dpg_release_validation.py inspect <path-to-outputdir-or-file> [--include-patterns <patterns>] [--plot] [--flags <severity-flags>]
74+
python ${O2DPG_ROOT}/RelVal/o2dpg_release_validation.py inspect <path-to-outputdir-or-summary-json> [--include-patterns <patterns>] [--plot] [--flags <severity-flags>] [-o <output-dir>]
8875
```
8976
If only a path is provided, a summary will be printed on the screen showing the number of `GOOD`, `CRIT_NC` and `BAD`.
9077
Adding patterns for `--include-patterns` only objects matching at least one of the patterns will be taken into account for the summary.
91-
If `--plot` is provided, another summary grid will be written into the same directory passed to the `inspect` command and it will be called `SummaryTestUser.png`. If `--flags` are given, only the objects where at least one test has one of the flags will be included in the grid.
78+
If `--plot` is provided, new summary plots (grid, pie charts, values compared to thresholds) will be produced. By default they are written to `<input-directory>/user_summary` or, if the `-o` option is provided, to the custom output directory.
79+
If `--flags` are given, only the objects where at least one test has one of the flags will be included in the grid.
9280

9381
### Make ready for InfluxDB
9482

9583
To convert the final output to something that can be digested by InfluxDB, use
9684
```bash
97-
python ${O2DPG_ROOT}/ReleaseValidation/o2dpg_release_validation.py influx --dir <rel-val-out-dir> [--tags k1=v1 k2=v2 ...] [--table-name <chosen-table-name>]
85+
python ${O2DPG_ROOT}/RelVal/o2dpg_release_validation.py influx --dir <rel-val-out-dir> [--tags k1=v1 k2=v2 ...] [--table-name <chosen-table-name>]
9886
```
9987
When the `--tags` argument is specified, these are injected as TAGS for InfluxDB in addition. The table name can also be specified explicitly; if not given, it defaults to `O2DPG_MC_ReleaseValidation`.
10088

@@ -110,25 +98,39 @@ There are various plots created during the RelVal run. For each compared file th
11098

11199
As mentioned above, the basic usage of the `rel-val` sub-command is straightforward. But there are quite a few more options available and some of them will be explained briefly below.
112100

113-
### Setting new thresholds from another RelVal run (towards threshold tuning)
101+
### Setting new/custom thresholds from another RelVal run
102+
Each RelVal run produces a `Summary.json` file in the corresponding output directories. Among other things, it contains the computed values of all tests for each compared histogram pair. Such a `Summary.json` can now be used as a input file for a future RelVal to set all thresholds according to the values. In fact, multiple such files can be passed and for each historgam-test combination, the mean or max of the previously calculated values can be used to set the new thresholds.
114103

115-
Imagine the scenario, where you assume that one has 2 outputs (either custom or full simulation output) which should be compatible. For instance, these could be 2 simulation runs with the same generator seed and reasonably high statistics and also otherwise with the same parameters.
116-
Running the RelVal on these directories will - as usual - yield the `<parent/output/dir/SummaryGlobal.json>` as well as `<parent/output/dir/sub/dirSummary.json>`. Now, assuming there is another simulation output from - for instance - another software version. To check, where this is truly worse in terms of the RelVal comparison, one could compare it to one of the "baseline" runs while setting all thresholds to the computed values of the first comparison. This can be done with
117104
```bash
118-
python ${O2DPG_ROOT}/ReleaseValidation/o2dpg_release_validation.py rel-val -i ${DIR1} ${DIR2} [-o <output/dir>] --use-values-as-thresholds <parent/output/dir/SummaryGlobal.json>
105+
python ${O2DPG_ROOT}/RelVal/o2dpg_release_validation.py rel-val -i <first-list-of-files> -j <second-list-of-files> --use-values-as-thresholds <list-of-summaries> [--combine-thresholds {mean,max}] [--test-<name>-threshold-margin <value>]
119106
```
120-
which will set each threshold individually per test and per histogram.
121-
122-
In addition each test threshold can be set globally for all histogram comparisons with
123-
* `--chi2-threshold <value>`,
124-
* `--rel-mean-diff-threshold <value>`,
125-
* `--rel-entries-diff-threshold <value>`.
107+
In addition, a margin for each test can be provided as shown in the command above. This is a factor by which the threshold is multiplied. So to add a `10%` margin for the chi2 test, simply put `test-chi2-threshold-margin 1.1`.
126108

127109
## RelVal for QC (examples)
128110

129111
### Comparing data with MC
130112

113+
There is an ongoing effort to unify the names of QC objects inside MC and data QC files. Some are already unified and the following command would run comparison of those. However, others are not yet unified and will not be considered in the comparison.
114+
131115
MC QC objects are usually distributed over multiple files while those from data are all contained in one single file. It is possible to directly compare them with
132116
```bash
133-
python ${O2DPG_ROOT}/ReleaseValidation/o2dpg_release_validation.py rel-val -i ${MC_PRODUCTION}/QC/*.root -j ${DATA_PRODUCTION}/QC.root [--inlcude-dirs <include-directories]
117+
python ${O2DPG_ROOT}/RelVal/o2dpg_release_validation.py rel-val -i ${MC_PRODUCTION}/QC/*.root -j ${DATA_PRODUCTION}/QC.root [--inlcude-dirs <include-directories]
134118
```
119+
120+
### Apply to entire simulation outcome
121+
122+
**This is still under development and does not yet work when e.g. comparing an MC directory to a data directory.**
123+
124+
In addition to simply comparing 2 ROOT files, the script offers the possibility of comparing 2 corresponding directories that contain simulation artifacts (and potentially QC and analysis results). It is not foreseen to run over everything inside those directories but the files must be specifiec via a small config file. See [this example](config/rel_val_sim_dirs_default.json). It is passed via the option `--dirs-config`. In addition, top-level keys can be enabled(disabled) with `--dirs-config-enable <keys>`(`dirs-config-disable <keys>`) where disabling takes precedence.
125+
126+
**NOTE** That each single one of the comparisons is only done if mutual files were found in the 2 corresponding directories. As an example, one could do
127+
```bash
128+
cd ${DIR1}
129+
python o2dpg_workflow_runner.py -f <workflow-json1>
130+
cd ${DIR2}
131+
# potentially something has changed in the software or the simulation/reconstruction parameters
132+
python o2dpg_workflow_runner.py -f <workflow-json2>
133+
python ${O2DPG_ROOT}/RelVal/o2dpg_release_validation.py rel-val -i ${DIR1} -j ${DIR2} --dirs-config ${O2DPG_ROOT}/RelVal/config/rel_val_sim_dirs_default.json --dirs-config-enable QC [-o <output/dir>] [<test-flags>]
134+
```
135+
This would run the RelVal von everything specified under the top key `QC`.
136+

‎RelVal/o2dpg_release_validation.py‎

Lines changed: 67 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -12,69 +12,73 @@
1212
# See O2DPG/RelVal/config/rel_val_sim_dirs_default.json for as an example
1313
#
1414
# The full help message would be
15-
#usage: o2dpg_release_validation.py rel-val [-h] -i [INPUT1 ...] -j
16-
# [INPUT2 ...]
17-
# [--chi2-threshold CHI2_THRESHOLD]
18-
# [--rel-mean-diff-threshold REL_MEAN_DIFF_THRESHOLD]
19-
# [--rel-entries-diff-threshold REL_ENTRIES_DIFF_THRESHOLD]
20-
# [--use-values-as-thresholds USE_VALUES_AS_THRESHOLDS]
21-
# [--chi2-threshold-margin CHI2_THRESHOLD_MARGIN]
22-
# [--rel-mean-diff-threshold-margin REL_MEAN_DIFF_THRESHOLD_MARGIN]
23-
# [--rel-entries-diff-threshold-margin REL_ENTRIES_DIFF_THRESHOLD_MARGIN]
24-
# [--with-test-chi2]
25-
# [--with-test-bincont]
26-
# [--with-test-numentries]
27-
# [--dir-config DIR_CONFIG]
28-
# [--dir-config-enable [DIR_CONFIG_ENABLE ...]]
29-
# [--dir-config-disable [DIR_CONFIG_DISABLE ...]]
30-
# [--include-dirs [INCLUDE_DIRS ...]]
31-
# [--add] [--output OUTPUT]
32-
#
33-
#optional arguments:
34-
# -h, --help show this help message and exit
35-
# -i [INPUT1 ...], --input1 [INPUT1 ...]
36-
# EITHER first set of input files for comparison OR
37-
# first input directory from simulation for comparison
38-
# -j [INPUT2 ...], --input2 [INPUT2 ...]
39-
# EITHER second set of input files for comparison OR
40-
# second input directory from simulation for comparison
41-
# --chi2-threshold CHI2_THRESHOLD
42-
# Chi2 threshold
43-
# --rel-mean-diff-threshold REL_MEAN_DIFF_THRESHOLD
44-
# Threshold of relative difference in mean
45-
# --rel-entries-diff-threshold REL_ENTRIES_DIFF_THRESHOLD
46-
# Threshold of relative difference in number of entries
47-
# --use-values-as-thresholds USE_VALUES_AS_THRESHOLDS
48-
# Use values from another run as thresholds for this one
49-
# --chi2-threshold-margin CHI2_THRESHOLD_MARGIN
50-
# Margin to apply to the chi2 threshold extracted from
51-
# file
52-
# --rel-mean-diff-threshold-margin REL_MEAN_DIFF_THRESHOLD_MARGIN
53-
# Margin to apply to the rel_mean_diff threshold
54-
# extracted from file
55-
# --rel-entries-diff-threshold-margin REL_ENTRIES_DIFF_THRESHOLD_MARGIN
56-
# Margin to apply to the rel_entries_diff threshold
57-
# extracted from file
58-
# --with-test-chi2 run chi2 test
59-
# --with-test-bincont run bin-content test
60-
# --with-test-numentries
61-
# run number-of-entries test
62-
# --dir-config DIR_CONFIG
63-
# What to take into account in a given directory
64-
# --dir-config-enable [DIR_CONFIG_ENABLE ...]
65-
# only enable these top keys in your dir-config
66-
# --dir-config-disable [DIR_CONFIG_DISABLE ...]
67-
# disable these top keys in your dir-config (precedence
68-
# over dir-config-enable)
69-
# --include-dirs [INCLUDE_DIRS ...]
70-
# only inlcude directories; note that each pattern is
71-
# assumed to start in the top-directory (at the moment
72-
# no regex or *)
73-
# --add If given and there is already a RelVal in the output
74-
# directory, extracted objects will be added to the
75-
# existing ones
76-
# --output OUTPUT, -o OUTPUT
77-
# output directory
15+
# usage: o2dpg_release_validation.py rel-val [-h] -i [INPUT1 ...] -j
16+
# [INPUT2 ...]
17+
# [--use-values-as-thresholds [USE_VALUES_AS_THRESHOLDS ...]]
18+
# [--combine-thresholds {mean,max}]
19+
# [--with-test-chi2]
20+
# [--test-chi2-threshold CHI2_THRESHOLD]
21+
# [--test-chi2-threshold-margin CHI2_THRESHOLD_MARGIN]
22+
# [--with-test-bin-cont]
23+
# [--test-bin-cont-threshold BIN_CONT_THRESHOLD]
24+
# [--test-bin-cont-threshold-margin BIN_CONT_THRESHOLD_MARGIN]
25+
# [--with-test-num-entries]
26+
# [--test-num-entries-threshold NUM_ENTRIES_THRESHOLD]
27+
# [--test-num-entries-threshold-margin NUM_ENTRIES_THRESHOLD_MARGIN]
28+
# [--dir-config DIR_CONFIG]
29+
# [--dir-config-enable [DIR_CONFIG_ENABLE ...]]
30+
# [--dir-config-disable [DIR_CONFIG_DISABLE ...]]
31+
# [--include-dirs [INCLUDE_DIRS ...]]
32+
# [--add] [--output OUTPUT]
33+
#
34+
# optional arguments:
35+
# -h, --help show this help message and exit
36+
# -i [INPUT1 ...], --input1 [INPUT1 ...]
37+
# EITHER first set of input files for comparison OR
38+
# first input directory from simulation for comparison
39+
# -j [INPUT2 ...], --input2 [INPUT2 ...]
40+
# EITHER second set of input files for comparison OR
41+
# second input directory from simulation for comparison
42+
# --use-values-as-thresholds [USE_VALUES_AS_THRESHOLDS ...]
43+
# Use values from another run as thresholds for this one
44+
# --combine-thresholds {mean,max}
45+
# Arithmetic mean or maximum is chosen as threshold
46+
# value
47+
# --with-test-chi2 run chi2 test
48+
# --test-chi2-threshold CHI2_THRESHOLD
49+
# chi2 threshold
50+
# --test-chi2-threshold-margin CHI2_THRESHOLD_MARGIN
51+
# Margin to apply to the chi2 threshold extracted from
52+
# file
53+
# --with-test-bin-cont run bin_cont test
54+
# --test-bin-cont-threshold BIN_CONT_THRESHOLD
55+
# bin_cont threshold
56+
# --test-bin-cont-threshold-margin BIN_CONT_THRESHOLD_MARGIN
57+
# Margin to apply to the bin_cont threshold extracted
58+
# from file
59+
# --with-test-num-entries
60+
# run num_entries test
61+
# --test-num-entries-threshold NUM_ENTRIES_THRESHOLD
62+
# num_entries threshold
63+
# --test-num-entries-threshold-margin NUM_ENTRIES_THRESHOLD_MARGIN
64+
# Margin to apply to the num_entries threshold extracted
65+
# from file
66+
# --dir-config DIR_CONFIG
67+
# What to take into account in a given directory
68+
# --dir-config-enable [DIR_CONFIG_ENABLE ...]
69+
# only enable these top keys in your dir-config
70+
# --dir-config-disable [DIR_CONFIG_DISABLE ...]
71+
# disable these top keys in your dir-config (precedence
72+
# over dir-config-enable)
73+
# --include-dirs [INCLUDE_DIRS ...]
74+
# only inlcude directories; note that each pattern is
75+
# assumed to start in the top-directory (at the moment
76+
# no regex or *)
77+
# --add If given and there is already a RelVal in the output
78+
# directory, extracted objects will be added to the
79+
# existing ones
80+
# --output OUTPUT, -o OUTPUT
81+
# output directory
7882

7983
import sys
8084
import argparse

0 commit comments

Comments
 (0)