You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
## How It Works: Test Bundles and Test Definitions
32
+
17
33
The project relies on the [pytest framework](https://docs.pytest.org/) to setup and execute the tests.
18
34
Nuts itself is written as a custom pytest plugin. In the background, [nornir](https://nornir.readthedocs.io/)
19
35
executes specific network tasks for the actual tests.
20
36
21
-
Additionally, nuts treats the test definition and the so-called test bundle as separate entities. The *test definition* is modeled as a custom `pytest.Class`, and a predefined set of test definitions can be found in the module `base_tests`. New test definitions can be added easily by the user of the plugin.
37
+
Nuts treats the test definition and the so-called test bundle as separate entities. The *test definition* is modeled as a custom `pytest.Class`, and a predefined set of test definitions can be found in the nuts module `base_tests`. New test definitions can be added easily by the user of the plugin.
22
38
23
39
The *test bundle* is a file that is parsed by pytest. The file provides data on the desired network state and describes which test definitions should be collected and executed by pytest.
24
40
The structure of the test bundle should enable people without in-depth python knowledge to add new test bundles or update existing ones to reflect changes in the network.
25
41
26
42
While the readme here is only a short overview, find the [documentation of nuts on readthedocs](https://nuts.readthedocs.io/en/latest/).
27
43
28
-
## Test bundle structure
44
+
###Test Bundle Structure
29
45
30
46
Currently only yaml files are supported as test bundles,
31
47
but other sources such as other file formats or database entries can be considered in later nuts versions.
@@ -42,7 +58,7 @@ Each test bundle contains the following structure:
42
58
```
43
59
`test_module`: The full path of the python module that contains the test class to be used.
44
60
This value is optional if the test class is registered in `index.py` of the pytest-nuts plugin.
45
-
Note that it can be relevant in which directory `pytest` is started if local test modules are used.
61
+
Note that it can be relevant in which directory `pytest` is started if local test modules are used. Using `test_modules` allows you to write your own test classes. **Note: We currently do not support self-written test modules, since upcoming refactorings might introduce breaking changes.**
46
62
47
63
`test_class`: The name of the python class which contains the tests that should be executed.
48
64
Note that currently every test in this class will be executed.
@@ -58,7 +74,7 @@ This allows the additional `max_drop` parameter in `test_execution`, since it is
58
74
59
75
`test_data`: Data that is used to parametrize the tests in the test class which have the `pytest.mark.nuts` annotation. It is additionally part of the `nuts_parameters` property.
60
76
61
-
### Examples
77
+
### Example: CDP Neighbors
62
78
Example of a test bundle for `TestNetmikoCdpNeighbors` which tests that `R1` is a CDP Neighbor of both `R2` and `R3`.
63
79
This example creates three different tests, one for each entry in the `test_data` list.
64
80
@@ -85,14 +101,7 @@ This example creates three different tests, one for each entry in the `test_data
85
101
...
86
102
```
87
103
88
-
## Manual installation instructions
89
-
Nuts uses [poetry](https://python-poetry.org/) as a dependency manager.
### How the Test Bundle Is Converted to a Pytest Test
96
105
97
106
When nuts is executed, pytest converts the test bundles (the yaml files) into tests. During test collection, the custom pytest marker `nuts` uses the data that has been defined in the test bundle mentioned above.
98
107
This annotation is a wrapper around the `pytest.mark.parametrize` annotation and allows the plugin to use the data entries from the test bundle. For each entry in the `test_data` section of the test bundle, the custom marker generates a single test case. To achieve this, the plugin transforms the entries into n-tuples, since `pytest.mark.parametrize` expects a list of n-tuples as input.
@@ -102,18 +111,16 @@ For each entry in `test_data` these fields are extracted and transformed to a tu
102
111
If any of these fields are not present in an entry of `test_data`, the corresponding test case will be skipped.
103
112
A second argument determines optional fields that can also be used in a test case as well - non-present values are passed into the function as `None`.
104
113
105
-
#### Example of a test class with custom marker
114
+
The following test-run of CDP neighbors for example checks the local port:
106
115
107
116
```python
108
-
@pytest.mark.usefixtures("check_nuts_result")# see below
This test-run of CDP neighbors checks the local port.
116
-
117
124
Before each test evaluation, the fixture `@pytest.mark.usefixtures("check_nuts_result")` checks the result of the network information that has been gathered by nornir in the background: It asserts that that no exception was thrown while doing so.
118
125
119
126
The required fields are `host`, `remote_host` and `local_port` - they must be present in the custom marker,
@@ -126,15 +133,15 @@ Each test module implements a context class to provide module-specific functiona
126
133
This guarantees a consistent interface across all tests for test setup and execution.
127
134
Currently, the predefined test classes use [nornir](https://nornir.readthedocs.io/en/latest/) in order to communicate
128
135
with the network devices, therefore the test classes derive all from a more specific `NornirNutsContext`,
129
-
which provides a nornir instance and nornir-specific helpers.
136
+
which provides a nornir instance and nornir-specific helpers. In the example above, it is a class called `CdpNeighborsContext` that derives from `NornirNutsContext`.
130
137
131
-
## Development
138
+
## Develop Your Own Test Classes
132
139
133
140
Nuts is essentially designed as a pytest-plugin and it is possible to add your own, self-written test classes.
134
-
A dev documentation on how to write your own test classes is planned for the future.
135
-
Until then, please read the regular [documentation of nuts](https://nuts.readthedocs.io/en/latest/) on how to use it.
141
+
A dev documentation on how to write your own test classes is planned for a future release.
142
+
Still, it is possible to write your own test classes nevertheless, even if we cannot guarantee that upcoming planned refactorings do not introduce breaking changes.
136
143
137
144
# Thanks
138
145
139
146
*[Matthias Gabriel](https://github.com/MatthiasGabriel), who laid the foundations of nuts.
140
-
*[Florian Bruhin](https://github.com/The-Compiler) for invaluable feedback.
147
+
*[Florian Bruhin (The Compiler)](https://github.com/The-Compiler) for invaluable feedback and advice.
0 commit comments