Skip to content

Commit 9b23c68

Browse files
EgietjeosingaatjeJWillegerstychodubDiogossilva03
authored
Merge pull request #45 from MASTERS-Y2Q1-ISEP: Model visualisations
New feature for adding a graphical representation of the model to the Robot log file. Co-authored-by: osingaatje <douwe@dosinga.nl> Co-authored-by: JWillegers <148167.jw@gmail.com> Co-authored-by: tychodub <t.b.dubbeling@student.utwente.nl> Co-authored-by: Diogossilva03 <diogo.xuya@gmail.com> Co-authored-by: JFoederer <github@famfoe.nl>
1 parent bb760f6 commit 9b23c68

27 files changed

Lines changed: 2269 additions & 43 deletions

‎.github/workflows/run-tests.yml‎

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# This workflow installs required Python dependencies and then runs the available tests.
22
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
33

4-
name: Run Acceptance and Unit tests
4+
name: Run Acceptance and Unit tests (with and without visualisation dependencies)
55

66
on:
77
pull_request:
@@ -21,10 +21,18 @@ jobs:
2121
uses: actions/setup-python@v3
2222
with:
2323
python-version: "3.10" # Only the oldest supported Python version is included here
24-
- name: Install dependencies
25-
run: |
26-
python -m pip install --upgrade pip # upgrade pip to latest version
27-
pip install . # install pyproject.toml dependencies - excludes optional dependencies (such as visualisation)
28-
- name: Run tests
24+
25+
- name: Install dependencies (without extra visualisation dependencies)
2926
run: |
30-
python run_tests.py
27+
python -m pip install --upgrade pip # upgrade pip to latest version
28+
pip install . # no additional [visualisation] dependencies
29+
30+
- name: Run tests (without visualisation dependencies)
31+
run: python run_tests.py
32+
33+
- name: Install extra visualisation dependencies
34+
run: pip install ".[visualisation]" # extra [visualisation] dependencies in pyproject.toml
35+
36+
- name: Run Tests (with visualisation dependencies)
37+
run: python run_tests.py
38+

‎README.md‎

Lines changed: 55 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ The recommended installation method is using [pip](http://pip-installer.org)
3636

3737
After installation include `robotmbt` as library in your robot file to get access to the new functionality. To run your test suite model-based, use the __Treat this test suite model-based__ keyword as suite setup. Check the _How to model_ section to learn how to make your scenarios suitable for running model-based.
3838

39-
```
39+
```robotframework
4040
*** Settings ***
4141
Library robotmbt
4242
Suite Setup Treat this test suite model-based
@@ -50,7 +50,8 @@ Modelling can be done directly from [Robot framework](https://robotframework.org
5050

5151
Consider these two scenarios:
5252

53-
```
53+
```robotframework
54+
*** Test Cases ***
5455
Buying a postcard
5556
When you buy a new postcard
5657
then you have a blank postcard
@@ -63,7 +64,8 @@ Preparing for a birthday party
6364

6465
Mapping the dependencies between scenarios is done by annotating the steps with modelling info. Modelling info is added to the documentation of the step as shown below. Regular documentation can still be added, as long as `*model info*` starts on a new line and a white line is included after the last `:OUT:` expression.
6566

66-
```
67+
```robotframework
68+
*** Keywords ***
6769
you buy a new postcard
6870
[Documentation] *model info*
6971
... :IN: None
@@ -117,7 +119,8 @@ All example scenarios naturally contain data. This information is embedded in th
117119

118120
#### Step argument modifiers
119121

120-
```
122+
```robotframework
123+
*** Test Cases ***
121124
Personalising a birthday card
122125
Given there is a birthday card
123126
when Johan writes their name on the birthday card
@@ -126,7 +129,8 @@ Personalising a birthday card
126129

127130
The above scenario uses the name `Johan` to create a concrete example. But now suppose that from a testing perspective `Johan` and `Frederique` are part of the same equivalence class. Then the step `Frederique writes their name on the birthday card` would yield an equally valid scenario. This can be achieved by adding a modifier (`:MOD:`) to the model info of the step. The format of a modifier is a Robot argument to which you assign a list of options. The modifier updates the argument value to a randomly chosen value from the specified options.
128131

129-
```
132+
```robotframework
133+
*** Keywords ***
130134
${person} writes their name on the birthday card
131135
[Documentation] *model info*
132136
... :MOD: ${person}= [Johan, Frederique]
@@ -138,7 +142,8 @@ ${person} writes their name on the birthday card
138142

139143
When constructing examples, they often express relations between multiple actors, where each actor can appear in multiple steps. This makes it important to know how modifiers behave when there are multiple modifiers in a scenario.
140144

141-
```
145+
```robotframework
146+
*** Test Cases ***
142147
Addressing a birthday card
143148
Given Tannaz is having their birthday
144149
and Johan has a birthday card
@@ -148,7 +153,8 @@ Addressing a birthday card
148153

149154
Have a look at the when-step above. We will assume the model already contains a domain term with two properties: `birthday.celebrant = Tannaz` and `birthday.guests = [Johan, Frederique]`.
150155

151-
```
156+
```robotframework
157+
*** Keywords ***
152158
${sender} writes the address of ${receiver} on the birthday card
153159
[Documentation] *model info*
154160
... :MOD: ${sender}= birthday.guests
@@ -175,10 +181,12 @@ It is not possible to add new options to an existing example value. Any constrai
175181

176182
It is possible for a step to keep the same options. The special `.*` notation lets you keep the available options as-is. Preceding steps must then supply the possible options. Some steps can, or must, deal with multiple independent sets of options that must not be mixed, because the expected results should differ. Suppose you have a set of valid and invalid passwords. You might be reluctant to include the superset of these as options to an authentication step. Instead, you can use `:MOD: ${password}= .*` as the modifier for that step. Like in the when-step for this scenario:
177183

178-
```
179-
Given 'secret' is too weak a password
180-
When user tries to update their password to 'secret'
181-
then the password is rejected
184+
```robotframework
185+
*** Test Cases ***
186+
Reject password
187+
Given 'secret' is too weak a password
188+
When user tries to update their password to 'secret'
189+
then the password is rejected
182190
```
183191

184192
In a then-step, modifiers behave slightly different. In then-steps no new option constraints are accepted for an argument. Its value must already have been determined during the given- and when-steps. In other words, regardless of the actual modifier, the expression behaves as if it were `.*`. The exception to this is when a then-step signals the first use of a new example value. In that case the argument value from the original scenario text is used.
@@ -193,12 +201,47 @@ For now, variable data considers strict equivalence classes only. This means tha
193201

194202
By default, trace generation is random. The random seed used for the trace is logged by _Treat this test suite model-based_. This seed can be used to rerun the same trace, if no external random factors influence the test run. To activate the seed, pass it as argument:
195203

196-
```
204+
```robotframework
197205
Treat this test suite model-based seed=eag-etou-cxi-leamv-jsi
198206
```
199207

200208
Using `seed=new` will force generation of a new reusable seed and is identical to omitting the seed argument. To completely bypass seed generation and use the system's random source, use `seed=None`. This has even more variation but does not produce a reusable seed.
201209

210+
### Graphs
211+
212+
A graph can be included in the log file to visualise how scenarios are linked. This helps in understanding a test suite's structure and reveals alternative paths that did not make it into the final trace.
213+
214+
To enable graph generation, some extra dependencies must be installed: `pip install robotframework-mbt[visualisation]`
215+
216+
Generate the graph by setting the graph style for the model-based suite. The graph will be included in the Robot log file as part of the keyword's logging.
217+
218+
```robotframework
219+
Treat this test suite Model-based graph=scenario
220+
```
221+
222+
Available graph styles:
223+
224+
* scenario
225+
* Compact view: Each scenario is shown as one node.
226+
* scenario-delta-value
227+
* Expanded view: Scenarios can become multiple nodes if they affect system state in different ways.
228+
229+
#### Exporting and importing graph data
230+
231+
Graph data can be stored by setting an output directory using argument `export_graph_data`. This createss a json file, named after the test suite, in the selected folder. Any accessable path can be used. For your convenience, Robot Framework offers an [automatic variable](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#automatic-variables) `${OUTPUT_DIR}` that points to this run's output directory. The `export_graph_data` argument can be used independent of the `graph` argument.
232+
233+
```robotframework
234+
Treat this test suite Model-based export_graph_data=${OUTPUT_DIR}
235+
```
236+
237+
To recreate a graph from previously exported graph data, use:
238+
239+
```robotframework
240+
Show model graph from exported file json_file_path=<file_path> graph_style=scenario
241+
```
242+
243+
This will draw a graph from the exported file, without the need to rerun the test suite. It is possible to select a different graph style than was used during the test run. If no graph style is selected, then the scenario graph style is used.
244+
202245
### Option management
203246

204247
If you want to set configuration options for use in multiple test suites without having to repeat them, the keywords __Set model-based options__ and __Update model-based options__ can be used to configure RobotMBT library options. _Set_ takes the provided options and discards any previously set options. _Update_ allows you to modify existing options or add new ones. Reset all options by calling _Set_ without arguments. Direct options provided to __Treat this test suite model-based__ take precedence over library options and affect only the current test suite.

‎atest/robotMBT tests/10__visualisation/01__scenario_graph/__init__.robot‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
*** Settings ***
2+
Suite Setup Clear prior exports ${OUTPUT_DIR}${/}run_model_with_graph.json
3+
Library ../graph_checker.py

‎atest/robotMBT tests/10__visualisation/01__scenario_graph/run_model_with_graph.robot‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
*** Settings ***
2+
Documentation This suite runs model-based with the graphing option enabled. The checks for the contents
3+
... of the graph are delegated to the second robot file in the same folder. Export/import
4+
... functionality is used to gain access to the graph contents.
5+
Suite Setup Treat this test suite Model-based graph=scenario export_graph_data=${OUTPUT_DIR}
6+
Resource ../../../resources/birthday_cards_data_variation.resource
7+
Library robotmbt
8+
9+
*** Test Cases ***
10+
Buying a card
11+
Given Jonathan is having their birthday
12+
and Douwe is a friend of Jonathan
13+
and Diogo is a friend of Jonathan
14+
and Tycho is a friend of Jonathan
15+
and Thomas is a friend of Jonathan
16+
When Douwe buys a birthday card
17+
then there is a blank birthday card available
18+
19+
Someone writes their name on the card
20+
Given there is a birthday card
21+
and Diogo's name is not yet on the birthday card
22+
when Diogo writes their name on the birthday card
23+
then the birthday card has 'Diogo' written on it
24+
25+
At least 3 people can write their name on the card
26+
Given the birthday card has 2 different names written on it
27+
and Tycho's name is not yet on the birthday card
28+
when Tycho writes their name on the birthday card
29+
then the birthday card has 3 different names written on it

‎atest/robotMBT tests/10__visualisation/01__scenario_graph/verify_model_graph.robot‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
*** Settings ***
2+
Documentation This suite takes the graph generated in the first suite in this folder
3+
... and checks the content's properties.
4+
Library ../graph_checker.py
5+
Library Collections
6+
Suite Setup Import graph data from ${OUTPUT_DIR}${/}run_model_with_graph.json
7+
8+
*** Test Cases ***
9+
Scenarios are nodes in the graph
10+
VAR @{expected_nodes} start
11+
... Buying a card
12+
... Someone writes their name on the card
13+
... At least 3 people can write their name on the card
14+
${node_count}= Number of graph nodes
15+
Should be equal ${node_count} ${4}
16+
@{all_nodes}= List of node titles
17+
Lists should be equal ${all_nodes} ${expected_nodes} ignore_order=True
18+
19+
Dependent nodes are connected by directed edges
20+
@{successors}= All successors to node Buying a card
21+
VAR @{next_node} Someone writes their name on the card
22+
Should be equal ${successors} ${next_node}
23+
@{successors}= All successors to node Someone writes their name on the card
24+
Should not contain ${successors} Buying a card
25+
26+
The dependency order for nodes is top-down
27+
${first_pos}= Vertical position of node Buying a card
28+
${second_pos}= Vertical position of node Someone writes their name on the card
29+
Should be true ${second_pos} < ${first_pos}
30+
31+
Repeating scenarios have a self-looping edge
32+
@{successors}= All successors to node Someone writes their name on the card
33+
Should contain ${successors} Someone writes their name on the card

‎atest/robotMBT tests/10__visualisation/02__scenario-delta-value_graph/__init__.robot‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
*** Settings ***
2+
Suite Setup Clear prior exports ${OUTPUT_DIR}${/}run_model_with_graph.json
3+
Library ../graph_checker.py

‎atest/robotMBT tests/10__visualisation/02__scenario-delta-value_graph/run_model_with_graph.robot‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
*** Settings ***
2+
Documentation This suite runs model-based with the graphing option enabled. The checks for the contents
3+
... of the graph are delegated to the second robot file in the same folder. Export/import
4+
... functionality is used to gain access to the graph contents.
5+
Suite Setup Treat this test suite Model-based graph=scenario-delta-value export_graph_data=${OUTPUT_DIR}
6+
Resource ../../../resources/birthday_cards_data_variation.resource
7+
Library robotmbt
8+
9+
*** Test Cases ***
10+
Buying a card
11+
Given Jonathan is having their birthday
12+
and Douwe is a friend of Jonathan
13+
and Diogo is a friend of Jonathan
14+
and Tycho is a friend of Jonathan
15+
and Thomas is a friend of Jonathan
16+
When Douwe buys a birthday card
17+
then there is a blank birthday card available
18+
19+
Someone writes their name on the card
20+
Given there is a birthday card
21+
and Diogo's name is not yet on the birthday card
22+
when Diogo writes their name on the birthday card
23+
then the birthday card has 'Diogo' written on it
24+
25+
At least 3 people can write their name on the card
26+
Given the birthday card has 2 different names written on it
27+
and Tycho's name is not yet on the birthday card
28+
when Tycho writes their name on the birthday card
29+
then the birthday card has 3 different names written on it

‎atest/robotMBT tests/10__visualisation/02__scenario-delta-value_graph/verify_model_graph.robot‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
*** Settings ***
2+
Documentation This suite takes the graph generated in the first suite in this folder
3+
... and checks how the content's properties for the scenario-delta-value
4+
... graph are different compared to the scenario graph.
5+
Library ../graph_checker.py
6+
Library Collections
7+
Suite Setup Import graph data from ${OUTPUT_DIR}${/}run_model_with_graph.json graph_type=scenario-delta-value
8+
9+
*** Test Cases ***
10+
Repeated scenario with different states become separate nodes
11+
[Documentation] In the standard scenario graph, the repeated scenario would show up
12+
... as a self-looping edge. In the scenario-delta-value variant, also
13+
... state changes are taken into account, meaning that the two variants
14+
... will each get their own node.
15+
${node_count}= Number of graph nodes
16+
Should be true ${node_count} > ${4}
17+
@{all_nodes}= List of node titles
18+
Should contain ${all_nodes} Someone writes their name on the card
19+
Should contain ${all_nodes} Someone writes their name on the card (rep 2)
20+
21+
Full node text contains state info
22+
[Documentation] Next to the scenario name, the node text for scenario-delta-value graphs
23+
... also contians information about the model state. 'Buying a card'
24+
... initialises the modeland contains all properties in their initial state.
25+
... 'host' and 'names' are properties being tracked by the model.
26+
${full_text}= Full node text of node Buying a card
27+
Should contain ${full_text} host
28+
Should contain ${full_text} names
29+
30+
Full node text only contains changed information
31+
[Documentation] When someone writes their name on the card, then the host or celebrant
32+
... does not change. The node for this scenario should only show the state
33+
... that changed during this scenario.
34+
${full_text}= Full node text of node Someone writes their name on the card (rep 2)
35+
Should contain ${full_text} names
36+
Should not contain ${full_text} host

‎atest/robotMBT tests/10__visualisation/03__verify_importing_graphs.robot‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
*** Settings ***
2+
Documentation This suite takes the graph generated earlier in the visualisation suite
3+
... and uses it to check import functionality.
4+
Library robotmbt
5+
Library graph_checker.py
6+
Library Collections
7+
8+
*** Variables ***
9+
${prior_export} ${OUTPUT_DIR}${/}run_model_with_graph.json
10+
11+
*** Test Cases ***
12+
Import as any graph type
13+
[Documentation] The data that is stored during export is always the compelte data set. This
14+
... has the advantage that different graph types can be reconstructed from the
15+
... data without rerunning the model.
16+
Import graph data from ${prior_export} graph_type=scenario
17+
Show model graph from exported file ${prior_export} graph_style=scenario
18+
${node_count}= Number of graph nodes
19+
Should be equal ${node_count} ${4}
20+
Import graph data from ${prior_export} graph_type=scenario-delta-value
21+
Show model graph from exported file ${prior_export} graph_style=scenario-delta-value
22+
${node_count}= Number of graph nodes
23+
Should be true ${node_count} > ${4}
24+
25+
Future major version bump imports are rejected
26+
${future_file}= Modify export file with future major version number ${prior_export}
27+
Run keyword and expect error *incompatible RobotMBT version
28+
... Import graph data from ${future_file}
29+
30+
Future minor verion bump imports are accepted
31+
${future_file}= Modify export file with future minor version number ${prior_export}
32+
Import graph data from ${future_file}
33+
${node_count}= Number of graph nodes
34+
Should be equal ${node_count} ${4}
35+
36+
Ill-formed files are rejected for import
37+
${corrupted_file}= Corrupt export file ${prior_export}
38+
Run keyword and expect error *could not be loaded as RobotMBT graph data
39+
... Import graph data from ${corrupted_file}

‎atest/robotMBT tests/10__visualisation/__init__.robot‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
*** Settings ***
2+
Suite Setup Skip if optional visualisation is not installed
3+
Library graph_checker.py
4+
5+
*** Keywords ***
6+
Skip if optional visualisation is not installed
7+
${partial_installation}= Graphing dependencies missing
8+
Skip If ${partial_installation} Visualisation dependencies not installed. Please read the README for information on how to do this.

0 commit comments

Comments
 (0)