Skip to content

Commit 8ab2611

Browse files
JFoedererJFoederer
authored
Merge pull request #50 from JFoederer/longer-traces
So far, test runs always ended when single coverage was achieved. I.e., when each scenario was hit at least once. This release introduces two new run targets: run duration and number of executed scenarios.
2 parents ec36d21 + f9f84e0 commit 8ab2611

57 files changed

Lines changed: 873 additions & 161 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 40 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# RobotMBT - the oneliner
22

3-
Model-based testing in Robot framework with test case generation
3+
Model-based testing in Robot framework with dynamic trace and test case generation
44

55
## Introduction
66

@@ -24,7 +24,7 @@ RobotMBT offers features to cover both _when_ and _what_ variations.
2424

2525
RobotMBT is suitable for sequencing complete scenarios, including action refinement for when-steps. Concrete example scenarios can be generalised for added data-driven variation. When all steps are properly annotated with modelling info, the library can resolve their dependencies and figure out the correct execution order. Each run a new test sequence is generated from the available options.
2626

27-
To be successful, the set of scenarios in the model must (for now) be composable into a single complete sequence, without leftovers. The same scenario can be inserted into the trace multiple times, creating loops, if repetition helps to reach the entry condition for later scenarios. Dead ends should be prevented, i.e., sequences from which there is no way forward and no way to loop back.
27+
To be successful, the set of scenarios in the model must be composable into a single complete sequence. There are no automatic resets or retries. The same scenario can be inserted into the trace multiple times, creating loops. Either to reach otherwise unreachable scenarios, or simply to create longer test runs. Dead ends should be prevented, i.e., sequences from which there is no way forward and no way to loop back.
2828

2929
## Getting started
3030

@@ -207,6 +207,36 @@ Modified example values do not cascade. If a modifier expression references anot
207207

208208
## Configuration options
209209

210+
Configure your test run by using any of the options from the list.
211+
212+
| option | purpose | values (default marked with *) |
213+
|-----------------------------------------|----------------------------------|--------------------------------|
214+
| [coverage_target](#setting-run-targets) | Each scenario must be executed at least this many times | 0 or 1* |
215+
| [scenario_target](#setting-run-targets) | The trace must have at least this many scenarios | 0* or higher |
216+
| [time_target](#setting-run-targets) | Setting a minimum test run duration | Robot time string |
217+
| [seed](#random-seed) | Re-running a prior trace | a specific seed, new* or None |
218+
| [batch_size](#batch-size) | Phased trace generation | 1 or higher (default 100*) |
219+
| [graph](#graphs) | Visualising the model | None*, scenario or scenario-delta-value |
220+
| [export_graph_data](#exporting-and-importing-graph-data) | Storing graphs as json data | None* or file path |
221+
222+
Options are available as named arguments:
223+
224+
```robotframework
225+
Treat this test suite model-based coverage_target=1 scenario_target=250
226+
```
227+
228+
If you want to set configuration options for use in multiple test suites without having to repeat them, the keywords __Set model-based options__ and __Update model-based options__ can be used to configure RobotMBT library options. _Set_ takes the provided options and discards any previously set options. _Update_ allows you to modify existing options or add new ones. Reset all options by calling _Set_ without arguments. Direct options provided to __Treat this test suite model-based__ take precedence over library options and affect only the current test suite.
229+
230+
Tip: [Robot dictionaries](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#dictionary-variable) (`&{ }`) can be used to group related options and pass them as one set.
231+
232+
### Setting run targets
233+
234+
By default, a trace will be generated that runs to _single coverage_, meaning that each scenario must be included in the trace at least once. This is equivalent to setting `coverage_target=1`. To continue generating longer traces after single coverage is achieved, a second target can be enabled. For example, `scenario_target=500` will continue to run until there are 500 scenarios in the trace. Note that when scenarios are split up due to when-step refinement, that each part will count as one scenario. Alternatively, setting `time_target=1 hour 30 min` will cause the test run to run at least this long. Refer to the Robot Framework documentation to find [supported time formats](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#time-as-time-string).
235+
236+
The test run will finish once all enabled targets are achieved. By default, only the coverage target is set. If you do not need guaranteed coverage, then `coverage_target=0` will disable the coverage check. Test runs can now finish before all scenarios are executed. This does not affect the trace generation process, which will still prefer new coverage over repetition.
237+
238+
Tip: _When generating large test suites, use Robot Framework's [Split log](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#splitting-logs) feature (`--splitlog`), to keep log file sizes manageable._
239+
210240
### Random seed
211241

212242
By default, trace generation is random. The random seed used for the trace is logged by _Treat this test suite model-based_. This seed can be used to rerun the same trace, if no external random factors influence the test run. To activate the seed, pass it as argument:
@@ -217,6 +247,14 @@ Treat this test suite model-based seed=eag-etou-cxi-leamv-jsi
217247

218248
Using `seed=new` will force generation of a new reusable seed and is identical to omitting the seed argument. To completely bypass seed generation and use the system's random source, use `seed=None`. This has even more variation but does not produce a reusable seed.
219249

250+
### Batch size
251+
252+
Trace generation is done in _Batches_, so that the test run can already start before the full trace is generated. Small batch sizes cause the test run to start quickly, whereas larger batch sizes give more room to find suitable traces. Batch size is configurable by setting `batch_size=`.
253+
254+
If the batch size is large enough to reach all run targets in a single batch, then the run will finish without further extensions. If not all targets are achieved in the first batch, then trace generation continues whenever new scenarios are needed. This is always at the end of a scenario. This scenario is tagged `mbt trace extension` and also contains the logging for the extended trace generation.
255+
256+
Tip: _Small batch sizes are good at exposing dead ends in your model._
257+
220258
### Graphs
221259

222260
A graph can be included in the log file to visualise how scenarios are linked. This helps in understanding a test suite's structure and reveals alternative paths that did not make it into the final trace.
@@ -252,12 +290,6 @@ Show model graph from exported file json_file_path=<file_path> graph_style
252290

253291
This will draw a graph from the exported file, without the need to rerun the test suite. It is possible to select a different graph style than was used during the test run. If no graph style is selected, then the scenario graph style is used.
254292

255-
### Option management
256-
257-
If you want to set configuration options for use in multiple test suites without having to repeat them, the keywords __Set model-based options__ and __Update model-based options__ can be used to configure RobotMBT library options. _Set_ takes the provided options and discards any previously set options. _Update_ allows you to modify existing options or add new ones. Reset all options by calling _Set_ without arguments. Direct options provided to __Treat this test suite model-based__ take precedence over library options and affect only the current test suite.
258-
259-
Tip: [Robot dictionaries](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#dictionary-variable) (`&{ }`) can be used to group related options and pass them as one set.
260-
261293
## Contributing
262294

263295
If you have feedback, ideas, or want to get involved in coding, then check out the [Contribution guidelines](https://github.com/JFoederer/robotframeworkMBT/blob/main/CONTRIBUTING.md).

‎atest/robotMBT tests/03__parse_model_info/MyProcessor.py‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
1-
class MyProcessor:
1+
from robotmbt import SuiteProcessor
22

3-
def process_test_suite(self, in_suite):
3+
4+
class MyProcessor(SuiteProcessor):
5+
def process_test_suite(self, in_suite, **kwargs):
6+
super().process_test_suite(in_suite, **kwargs)
47
self.in_suite = in_suite
58
self._fail_on_step_errors()
69
msg = "Model info not properly parsed"

‎atest/robotMBT tests/03__parse_model_info/correct_model_info.robot‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
*** Settings ***
22
Suite Setup Treat this test suite Model-based
33
Library MyProcessor.py
4-
Library robotmbt processor_lib=MyProcessor
4+
Library robotmbt processor=MyProcessor
55

66
*** Test cases ***
77
concise model info

‎atest/robotMBT tests/03__parse_model_info/incorrect_model_info.robot‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
*** Settings ***
22
Suite Setup Expect failing suite processing
33
Library MyProcessor.py
4-
Library robotmbt processor_lib=MyProcessor
4+
Library robotmbt processor=MyProcessor
55

66
*** Test Cases ***
77
fail on empty model info

atest/robotMBT tests/07__processor_options/option_handling/01__pass_option_directly.robot renamed to atest/robotMBT tests/07__processor_options/01__option_handling/01__pass_option_directly.robot

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0}
33
... AND Treat this test suite Model-based repeat=2
44
Suite Teardown Should be equal ${test_count} ${2}
55
Library suiterepeater.py
6-
Library robotmbt processor_lib=suiterepeater
6+
Library robotmbt processor=suiterepeater
77

88
*** Test Cases ***
99
only test case

atest/robotMBT tests/07__processor_options/option_handling/02__set_option_by_keyword.robot renamed to atest/robotMBT tests/07__processor_options/01__option_handling/02__set_option_by_keyword.robot

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0}
44
... AND Treat this test suite Model-based
55
Suite Teardown Should be equal ${test_count} ${2}
66
Library suiterepeater.py
7-
Library robotmbt processor_lib=suiterepeater
7+
Library robotmbt processor=suiterepeater
88

99
*** Test Cases ***
1010
only test case

atest/robotMBT tests/07__processor_options/option_handling/03__update_option_by_keyword.robot renamed to atest/robotMBT tests/07__processor_options/01__option_handling/03__update_option_by_keyword.robot

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0}
55
... AND Treat this test suite Model-based
66
Suite Teardown Should be equal ${test_count} ${3}
77
Library suiterepeater.py
8-
Library robotmbt processor_lib=suiterepeater
8+
Library robotmbt processor=suiterepeater
99

1010
*** Test Cases ***
1111
only test case

atest/robotMBT tests/07__processor_options/option_handling/04__update_option_at_trigger.robot renamed to atest/robotMBT tests/07__processor_options/01__option_handling/04__update_option_at_trigger.robot

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0}
44
... AND Treat this test suite Model-based repeat=3
55
Suite Teardown Should be equal ${test_count} ${3}
66
Library suiterepeater.py
7-
Library robotmbt processor_lib=suiterepeater
7+
Library robotmbt processor=suiterepeater
88

99
*** Test Cases ***
1010
only test case

atest/robotMBT tests/07__processor_options/option_handling/05__use_update_without_setter.robot renamed to atest/robotMBT tests/07__processor_options/01__option_handling/05__use_update_without_setter.robot

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0}
44
... AND Treat this test suite Model-based
55
Suite Teardown Should be equal ${test_count} ${2}
66
Library suiterepeater.py
7-
Library robotmbt processor_lib=suiterepeater
7+
Library robotmbt processor=suiterepeater
88

99
*** Test Cases ***
1010
only test case

atest/robotMBT tests/07__processor_options/option_handling/06__pass_multiple_options.robot renamed to atest/robotMBT tests/07__processor_options/01__option_handling/06__pass_multiple_options.robot

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ Suite Setup Run keywords Set suite variable ${test_count} ${0}
33
... AND Treat this test suite Model-based repeat=2 bonus_scenario=${True}
44
Suite Teardown Should be equal ${test_count} ${3}
55
Library suiterepeater.py
6-
Library robotmbt processor_lib=suiterepeater
6+
Library robotmbt processor=suiterepeater
77

88
*** Test Cases ***
99
only test case

0 commit comments

Comments
 (0)