Merge pull request #33: Rerun previously generated trace

JFoederer · web-flow · commit 33c0086ac09a · 2025-05-30T11:35:19.000+02:00
Test trace generation is randomised by default. But what if you hit this unique case no one thought of before? Or that special combination that reveals a defect? Then you want to have the option to rerun that specific trace. Version 0.7 introduces a `seed` argument that lets you reproduce a previously generated trace.
diff --git a/README.md b/README.md
@@ -183,6 +183,18 @@ In a then-step, modifiers behave slightly different. In then-steps no new option
 
 This first implementation for variable data considers strict equivalence classes only. This means that all variants are considered equal for all purposes. If, for a certain scenario, a single valid example variant has been generated and executed, then this scenario is considered covered. There are no options yet to indicate deeper coverage targets based on data variations. It also implies that whenever any variant is valid, all scenario variants must be valid. And that regardless of which variant is chosen, the exact same scenarios can be chosen as the next one. This does however not mean that once a variant is chosen, that this variant will be used throughout the whole trace. If a scenario is selected multiple times in the same trace, then each occurrence will get new randomly selected data.
 
+## Configuration options
+
+### Random seed
+
+By default, trace generation is random. The random seed used for the trace is logged (debug level) by _Treat this test suite model-based_. This seed can be used to rerun the same trace, if no external random factors influence the test run. To activate the seed, pass it as argument:
+
+```
+Treat this test suite model-based    seed=eag-etou-cxi-leamv-jsi
+```
+
+Using `seed=new` will force generation of a new reusable seed and is identical to omitting the seed argument. To completely bypass seed generation and use the system's random source, use `seed=None`. This has even more variation but does not produce a reusable seed.
+
 ## Disclaimer
 
 Please note that this library is in a premature state and hasn't reached its first official (1.0) release yet. Developments are ongoing within the context of the [TiCToC](https://tictoc.cs.ru.nl/) research project. Interface changes are still frequent, and no deprecation warnings are being issued yet.
diff --git a/atest/robotMBT tests/07__processor_options/option_handling/01__pass_option_directly.robot b/atest/robotMBT tests/07__processor_options/option_handling/01__pass_option_directly.robot
@@ -2,8 +2,8 @@
 Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based    repeat=2
 Suite Teardown    Should be equal    ${test_count}    ${2}
-Library           SuiteRepeater.py
-Library           robotmbt    processor_lib=SuiteRepeater
+Library           suiterepeater.py
+Library           robotmbt    processor_lib=suiterepeater
 
 *** Test Cases ***
 only test case
diff --git a/atest/robotMBT tests/07__processor_options/option_handling/02__set_option_by_keyword.robot b/atest/robotMBT tests/07__processor_options/option_handling/02__set_option_by_keyword.robot
@@ -3,8 +3,8 @@ Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Set model-based options    repeat=2
 ...                        AND    Treat this test suite Model-based
 Suite Teardown    Should be equal    ${test_count}    ${2}
-Library           SuiteRepeater.py
-Library           robotmbt    processor_lib=SuiteRepeater
+Library           suiterepeater.py
+Library           robotmbt    processor_lib=suiterepeater
 
 *** Test Cases ***
 only test case
diff --git a/atest/robotMBT tests/07__processor_options/option_handling/03__update_option_by_keyword.robot b/atest/robotMBT tests/07__processor_options/option_handling/03__update_option_by_keyword.robot
@@ -4,8 +4,8 @@ Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Update model-based options    repeat=3
 ...                        AND    Treat this test suite Model-based
 Suite Teardown    Should be equal    ${test_count}    ${3}
-Library           SuiteRepeater.py
-Library           robotmbt    processor_lib=SuiteRepeater
+Library           suiterepeater.py
+Library           robotmbt    processor_lib=suiterepeater
 
 *** Test Cases ***
 only test case
diff --git a/atest/robotMBT tests/07__processor_options/option_handling/04__update_option_at_trigger.robot b/atest/robotMBT tests/07__processor_options/option_handling/04__update_option_at_trigger.robot
@@ -3,8 +3,8 @@ Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Set model-based options    repeat=2
 ...                        AND    Treat this test suite Model-based    repeat=3
 Suite Teardown    Should be equal    ${test_count}    ${3}
-Library           SuiteRepeater.py
-Library           robotmbt    processor_lib=SuiteRepeater
+Library           suiterepeater.py
+Library           robotmbt    processor_lib=suiterepeater
 
 *** Test Cases ***
 only test case
diff --git a/atest/robotMBT tests/07__processor_options/option_handling/05__use_update_without_setter.robot b/atest/robotMBT tests/07__processor_options/option_handling/05__use_update_without_setter.robot
@@ -3,8 +3,8 @@ Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Update model-based options    repeat=2
 ...                        AND    Treat this test suite Model-based
 Suite Teardown    Should be equal    ${test_count}    ${2}
-Library           SuiteRepeater.py
-Library           robotmbt    processor_lib=SuiteRepeater
+Library           suiterepeater.py
+Library           robotmbt    processor_lib=suiterepeater
 
 *** Test Cases ***
 only test case
diff --git a/atest/robotMBT tests/07__processor_options/option_handling/06__pass_multiple_options.robot b/atest/robotMBT tests/07__processor_options/option_handling/06__pass_multiple_options.robot
@@ -2,8 +2,8 @@
 Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based    repeat=2    bonus_scenario=${True}
 Suite Teardown    Should be equal    ${test_count}    ${3}
-Library           SuiteRepeater.py
-Library           robotmbt    processor_lib=SuiteRepeater
+Library           suiterepeater.py
+Library           robotmbt    processor_lib=suiterepeater
 
 *** Test Cases ***
 only test case
diff --git a/atest/robotMBT tests/07__processor_options/option_handling/07__set_clears_other_options.robot b/atest/robotMBT tests/07__processor_options/option_handling/07__set_clears_other_options.robot
@@ -4,8 +4,8 @@ Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Set model-based options    repeat=2
 ...                        AND    Treat this test suite Model-based
 Suite Teardown    Should be equal    ${test_count}    ${2}
-Library           SuiteRepeater.py
-Library           robotmbt    processor_lib=SuiteRepeater
+Library           suiterepeater.py
+Library           robotmbt    processor_lib=suiterepeater
 
 *** Test Cases ***
 only test case
diff --git a/atest/robotMBT tests/07__processor_options/option_handling/08__empty_setter_clears_all_options.robot b/atest/robotMBT tests/07__processor_options/option_handling/08__empty_setter_clears_all_options.robot
@@ -4,8 +4,8 @@ Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Set model-based options
 ...                        AND    Treat this test suite Model-based
 Suite Teardown    Should be equal    ${test_count}    ${1}
-Library           SuiteRepeater.py
-Library           robotmbt    processor_lib=SuiteRepeater
+Library           suiterepeater.py
+Library           robotmbt    processor_lib=suiterepeater
 
 *** Test Cases ***
 only test case
diff --git a/atest/robotMBT tests/07__processor_options/option_handling/09__multiple_options_from_dict.robot b/atest/robotMBT tests/07__processor_options/option_handling/09__multiple_options_from_dict.robot
@@ -2,8 +2,8 @@
 Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based    &{mbt_options}
 Suite Teardown    Should be equal    ${test_count}    ${3}
-Library           SuiteRepeater.py
-Library           robotmbt    processor_lib=SuiteRepeater
+Library           suiterepeater.py
+Library           robotmbt    processor_lib=suiterepeater
 
 *** Variables ***
 &{mbt_options}    repeat=2    bonus_scenario=${True}
diff --git a/atest/robotMBT tests/07__processor_options/option_handling/10__partial_option_update.robot b/atest/robotMBT tests/07__processor_options/option_handling/10__partial_option_update.robot
@@ -3,8 +3,8 @@ Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Set model-based options    &{mbt_options}
 ...                        AND    Treat this test suite Model-based    bonus_scenario=${False}
 Suite Teardown    Should be equal    ${test_count}    ${2}
-Library           SuiteRepeater.py
-Library           robotmbt    processor_lib=SuiteRepeater
+Library           suiterepeater.py
+Library           robotmbt    processor_lib=suiterepeater
 
 *** Variables ***
 &{mbt_options}    repeat=2    bonus_scenario=${True}
diff --git a/atest/robotMBT tests/07__processor_options/option_handling/11__argument_restrictions.robot b/atest/robotMBT tests/07__processor_options/option_handling/11__argument_restrictions.robot
@@ -1,6 +1,6 @@
 *** Settings ***
-Library           SuiteRepeater.py
-Library           robotmbt    processor_lib=SuiteRepeater    processor=mandatory_repeat_argument
+Library           suiterepeater.py
+Library           robotmbt    processor_lib=suiterepeater    processor=mandatory_repeat_argument
 
 *** Test Cases ***
 arguments can be mandatory
diff --git a/atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/__init__.robot b/atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/__init__.robot
@@ -3,4 +3,4 @@ Documentation     In this suite one of the processor options is set on the highe
 ...               which is then reused in both sub suites. Each sub suite adds their own value
 ...               for a second configuration option.
 Suite Setup       Set model-based options    repeat=2
-Library           robotmbt    processor_lib=SuiteRepeater
+Library           robotmbt    processor_lib=suiterepeater
diff --git a/atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/with_bonus_scenario.robot b/atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/with_bonus_scenario.robot
@@ -2,7 +2,7 @@
 Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based    bonus_scenario=${True}
 Suite Teardown    Should be equal    ${test_count}    ${3}
-Library           robotmbt    processor_lib=SuiteRepeater
+Library           robotmbt    processor_lib=suiterepeater
 
 *** Test Cases ***
 only test case
diff --git a/atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/without_bonus_scenario.robot b/atest/robotMBT tests/07__processor_options/option_handling/12__settings_can_span_multiple_suites/without_bonus_scenario.robot
@@ -2,7 +2,7 @@
 Suite Setup       Run keywords    Set suite variable    ${test_count}    ${0}
 ...                        AND    Treat this test suite Model-based    bonus_scenario=${False}
 Suite Teardown    Should be equal    ${test_count}    ${2}
-Library           robotmbt    processor_lib=SuiteRepeater
+Library           robotmbt    processor_lib=suiterepeater
 
 *** Test Cases ***
 only test case
diff --git a/atest/robotMBT tests/07__processor_options/option_handling/suiterepeater.py b/atest/robotMBT tests/07__processor_options/option_handling/suiterepeater.py
@@ -1,5 +1,8 @@
 import copy
 
+from robot.api.deco import library
+
+@library(auto_keywords=None, listener=True)
 class SuiteRepeater:
     """
     Given a test suite, repeats all scenarios 'repeat' times (default=1)
@@ -15,7 +18,7 @@ def process_test_suite(self, in_suite, repeat=1, **kwargs):
         for i in range(len(out_suite.scenarios)):
             out_suite.scenarios[i] = out_suite.scenarios[i].copy()
             if i:
-                out_suite.scenarios[i].name += f" (rep {i})"
+                out_suite.scenarios[i].name += f" (rep {i+1})"
         return out_suite
 
     def mandatory_repeat_argument(self, in_suite, *, repeat, bonus_scenario=False):
diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/__init__.robot b/atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/__init__.robot
@@ -0,0 +1,18 @@
+*** Settings ***
+Documentation     In this suite a set of sub-suites are executed using trace generation. Their
+...               trace generation should be independent and each yield a random trace. This is
+...               verified by checking that all traces are unique. The traces consist of 10
+...               indepent scenarios, giving over 3.500.000 options. This makes the  risk of
+...               hitting the same trace by chance acceptable.
+Suite Setup       Reset traces
+Suite Teardown    All traces should be different
+Library           robotmbt
+Library           traces.py
+
+*** Keywords ***
+All traces should be different
+    @{traces}=    Get all traces
+    FOR    ${trace}    IN    @{traces}
+        ${count}=    Get count    ${traces}    ${trace}
+        Should be equal    ${count}    ${1}
+    END
diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/no_seed.robot b/atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/no_seed.robot
@@ -0,0 +1,35 @@
+*** Settings ***
+Suite Setup       Treat this test suite Model-based
+Library           robotmbt
+Library           traces.py
+
+*** Test Cases ***
+scenario 0
+    Trace 'no seed', scenario number 0 is executed
+
+scenario 1
+    Trace 'no seed', scenario number 1 is executed
+
+scenario 2
+    Trace 'no seed', scenario number 2 is executed
+
+scenario 3
+    Trace 'no seed', scenario number 3 is executed
+
+scenario 4
+    Trace 'no seed', scenario number 4 is executed
+
+scenario 5
+    Trace 'no seed', scenario number 5 is executed
+
+scenario 6
+    Trace 'no seed', scenario number 6 is executed
+
+scenario 7
+    Trace 'no seed', scenario number 7 is executed
+
+scenario 8
+    Trace 'no seed', scenario number 8 is executed
+
+scenario 9
+    Trace 'no seed', scenario number 9 is executed
diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/seed_new.robot b/atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/seed_new.robot
@@ -0,0 +1,35 @@
+*** Settings ***
+Suite Setup       Treat this test suite Model-based    seed=new
+Library           robotmbt
+Library           traces.py
+
+*** Test Cases ***
+scenario 0
+    Trace 'seed=new', scenario number 0 is executed
+
+scenario 1
+    Trace 'seed=new', scenario number 1 is executed
+
+scenario 2
+    Trace 'seed=new', scenario number 2 is executed
+
+scenario 3
+    Trace 'seed=new', scenario number 3 is executed
+
+scenario 4
+    Trace 'seed=new', scenario number 4 is executed
+
+scenario 5
+    Trace 'seed=new', scenario number 5 is executed
+
+scenario 6
+    Trace 'seed=new', scenario number 6 is executed
+
+scenario 7
+    Trace 'seed=new', scenario number 7 is executed
+
+scenario 8
+    Trace 'seed=new', scenario number 8 is executed
+
+scenario 9
+    Trace 'seed=new', scenario number 9 is executed
diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/seed_none.robot b/atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/seed_none.robot
@@ -0,0 +1,35 @@
+*** Settings ***
+Suite Setup       Treat this test suite Model-based    seed=${None}
+Library           robotmbt
+Library           traces.py
+
+*** Test Cases ***
+scenario 0
+    Trace 'seed=None', scenario number 0 is executed
+
+scenario 1
+    Trace 'seed=None', scenario number 1 is executed
+
+scenario 2
+    Trace 'seed=None', scenario number 2 is executed
+
+scenario 3
+    Trace 'seed=None', scenario number 3 is executed
+
+scenario 4
+    Trace 'seed=None', scenario number 4 is executed
+
+scenario 5
+    Trace 'seed=None', scenario number 5 is executed
+
+scenario 6
+    Trace 'seed=None', scenario number 6 is executed
+
+scenario 7
+    Trace 'seed=None', scenario number 7 is executed
+
+scenario 8
+    Trace 'seed=None', scenario number 8 is executed
+
+scenario 9
+    Trace 'seed=None', scenario number 9 is executed
diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/traces.py b/atest/robotMBT tests/07__processor_options/random_seeds/01__generating_random_traces/traces.py
@@ -0,0 +1,20 @@
+from robot.api.deco import keyword
+
+class traces:
+    ROBOT_LIBRARY_SCOPE = 'GLOBAL'
+    def reset_traces(self):
+        self.traces = {}
+
+    @keyword("Trace '${n}', scenario number ${m} is executed")
+    def add_test(self, trace, test_id:str):
+        """*model info*
+        :IN:  None
+        :OUT: None
+        """
+        if trace in self.traces:
+            self.traces[trace] += test_id
+        else:
+            self.traces[trace] = test_id
+
+    def get_all_traces(self):
+        return self.traces.values()
diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/02__reusing_seed_reproduces_trace.robot b/atest/robotMBT tests/07__processor_options/random_seeds/02__reusing_seed_reproduces_trace.robot
@@ -0,0 +1,47 @@
+*** Settings ***
+Documentation     This suite uses the `seed` argument to reproduce a previously generated trace.
+...               The suite consists of 10 independent scenarios, giving 10! (>3.500.000) possible
+...               traces. The suite passes if the recorded trace is reproduced exactly, as
+...               indicated by the number string.
+Suite Setup       Run keywords    Set suite variable    ${trace}    ${empty}
+...                        AND    Treat this test suite Model-based    seed=aqmou-eelcuu-sniu-ugsyek-jyhoor
+Suite Teardown    Should be equal    ${trace}    6930142758
+Library           robotmbt
+
+*** Test Cases ***
+scenario 0
+    scenario number 0 is executed
+
+scenario 1
+    scenario number 1 is executed
+
+scenario 2
+    scenario number 2 is executed
+
+scenario 3
+    scenario number 3 is executed
+
+scenario 4
+    scenario number 4 is executed
+
+scenario 5
+    scenario number 5 is executed
+
+scenario 6
+    scenario number 6 is executed
+
+scenario 7
+    scenario number 7 is executed
+
+scenario 8
+    scenario number 8 is executed
+
+scenario 9
+    scenario number 9 is executed
+
+*** Keywords ***
+scenario number ${n} is executed
+    [Documentation]    *model info*
+    ...    :IN:  None
+    ...    :OUT: None
+    Set Suite Variable    ${trace}    ${trace}${n}
diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/03__retrace_with_refinement.robot b/atest/robotMBT tests/07__processor_options/random_seeds/03__retrace_with_refinement.robot
diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/04__retrace_with_step_modifiers.robot b/atest/robotMBT tests/07__processor_options/random_seeds/04__retrace_with_step_modifiers.robot
diff --git a/atest/robotMBT tests/07__processor_options/random_seeds/05__retrace_combined.robot b/atest/robotMBT tests/07__processor_options/random_seeds/05__retrace_combined.robot
diff --git a/demo/Titanic/Titanic_scenarios/step_defs.resource b/demo/Titanic/Titanic_scenarios/step_defs.resource
diff --git a/pyproject.toml b/pyproject.toml
diff --git a/robotmbt/substitutionmap.py b/robotmbt/substitutionmap.py
diff --git a/robotmbt/suiteprocessors.py b/robotmbt/suiteprocessors.py
diff --git a/robotmbt/suitereplacer.py b/robotmbt/suitereplacer.py
diff --git a/robotmbt/version.py b/robotmbt/version.py
diff --git a/utest/test_substitutionmap.py b/utest/test_substitutionmap.py
diff --git a/utest/test_suiteprocessors.py b/utest/test_suiteprocessors.py
