Merge pull request #47: Algorithm update - Combine targeted exploration with randomisation

Author: JFoederer

.github/workflows/run-tests.yml

@@ -25,14 +25,13 @@ jobs:
     - name: Install dependencies (without extra visualisation dependencies)
       run: |
         python -m pip install --upgrade pip           # upgrade pip to latest version
-        pip install .                                 # no additional [visualisation] dependencies
+        pip install .                                 # using the minimum installation
 
     - name: Run tests (without visualisation dependencies)
       run:  python run_tests.py
 
     - name: Install extra visualisation dependencies
-      run:  pip install ".[visualisation]"            # extra [visualisation] dependencies in pyproject.toml
+      run:  pip install .[full]
 
     - name: Run Tests (with visualisation dependencies)
       run:  python run_tests.py
-

README.md

@@ -24,24 +24,28 @@ RobotMBT offers features to cover both _when_ and _what_ variations.
 
 RobotMBT is suitable for sequencing complete scenarios, including action refinement for when-steps. Concrete example scenarios can be generalised for added data-driven variation. When all steps are properly annotated with modelling info, the library can resolve their dependencies and figure out the correct execution order. Each run a new test sequence is generated from the available options.
 
-To be successful, the set of scenarios in the model must (for now) be composable into a single complete sequence, without leftovers. The same scenario can be inserted multiple times if repetition helps to reach the entry condition for later scenarios.
+To be successful, the set of scenarios in the model must (for now) be composable into a single complete sequence, without leftovers. The same scenario can be inserted into the trace multiple times, creating loops, if repetition helps to reach the entry condition for later scenarios. Dead ends should be prevented, i.e., sequences from which there is no way forward and no way to loop back.
 
 ## Getting started
 
-To get a feel for what this library can do, have a look at our [Titanic themed demo](https://github.com/JFoederer/robotframeworkMBT/tree/main/demo/Titanic), which is executable as a [Robot framework](https://robotframework.org/) test suite.
-
 The recommended installation method is using [pip](http://pip-installer.org)
 
+    pip install --upgrade robotframework-mbt[full]
+
+The full installation includes additional graphical dependencies to help during model creation. These are optional. The minimum installation includes everything you need to create and run model-based test suites.
+
     pip install --upgrade robotframework-mbt
 
-After installation include `robotmbt` as library in your robot file to get access to the new functionality. To run your test suite model-based, use the __Treat this test suite model-based__ keyword as suite setup. Check the _How to model_ section to learn how to make your scenarios suitable for running model-based.
+After installation, include `robotmbt` as library in your robot file to get access to the new functionality. To run your test suite model-based, use the __Treat this test suite model-based__ keyword as suite setup. Check the _How to model_ section to learn how to make your scenarios suitable for running model-based.
 
 ```robotframework
 *** Settings ***
 Library        robotmbt
 Suite Setup    Treat this test suite model-based
 ```
 
+To get a feel for what this library can do, have a look at our [Titanic themed demo](https://github.com/JFoederer/robotframeworkMBT/tree/main/demo/Titanic), which is executable as a [Robot framework](https://robotframework.org/) test suite.
+
 ## How to model
 
 Modelling can be done directly from [Robot framework](https://robotframework.org/), without the need for additional tooling besides [RobotMBT](https://github.com/JFoederer/robotframeworkMBT).
@@ -211,14 +215,14 @@ Using `seed=new` will force generation of a new reusable seed and is identical t
 
 A graph can be included in the log file to visualise how scenarios are linked. This helps in understanding a test suite's structure and reveals alternative paths that did not make it into the final trace.
 
-To enable graph generation, some extra dependencies must be installed: `pip install robotframework-mbt[visualisation]`
-
 Generate the graph by setting the graph style for the model-based suite. The graph will be included in the Robot log file as part of the keyword's logging.
 
 ```robotframework
 Treat this test suite Model-based  graph=scenario
 ```
 
+_Note_: The graphing functionality requires the _full_ installation package: `pip install robotframework-mbt[full]`
+
 Available graph styles:
 
 * scenario

atest/robotMBT tests/07__processor_options/random_seeds/02__reusing_seed_reproduces_trace.robot

@@ -5,7 +5,7 @@ Documentation     This suite uses the `seed` argument to reproduce a previously
 ...               indicated by the number string.
 Suite Setup       Run keywords    Set suite variable    ${trace}    ${empty}
 ...                        AND    Treat this test suite Model-based    seed=aqmou-eelcuu-sniu-ugsyek-jyhoor
-Suite Teardown    Should be equal    ${trace}    6930142758
+Suite Teardown    Should be equal    ${trace}    6781950324
 Library           robotmbt
 
 *** Test Cases ***

atest/robotMBT tests/07__processor_options/random_seeds/03__retrace_with_refinement.robot

@@ -9,7 +9,7 @@ Documentation     This suite uses the `seed` argument to reproduce a previously
 ...               traces possible of varying length.
 Suite Setup       Run keywords    Set suite variable    ${trace}    ${empty}
 ...                        AND    Treat this test suite Model-based    seed=xou-pumj-ihj-oibiyc-surer
-Suite Teardown    Should be equal    ${trace}    B1A5BY3B4BX6B2
+Suite Teardown    Should be equal    ${trace}    AY3B2B4B5BX1A6
 Library           robotmbt
 
 *** Test Cases ***

atest/robotMBT tests/07__processor_options/random_seeds/04__retrace_with_step_modifiers.robot

@@ -6,7 +6,7 @@ Documentation     This suite uses the `seed` argument to reproduce a previously
 ...               included, the modifers will number the scenarios at random. The reproduced trace
 ...               must match for both the scenario order and the data order.
 Suite Setup       Treat this test suite Model-based    seed=iulr-vih-esycu-eyl-yfa
-Suite Teardown    Should be equal    ${trace}    H6G3E5I4D9F8J2B1A7C0
+Suite Teardown    Should be equal    ${trace}    H0J3G7F9E5C8A1I2D4B6
 Library           robotmbt
 
 *** Test Cases ***

atest/robotMBT tests/07__processor_options/random_seeds/05__retrace_combined.robot

@@ -14,9 +14,9 @@ Documentation     This suite uses the `seed` argument to reproduce a previously
 ...               The looping part is enhanced with refinement. The high-level scenario includes
 ...               a data choice by using step modifiers. The lower level includes a path choice.
 ...               Both low-level scenarios are equally valid, the only difference is that the data
-...               choice is included either once or twice.
-Suite Setup       Treat this test suite Model-based    seed=gujuqt-iakm-oexo-xnu-huba
-Suite Teardown    Should be equal    ${trace}    ATQQPRSPRPXY
+...               choice is included either once (single letter) or twice (letter repeated).
+Suite Setup       Treat this test suite Model-based    seed=efyuti-oan-naimfe-noelua-ler
+Suite Teardown    Should be equal    ${trace}    ATTSPRRPPXY
 Library           robotmbt
 
 *** Test Cases ***

pyproject.toml

@@ -7,7 +7,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "robotframework-mbt"
-version = "0.11.0"
+version = "0.12.0"
 description = "Model-Based Testing in Robot framework with test case generation"
 readme = "README.md"
 authors = [{ name = "Johan Foederer", email = "github@famfoe.nl" }]
@@ -30,4 +30,4 @@ include = ["robotmbt*"]
 Homepage = "https://github.com/JFoederer/robotframeworkMBT"
 
 [project.optional-dependencies]
-visualisation = ["networkx", "bokeh", "grandalf", "jsonpickle"]
+full = ["networkx", "bokeh", "grandalf", "jsonpickle"]

robotmbt/modeller.py

@@ -57,11 +57,11 @@ def try_to_fit_in_scenario(candidate: Scenario, tracestate: TraceState):
     elif not remainder:  # the scenario processed in full
         model.end_scenario_scope()
         tracestate.confirm_full_scenario(inserted.src_id, inserted, model)
-        logger.debug(f"Inserted scenario {inserted.src_id}, {inserted.name}")
+        logger.debug(f"Scenario {inserted.src_id} inserted: {inserted.name}")
         if tracestate.is_refinement_active():
             handle_refinement_exit(inserted, tracestate)
     else:  # the scenario is split into two parts, ready for refinement
-        logger.debug(f"Partially inserted scenario {inserted.src_id}, {inserted.name}\n"
+        logger.debug(f"Scenario {inserted.src_id} partially inserted: {inserted.name}\n"
                      f"Refinement needed at step: {remainder.steps[1]}")
         inserted.name = f"{inserted.name} (part {tracestate.highest_part(inserted.src_id)+1})"
         tracestate.push_partial_scenario(inserted.src_id, inserted, model, remainder)
@@ -79,12 +79,12 @@ def process_scenario(scenario: Scenario, model: ModelSpace) -> tuple[Scenario, S
                         part1, part2 = split_for_refinement(scenario, step)
                         return part1, part2, dict()
                     else:
-                        return None, None, dict(fail_msg=f"Unable to insert scenario {scenario.src_id}, "
+                        return None, None, dict(fail_msg=f"Rejecting scenario {scenario.src_id}, "
                                                 f"{scenario.name}, due to step '{step}': [{expr}] is False")
             except TimeoutExceeded:
                 raise
             except Exception as err:
-                return None, None, dict(fail_msg=f"Unable to insert scenario {scenario.src_id}, "
+                return None, None, dict(fail_msg=f"Rejecting scenario {scenario.src_id}, "
                                         f"{scenario.name}, due to step '{step}': [{expr}] {err}")
     return scenario.copy(), None, dict()
 
@@ -141,8 +141,8 @@ def handle_refinement_exit(inserted_refinement: Scenario, tracestate: TraceState
 
     if not exit_conditions_processed:
         rewind(tracestate)  # Reject insterted scenario. Even though it fits, it is not a refinement.
-        logger.debug(f"Reconsidering scenario {inserted_refinement.src_id}, {inserted_refinement.name}, "
-                     f"did not meet refinement exit condition: {exit_conditions}")
+        logger.debug(f"Reconsidering scenario {inserted_refinement.src_id}, "
+                     f"refinement exit condition not met for scenario {refinement_tail.src_id}: {exit_conditions}")
         return
 
     model = tracestate.model
@@ -157,7 +157,7 @@ def handle_refinement_exit(inserted_refinement: Scenario, tracestate: TraceState
     elif not remainder:
         model.end_scenario_scope()
         tracestate.confirm_full_scenario(tail_inserted.src_id, tail_inserted, model)
-        logger.debug(f"Scenario '{tail_inserted.name}' completed after refinement")
+        logger.debug(f"Scenario {tail_inserted.src_id} completed after refinement: {tail_inserted.name}")
         if tracestate.is_refinement_active():
             handle_refinement_exit(tail_inserted, tracestate)
     else:
@@ -219,20 +219,20 @@ def generate_scenario_variant(scenario: Scenario, model: ModelSpace) -> Scenario
     except TimeoutExceeded:
         raise
     except Exception as err:
-        logger.debug(f"Unable to insert scenario {scenario.src_id}, {scenario.name}, due to modifier\n"
+        logger.debug(f"Rejecting scenario {scenario.src_id}, {scenario.name}, due to modifier\n"
                      f"    In step {step}: {err}")
         return None
 
     try:
         subs.solve()
     except ValueError as err:
-        logger.debug(f"Unable to insert scenario {scenario.src_id}, {scenario.name}, due to modifier\n"
+        logger.debug(f"Rejecting scenario {scenario.src_id}, {scenario.name}, due to modifier\n"
                      f"    {err}: {subs}")
         return None
 
     # Update scenario with generated values
     if subs.solution:
-        logger.debug(f"Example variant generated with argument substitution: {subs}")
+        logger.debug(f"Example variant generated with argument substitution (↦ replaced by): {subs}")
     scenario.data_choices = subs
     for step in scenario.steps:
         if 'MOD' in step.model_info:

robotmbt/substitutionmap.py

@@ -49,7 +49,7 @@ def __init__(self):
 
     def __str__(self):
         src = self.solution or self.substitutions
-        return ", ".join([f"{k} ⤝ {v}" for k, v in src.items()])
+        return ", ".join([f"{k} ↦ {v}" for k, v in src.items()])
 
     def copy(self):
         new = SubstitutionMap()