Verification: Update test specification details

Enhance verification process to include pending parts of 8-9.4.2.2 and 8-9.4.3.3

Closes #566

Signed-off-by: Philipp Ahmann <philipp.ahmann@de.bosch.com>

Author: pahmann

process/process_areas/verification/guidance/verification_guideline.rst

@@ -99,9 +99,14 @@ In order to check the test results for the impact of a change or addition, it is
 execute affected test cases locally upfront using the execution framework of the build tooling
 following basically the steps the CI does locally.
 
+Tests have to be identifiable by a unique identifier, as described in :need:`gd_guidl__verification_specification`.
+They need need to have a clear passed or failed result and a documented configuration to enable proper evaluation of the result.
+
 Automated tests can also be executed locally, as the sources and binaries are available for re-execution.
 Failing test cases during re-execution can be reported following the guide :need:`gd_temp__problem_template`.
 
+During a relase, for any non executed test case, the reason for non-execution has to be documented in the
+:need:`wp__platform_sw_release_note` or :need:`wp__module_sw_release_plan` depending on the level (unit to platform) of the test case.
 
 Execution of manual test cases
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -112,20 +117,43 @@ test cases and reports the result in the same logging format as automated tests
 This enables parsable execution logs which can be used in automated test result collection.
 Failing test cases during manual execution are reported by following the guide :need:`gd_temp__problem_template`.
 
+Also manual test cases are written following the :need:`gd_guidl__verification_specification` and have a unique identifier.
+
+A tag `manual` is suggested to mark the test case as manual, so that it can be filtered out from automated test execution.
+
+In case a manual test case is not executed for a release, the reason for non-execution has to be documented in the
+:need:`wp__platform_sw_release_note` or :need:`wp__module_sw_release_plan` depending on the level of the test case.
+
 Reporting of failing test cases
 -------------------------------
 
 Any failing test case requires an ISSUE.
+
 The passing rate of safety-critical test cases need to be 100% in order to release the affected component.
-In case of a lower pass rate than 100% for QM level tests, the :need:`rl__project_lead` and
-:need:`rl__project_lead` can decide, if the platform is in a releasable state. The accepted minimal
-path rate is defined in the :need:`wp__verification_plan`. Due to the high degree of automation, a
-it is recommended that a path rate lower 95% is not acceptable.
+In case of a lower pass rate than 100% for QM level tests, the :need:`rl__project_lead`
+can decide, if the platform is in a releasable state. The accepted minimal pass rate is defined
+in the :need:`wp__verification_plan`. Due to the high degree of automation, it is recommended
+that a pass rate lower 95% is not acceptable. This percentage may increase with the maturity of
+the overall platform and the test coverage.
 
-In case an existing test case is failing due to regression in the CI, the respective issuer of the
+In case an existing test case is failing due to regression in the CI, the respective issue of the
 PR in their role as :need:`rl__contributor` is responsible for fixing the test case as part of
 respective PR.
 
+Test case results are also documented in the :need:`wp__verification_platform_ver_report` and
+:need:`wp__verification_module_ver_report`.
+
+Reporting of not executed or skipped test cases
+-----------------------------------------------
+
+In case a test case is not executed or skipped, a rational has to be provided in the release notes
+in case they are link against a requirement. A skipped or not executed test case is not counting into
+the value for the requirements coverage, as only executed and passed test cases can be counted for the
+coverage of a requirement.
+
+Skipped or not executed test cases are also documented in the :need:`wp__verification_platform_ver_report` and
+:need:`wp__verification_module_ver_report`.
+
 Reuse of existing test cases
 ----------------------------
 

process/process_areas/verification/guidance/verification_process_reqs.rst

@@ -146,6 +146,7 @@ Process Requirements
     The tool automation shall automatically generate the Verification reports.
     These may be independent documents (i.e. not integrated into docs-as-code based repositories).
     The content of the reports is specified in :need:`gd_temp__platform_ver_report` and :need:`gd_temp__mod_ver_report`.
+    The execution results of a test cases are marked with a clear passed of failed result.
 
 .. gd_req:: Verification Report Archiving
     :id: gd_req__verification_report_archiving
@@ -174,9 +175,9 @@ Process Requirements
     - TestType and DerivationTechnique shall be set
     - Description shall not be empty
     - In a Platform Integration Test Partially/FullyVerifies shall be set to a Platform Requirement
-    - If Partially/FullyVerifies are set in Feature Integration Test these shall link to Feature Requirements
-    - If Partially/FullyVerifies are set in Component Integration Test these shall link to Component Requirements
-    - If Partially/FullyVerifies are set in Unit Test these shall link to Component Requirements
+    - If Partially/FullyVerifies are set in Feature Integration Test these shall link to at least one Feature Requirement
+    - If Partially/FullyVerifies are set in Component Integration Test these shall link to at least one Component Requirement
+    - If Partially/FullyVerifies are set in Unit Test these shall link to at least one Component Requirement
 
 .. gd_req:: Verification Documentation Checks Extended
     :id: gd_req__verification_checks_extended

process/process_areas/verification/guidance/verification_specification.rst

@@ -55,8 +55,9 @@ A test specification contains the following attributes.
 
        - The objective of the test.
        - Inputs
-       - Expected outcome (e.g. "A success message is displayed.")
+       - Expected outcome/output/result (e.g. "A success message is displayed." or "Result should be 42.")
        - Test environment (e.g. network configuration, clean system state)
+       - Expected time sequence of events and behavior (where applicable and an expectation)
      -
    * - TestType
      - Examples are:
@@ -82,6 +83,14 @@ A test specification contains the following attributes.
 
 The implementation of :need:`wp__verification_plan` defines the full list of allowed types and methods.
 
+It is assumed that tests will be written as code (also for manual tests, which are script based)
+and each test case will have a unique identifier, by its script, execution call, or function name.
+The call used to execute the test marks the uniqueness of the test case and its identification,
+e.g. guaranteeing proper traceability and reproducibility.
+
+As the tests are stored in a repository close to the implementation code, versioning is done by the versioning of the repository.
+
+Any specification and resulting implementation ends with a clear passed or failed result.
 
 Test description
 ----------------

process/process_areas/verification/verification_concept.rst

@@ -124,7 +124,6 @@ Usually the defined methods are not applied on each verification level between u
 Also their execution may differ whether it is a QM or ASIL rated test case.
 The rigor is described in the implementation of :need:`wp__verification_plan`.
 
-
 Automated test cases (as well as manual, where applicable) shall contain further information about which methods have been applied.
 The corresponding guidance is given here: :need:`gd_guidl__verification_guide`.
 The identifier of the respective method is to be used as meta data (*TestType* and *DerivationTechnique*).
@@ -161,17 +160,20 @@ tests or component tests. This linking is done using metatags. This is also true
 integration tests linking to the component requirements and architecture.
 
 Traceability of feature integration tests shall be established through linking those test cases to
-feature requirements and architecture as features describe the integrated behaviour of all components.
+feature requirements and architecture as features describe the integrated behavior of all components.
 
 Traceability of platform integration tests shall be established through linking those test cases to
-stakeholder requirements as stakeholder requirements describe the platform behaviour.
+stakeholder requirements as stakeholder requirements describe the platform behavior.
 
 Note that all the above tests shall only link to requirements of type "Functional" and "Interface".
 The verification of requirements of types "Process" and "Non-Functional" will be done via Analysis,
 which is part of the requirement inspection :need:`doc__feature_name_req_inspection` and `Component Requirements Inspection Checklist <https://eclipse-score.github.io/module_template/main/score/component_example/docs/requirements/chklst_req_inspection.html>`__.
 Requirements always include Assumptions Of Use.
 
-A more detailed description of how to link code to requirements is available here: :need:`gd_req__verification_link_tests`
+A more detailed description of how to link code to requirements is available by :need:`gd_req__verification_link_tests`.
+
+Another element of traceability for a proper backlink is the unique identification of test cases as described in the
+:need:`gd_guidl__verification_specification`.
 
 Workflow for Verification Guidance
 ----------------------------------

process/process_areas/verification/verification_getstrt.rst

@@ -41,8 +41,8 @@ General Workflow
 
 The workflows can be split into 4 major parts:
 
-* Test planning filling the template :need:`gd_temp__verification_plan`.
-* Test specification and implementation for the respective testing level
+* Test planning by filling the template :need:`gd_temp__verification_plan`.
+* Test specification and implementation for the respective testing level (unit to platform).
 * Test execution by the CI.
   (Manual test cases are treated as automated test with user interaction and timeouts.)
 * Test reports are created when all verification artifacts on a module and platform level are