Docs: Improve XML documentation and logging (#584)

Author: MaximilianSoerenPollak

docs/how-to/test_to_doc_links.rst

@@ -20,17 +20,24 @@ Reference Docs in Tests
 This guide explains how to annotate test cases so that
 docs-as-code automatically creates traceability links between tests and requirements.
 
+Details of implementation
+-------------------------
+
 The mechanism is language-agnostic:
-Bazel produces JUnit-style ``test.xml`` files under ``bazel-testlogs/``.
-The ``score_source_code_linker`` extension parses those files, extracts test metadata
+The ``score_source_code_linker`` extension parses ``test.xml`` files, extracts test metadata
 (name, file, line, result, and verification properties),
 and creates backlinks on the referenced requirements.
 
+The extension will look for ``test.xml`` files in both ``bazel-testlogs/``
+as well as a folder named ``tests-report/``. This folder can be created manually if the tests
+require some pre-run step or are matrix tests or similar.
+
+
 Required Properties
 -------------------
 
 Every linked test must declare the following properties
-(see :need:`gd_guidl__verification_specification`):
+(see :need:`gd_guidl__verification_specification` for detailed values):
 
 ``PartiallyVerifies`` *and/or* ``FullyVerifies``
    Comma-separated list of requirement IDs that the test covers.
@@ -43,8 +50,44 @@ Every linked test must declare the following properties
 
 ``Description``
    A human-readable explanation of the test objective and expected outcome.
-
-If any mandatory property is missing the test will be skipped during link creation.
+   *In Python tests, a docstring takes the place of the description attribute.*
+
+What should a test.xml look like?
+---------------------------------
+
+Each testcase has file as well as it's line number as additional attributes.
+Then as described above, each testcase also has properties inside the XML.
+
+.. code-block:: xml
+
+   <?xml version="1.0" encoding="utf-8"?>
+   <testsuites name="pytest tests">
+     <testsuite errors="0" failures="0" hostname="LG-0005" name="pytest" skipped="0" tests="118" time="6.617" timestamp="2026-06-08T17:56:07.635773+00:00">
+       <testcase classname="src.extensions.score_source_code_linker.tests.test_testlink" file="src/extensions/score_source_code_linker/tests/test_testlink.py" line="40" name="test_testlink_serialization_roundtrip" time="0.000">
+         <properties>
+           <property name="PartiallyVerifies" value="tool_req__docs_test_link_testcase"></property>
+           <property name="TestType" value="requirements-based"></property>
+           <property name="DerivationTechnique" value="requirements-analysis"></property>
+         </properties>
+       </testcase>
+       <testcase classname="src.extensions.score_source_code_linker.tests.test_testlink" file="src/extensions/score_source_code_linker/tests/test_testlink.py" line="85" name="test_clean_text_removes_ansi_and_html_unescapes" time="0.000">
+         <properties>
+           <property name="PartiallyVerifies" value="tool_req__docs_test_link_testcase"></property>
+           <property name="TestType" value="requirements-based"></property>
+           <property name="DerivationTechnique" value="requirements-analysis"></property>
+         </properties>
+       </testcase>
+     </testsuite>
+   </testsuites>
+
+If you are not working in Python and using the provided pytest plugin, please ensure that the xml that is written in the end
+looks like this, otherwise the extension will not be able to parse the xml correctly.
+
+What happens when properties are missing
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When properties or attributes are missing for some or all tests inside the test.xml, testcases are still generated.
+However, testcases can only be linked to the requirements if either Partially- or FullyVerifies is filled.
 
 Language-Specific Annotations
 -----------------------------
@@ -58,15 +101,16 @@ C++ (gTest)
    See :need:`gd_req__verification_link_tests_cpp`.
 
 Rust
-   Use the ``#[record_property]`` attribute macro from the ``test_properties`` crate.
-   See :need:`gd_req__verification_link_tests_rust`.
+   Currently there is no provided official way to do this in Rust.
+   We are working with the rust community to figure this out.
 
 Python (pytest)
    Use the ``@add_test_properties`` decorator; the docstring serves as ``Description``.
    See :need:`gd_req__verification_link_tests_python`.
 
 See :need:`gd_temp__verification_specification` for code templates.
 
+
 Running Tests and Building Docs
 -------------------------------
 
@@ -84,9 +128,59 @@ Running Tests and Building Docs
 
 The resulting documentation shows backlinks on each requirement that is referenced by at least one test.
 
+
+How do I use the generated testcases?
+-------------------------------------
+
+Testcases can be used in different ways inside and outside your documentation.
+You can create lists or pie diagrams to showcase all tests as well as statistics on them.
+For example you can show all tests like so
+
+.. code-block:: rst
+
+   .. needpie:: Test Results
+      :labels: passed, failed, skipped
+      :colors: green, red, orange
+
+      type == 'testcase' and result == 'passed'
+      type == 'testcase' and result == 'failed'
+      type == 'testcase' and result == 'skipped'
+
+
+.. needpie:: Test Results
+   :labels: passed, failed, skipped
+   :colors: green, red, orange
+
+   type == 'testcase' and result == 'passed'
+   type == 'testcase' and result == 'failed'
+   type == 'testcase' and result == 'skipped'
+
+Or show the different types of properties in case you want
+
+.. code-block:: rst
+
+   .. needpie:: Test Types Used In Testcases
+      :labels: fault-injection, interface-test, requirements-based, resource-usage
+      :legend:
+
+      type == 'testcase' and test_type == 'fault-injection'
+      type == 'testcase' and test_type == 'interface-test'
+      type == 'testcase' and test_type == 'requirements-based'
+      type == 'testcase' and test_type == 'resource-usage'
+
+
+.. needpie:: Test Types Used In Testcases
+   :labels: fault-injection, interface-test, requirements-based, resource-usage
+   :legend:
+
+   type == 'testcase' and test_type == 'fault-injection'
+   type == 'testcase' and test_type == 'interface-test'
+   type == 'testcase' and test_type == 'requirements-based'
+   type == 'testcase' and test_type == 'resource-usage'
+
+
 Limitations
 -----------
 
 - Tests must be executed by Bazel before building docs so ``test.xml`` files exist.
 - Not compatible with Esbonio / live preview (no ``bazel-testlogs/`` available).
-- All mandatory properties must be present; partial metadata causes the test to be silently skipped.

src/extensions/score_source_code_linker/__init__.py

@@ -212,13 +212,14 @@ def setup_test_code_linker(app: Sphinx, env: BuildEnvironment):
         )
         # sanity check if extension is enabled
         bazel_testlogs = ws_root / "bazel-testlogs"
-        if not bazel_testlogs.exists():
+        test_folder = ws_root / "tests-report"
+        if not (bazel_testlogs.exists() or test_folder.exists()):
             LOGGER.info(f"{'=' * 80}", type="score_source_code_linker")
             LOGGER.info(
                 f"{'=' * 32}SCORE XML PARSER{'=' * 32}", type="score_source_code_linker"
             )
             LOGGER.info(
-                "'bazel-testlogs' was not found. If test data should be parsed,"
+                "'bazel-testlogs' and 'tests-report' both were not found. If test data should be parsed,"
                 + "please run tests before building the documentation",
                 type="score_source_code_linker",
             )

src/extensions/score_source_code_linker/testlink.py

@@ -197,10 +197,11 @@ def __post_init__(self):
     def check_verifies_fields(self) -> bool:
         if self.PartiallyVerifies is None and self.FullyVerifies is None:
             # This might be a warning in the future, but for now we want be lenient.
-            LOGGER.info(
+            # Changing to debug to keep stdout spam to minimum under default settings
+            LOGGER.debug(
                 f"TestCase: {self.name} Error. Either 'PartiallyVerifies' or "
-                "'FullyVerifies' must be provided."
-                "This test case will be skipped and not linked.",
+                "'FullyVerifies' must be provided to create a link to needs"
+                "Testcase will still be created, but not linked.",
                 type="score_source_code_linker",
             )
             return False

src/extensions/score_source_code_linker/xml_parser.py

@@ -265,17 +265,32 @@ def find_xml_files(search_path: Path) -> list[Path]:
     """
 
     test_file_name = "test.xml"
-    return [x for x in search_path.rglob(test_file_name)]
+    test_files = [x for x in search_path.rglob(test_file_name)]
+    logger.info(f"Found {len(test_files)} test files in total. Parsing them now")
+    if not test_files:
+        logger.info(
+            "Did not find any test.xml files. "
+            "If you expected xml files to be found, please ensure that you have ran your tests "
+            "and put all testfiles either in tests-reports or bazel-testlogs."
+        )
+    return test_files
 
 
 def find_test_folder(base_path: Path | None = None) -> Path | None:
     ws_root = base_path if base_path is not None else find_ws_root()
     assert ws_root is not None
     if os.path.isdir(ws_root / "tests-report"):
+        logger.info(
+            "Found `tests-report` folder. Searching for test.xml files in there"
+        )
         return ws_root / "tests-report"
     if os.path.isdir(ws_root / "bazel-testlogs"):
+        logger.info(
+            "tests-report folder not found. Defaulting to `bazel-testlogs`. Searching for test.xml files in there"
+        )
         return ws_root / "bazel-testlogs"
-    logger.info("could not find tests-report or bazel-testlogs to parse testcases")
+    # This should only happen if bazel was never started in the first place?
+    logger.info("Could not find tests-report or bazel-testlogs to parse testcases")
     return None
 
 
@@ -291,6 +306,9 @@ def run_xml_parser(app: Sphinx, env: BuildEnvironment):
     xml_file_paths = find_xml_files(testlogs_dir)
     test_case_needs = build_test_needs_from_files(app, env, xml_file_paths)
     # Saving the test case needs for cache
+    logger.info(
+        f"Saving {len(test_case_needs)} test case needs to the cache `score_testcaseneeds_cache.json` in _build/."
+    )
     store_data_of_test_case_json(
         app.outdir / "score_testcaseneeds_cache.json", test_case_needs
     )
@@ -299,6 +317,9 @@ def run_xml_parser(app: Sphinx, env: BuildEnvironment):
     )
     # This is not ideal, due to duplication, but I can't think of a better solution
     # right now
+    logger.info(
+        f"Saving {len(output)} parsed testcases to the cache `score_xml_parser_cache.json` in _build/."
+    )
     store_test_xml_parsed_json(app.outdir / "score_xml_parser_cache.json", output)
 
 
@@ -315,10 +336,15 @@ def build_test_needs_from_files(
     tcns: list[DataOfTestCase] = []
     for file in xml_paths:
         # Last value can be ignored. The 'is_valid' function already prints infos
-        test_cases, tests_missing_all_props, _ = read_test_xml_file(file)
+        test_cases, tests_missing_all_props, tests_missing_some_props = (
+            read_test_xml_file(file)
+        )
         non_prop_tests = ", ".join(n for n in tests_missing_all_props)
         if non_prop_tests:
             logger.info(f"Tests missing all properties: {non_prop_tests}")
+        missing_prop_tests = ", ".join(n for n in tests_missing_some_props)
+        if missing_prop_tests:
+            logger.info(f"Tests missing some properties: {missing_prop_tests}")
         tcns.extend(test_cases)
         for tc in test_cases:
             construct_and_add_need(app, tc)
@@ -343,7 +369,8 @@ def construct_and_add_need(app: Sphinx, tn: DataOfTestCase):
     name = tn.name
     external_url = ""
     if tn.repo_name is None or tn.hash is None or tn.url is None:
-        logger.info(
+        # I would change this to debug for now, as it seems too spammy in 'info'
+        logger.debug(
             "Creating testcase need with fallback URL due to incomplete repo metadata: "
             f"name={name}, file={file}, repo_name={tn.repo_name}, "
             f"hash={tn.hash}, url={tn.url}",