Add external tutorial sources support (git/archive) for systemtests

Allow tests.yaml entries to load tutorials from external git repos or
archives. Rebased on develop and documented usage in the README.

Author: PranjalManhgaye

changelog-entries/732.md

@@ -0,0 +1 @@
+- Added optional `source` blocks in `tests.yaml` so system tests can load tutorials from external git repositories or archives ([#732](https://github.com/precice/tutorials/pull/732)).

tools/tests/README.md

@@ -36,7 +36,7 @@ Workflow for the preCICE v3 release testing:
 
 6. Download the build artifacts from Summary > runs.
 
-    - In there, you may want to check the `stdout.log` and `stderr.log` files.
+    - In there, you may want to check the `system-tests-stdout.log` and `system-tests-stderr.log` files.
     - The produced results are in `precice-exports/`, the reference results in `reference-results-unpacked`.
     - Compare using, e.g., ParaView or [fieldcompare](https://gitlab.com/dglaeser/fieldcompare): `fieldcompare dir precice-exports/ reference/`. The `--diff` option will give you `precice-exports/diff_*.vtu` files, while you can also try different tolerances with `-rtol` and `-atol`.
 
@@ -105,7 +105,9 @@ In this case, building and running seems to work out, but the tests fail because
 
 The easiest way to debug a systemtest run is first to have a look at the output written into the action on GitHub.
 If this does not provide enough hints, the next step is to download the generated `system_tests_run_<run_id>_<run_attempt>` artifact. Note that by default this will only be generated if the systemtests fail.
-Inside the archive, a test-specific subfolder like `flow-over-heated-plate_fluid-openfoam-solid-fenics_2023-11-19-211723` contains two log files: a `stderr.log` and `stdout.log`. This can be a starting point for a further investigation.
+Inside the archive, a test-specific subfolder like `flow-over-heated-plate_fluid-openfoam-solid-fenics_2023-11-19-211723` contains two log files: `system-tests-stderr.log` and `system-tests-stdout.log`. This can be a starting point for a further investigation. When fieldcompare runs with `--diff`, it writes VTK diff files under `precice-exports/`; if the comparison fails, those files are copied into a `diff-results/` subfolder in the same run directory (mirroring any subpaths under `precice-exports/`) so you can open them (e.g. in ParaView) to see where results differ from the reference. On successful comparisons, `diff-results/` is therefore absent.
+
+For implicit-coupling runs, `precice-*-iterations.log` files are collected into `iterations-logs/` and compared by hash against archived reference copies (stored next to each reference `.tar.gz` in a `*.iterations-logs/` directory, or legacy `.iterations-hashes.json` sidecars). A mismatch fails the test.
 
 ## Adding new tests
 
@@ -118,6 +120,8 @@ In order for the systemtests to pick up the tutorial we need to define a `metada
 To add a testsuite just open the `tests.yaml` file and use the output of `python print_case_combinations.py` to add the right case combinations you want to test. Note that you can specify a `reference_result` which is not yet present. The `generate_reference_data.py` will pick that up and create it for you.
 Note that its important to carefully check the paths of the `reference_result` in order to not have typos in there. Also note that same cases in different testsuites should use the same `reference_result`.
 
+To cap the preCICE simulation time for a specific test without editing `precice-config.xml`, add an optional `max_time` (positive float, overrides `<max-time>`) or `max_time_windows` (positive integer, overrides `<max-time-windows>`) field to the tutorial entry. Applies to both test runs and reference result generation.
+
 ### Generate reference results
 
 Since we need data to compare against, you need to run `python generate_reference_data.py`. This process might take a while.
@@ -167,7 +171,7 @@ Metadata and workflow/script files:
     - ...
   - `dockerfiles/`
     - Multi-stage build Dockerfiles that define how to build each component, in a layered approach
-  - `docker-compose.template.yaml`: Describes how to prepare each test (Docker Componse service template)
+  - `docker-compose.template.yaml`: Describes how to prepare each test (Docker Compose service template)
   - `docker-compose.field_compare.template.yaml`: Describes how to compare results with fieldcompare (Docker Compose service template)
   - `components.yaml`: Declares the available components and their parameters/options
   - `reference_results.metadata.template`: Template for reporting the versions used to generate the reference results
@@ -231,9 +235,9 @@ cases:
 Description:
 
 - `name`: A human-readable, descriptive name
-- `path`: Where the tutorial is located, relative to the tutorials repository
+- `path`: Where the tutorial is located, relative to the tutorials repository (or the tutorial folder name for external sources)
 - `url`: A web page with more information on the tutorial
-- `participants`: A list of preCICE participants, typically corresponing to different domains of the simulation
+- `participants`: A list of preCICE participants, typically corresponding to different domains of the simulation
 - `cases`: A list of solver configuration directories. Each element of the list includes:
   - `participant`: Which participant this solver case can serve as
   - `directory`: Where the case directory is located, relative to the tutorial directory
@@ -279,7 +283,9 @@ This `openfoam-adapter` component has the following attributes:
 
 Since the docker containers are still a bit mixed in terms of capabilities and support for different build_argument combinations the following rules apply:
 
-- A build_argument ending in **_REF** means that it refers to a git commit-ish (like a tag or commit) beeing used to build the image. Its important to not use branch names here as we heavily rely on dockers build cache to speedup things. But since the input variable to the docker builder will not change, we might have wrong cache hits.
+- A build argument ending in `_REF` refers to a git commit-ish (like a tag or commit) being used to build the image. It is important to not use branch names here as we heavily rely on Docker's build cache to speedup things. But since the input variable to the docker builder will not change, we might have wrong cache hits.
+- Some workflows set variables ending in `_PR`. These specify the GitHub pull request which provides the above `_REF` and can be on a fork.
+- A build argument ending in `_VERSION` refers to the version of a third-party dependency to use (e.g., DUNE).
 - All other build_arguments are free of rules and up to the container maintainer.
 
 ### Component templates
@@ -331,15 +337,43 @@ test_suites:
           - fluid-openfoam
           - solid-fenics
         reference_result: ./flow-over-heated-plate/reference-results/fluid-openfoam_solid-fenics.tar.gz
+        timeout: 1200
 ```
 
+The optional `timeout` field (in seconds) sets the maximum time for the solver run and fieldcompare phases of that specific case. If omitted, it defaults to `GLOBAL_TIMEOUT` (currently 900s, overridable via the `PRECICE_SYSTEMTESTS_TIMEOUT` environment variable).
+
+#### External tutorial sources
+
+By default, every `path` must exist in the local `precice/tutorials` checkout. For tutorials maintained elsewhere, add an optional `source` block:
+
+```yaml
+- path: flow-over-heated-plate
+  source:
+    type: git
+    url: https://github.com/precice/tutorials.git
+    ref: develop
+    subdir: .   # optional subdirectory inside the repository
+  case_combination:
+    - fluid-openfoam
+    - solid-openfoam
+  reference_result: ./flow-over-heated-plate/reference-results/fluid-openfoam_solid-openfoam.tar.gz
+```
+
+Supported `type` values:
+
+- `local` (default): use the tutorial from this repository.
+- `git`: shallow-clone `url` at `ref` (cached under `~/.cache/precice-tutorials` or `PRECICE_EXTERNAL_CACHE_DIR`).
+- `archive`: download and extract a `.tar.gz` / `.zip` from `url`.
+
+The runner copies the resolved tutorial into the run directory, then continues with the usual Docker build/run and fieldcompare steps. `TUTORIALS_REF` / `TUTORIALS_PR` build arguments still apply only to **local** tutorials; external tutorials are pinned by `source.ref`.
+
 This defines two test suites, namely `openfoam_adapter_pr` and `openfoam_adapter_release`. Each of them defines which case combinations of which tutorials to run.
 
 ### Generate Reference Results
 
 #### via GitHub workflow (recommended)
 
-The preferred way of adding reference results is via the manual triggerable `Generate reference results (manual)` workflow. This takes two inputs:
+The preferred way of adding reference results is via the manual `Generate reference results (manual)` workflow. This takes two inputs:
 
 - `from_ref`: branch where the new test configuration (e.g added tests, new reference_versions.yaml) is
 - `commit_msg`: commit message for adding the reference results into the branch

tools/tests/metadata_parser/metdata.py

@@ -6,6 +6,9 @@
 import itertools
 from paths import PRECICE_TESTS_DIR, PRECICE_TUTORIAL_DIR
 
+# Import TutorialSource from systemtests.sources (used for external tutorial sources).
+from systemtests.sources import TutorialSource
+
 
 @dataclass
 class BuildArgument:
@@ -279,13 +282,15 @@ def from_cases_tuple(cls, cases: Tuple[Case], tutorial: Tutorial):
 class ReferenceResult:
     path: Path
     case_combination: CaseCombination
+    base_dir: Optional[Path] = None
 
     def __repr__(self) -> str:
         return f"{self.path.as_posix()}"
 
     def __post_init__(self):
         # built full path
-        self.path = PRECICE_TUTORIAL_DIR / self.path
+        base = self.base_dir if self.base_dir is not None else PRECICE_TUTORIAL_DIR
+        self.path = Path(base) / self.path
 
 
 @dataclass
@@ -299,6 +304,7 @@ class Tutorial:
     url: str
     participants: List[str]
     cases: List[Case]
+    source: "TutorialSource" = field(default_factory=TutorialSource.local)
     case_combinations: List[CaseCombination] = field(init=False)
 
     def __post_init__(self):
@@ -355,29 +361,36 @@ def get_case_by_string(self, case_name: str) -> Optional[Case]:
         return None
 
     @classmethod
-    def from_yaml(cls, path, available_components):
+    def from_yaml(cls, path, available_components, base_dir=None, source=None):
         """
         Creates a Tutorial instance from a YAML file.
 
         Args:
-            path: The path to the YAML file.
+            path: The path to the metadata.yaml file.
             available_components: The Components instance containing available components.
+            base_dir: Optional base directory for resolving tutorial path (for external sources).
+                      Defaults to PRECICE_TUTORIAL_DIR.
+            source: Optional TutorialSource (for external tutorials).
 
         Returns:
             An instance of Tutorial.
         """
         with open(path, 'r') as f:
             data = yaml.safe_load(f)
             name = data['name']
-            path = PRECICE_TUTORIAL_DIR / data['path']
+            base = base_dir if base_dir is not None else PRECICE_TUTORIAL_DIR
+            tutorial_path = Path(base) / data['path']
             url = data['url']
             participants = data.get('participants', [])
             cases_raw = data.get('cases', {})
             cases = []
             for case_name in cases_raw.keys():
                 cases.append(Case.from_dict(
                     case_name, cases_raw[case_name], available_components))
-            return cls(name, path, url, participants, cases)
+            tut = cls(name, tutorial_path, url, participants, cases)
+            if source is not None:
+                tut.source = source
+            return tut
 
 
 class Tutorials(list):
@@ -440,4 +453,4 @@ def from_path(cls, path):
         for yaml_path in yaml_files:
             tut = Tutorial.from_yaml(yaml_path, available_components)
             tutorials.append(tut)
-        return cls(tutorials)
+        return cls(tutorials)