Merge branch 'main' into feat/scs-0102-v2

Author: garloff

.gitignore

@@ -7,3 +7,4 @@ node_modules
 Tests/kaas/results/
 Tests/kaas/kaas-sonobuoy-tests/results/
 *.tar.gz
+.direnv

.zuul.d/config.yaml

@@ -54,6 +54,7 @@
     # a) leave room for long-running tests (mostly when some cloud is unreachable)
     # b) keep in mind that this job covers ALL test subjects (at most 4 in parallel)
     timeout: 3600
+    attempts: 2  # usually, retries are not too promising; better to save resources
     vars:
       preset: iaas
 - job:
@@ -63,6 +64,7 @@
     # a) these tests take a lot of time, I'm afraid, particularly Sonobuoy
     # b) keep in mind that this job covers ALL test subjects (at most 4 in parallel)
     timeout: 21600  # 6 hrs -- 5 hrs was almost sufficient (reports came through sometimes)
+    attempts: 1  # this job is heavy, and retries aren't very promising
     vars:
       preset: kaas
       iaas: false

.zuul.d/secure.yaml

@@ -1,6 +1,6 @@
 # Project Board Term 2026
 
-Spokesperson:
+Spokesperson: Kurt Garloff, @garloff
 
 | Name, Firstname         | Github Handle  | E-Mail                              | Remark                         |
 | ----------------------- | -------------- | ----------------------------------- |---------------------------------

…nance/2025-01-project-board-term-2025.md …nity-Governance/2025-01-project-board.mdCommunity-Governance/2025-01-project-board-term-2025.md renamed to Community-Governance/2025-01-project-board.md Community-Governance/2025-01-project-board-term-2025.md renamed to Community-Governance/2025-01-project-board.md

@@ -86,22 +86,23 @@ While instructive, this view is still a bit simplified. Let's get more precise n
    A statement about the subject that can be evaluated unambiguously to be either satisfied or not. The result is either `PASS` or `FAIL`, or—if the test could not be performed—`DNF` (did not finish).
    A test case can be as simple as "the subject conforms to standard X", but a standard can also be decomposed into multiple test cases, which can then be reported on (also to the customers) individually.
    This latter option has the advantage that we can show explicitly if the subject complies with optional parts of the standard.
-3. _Check_:
-   A script that determines and reports the results of certain test cases. The report is printed to stdout, and each test case is reported as a single line of the form `testcase-id: [PASS/FAIL]`. The result `DNF` is not reported. Lines of other forms are permissible and will be ignored.
-   We also occasionally extend the concept of _check_ to manual audits.
-4. _Module_:
-   A collection of test cases and corresponding checks, together with additional meta information such as the result lifetime, description, and a list of tags for a test case.
-   Ultimately, we aim to specify one module for each version of each standard: the module translates the standard into something measurable and, ideally, executable to be used for certification.
-5. _Selector (expression)_:
-   An expression used to select test cases by referring to the tags that must (or must not) be present.
-6. _Target_:
-   A named collection of test cases specified using selector expressions.
+3. _Script_:
+   A script that determines and reports the results of certain test cases,
+   which are specified with the script, together with additional meta information such as the result lifetime, and a description for each test case.
+   The report is printed to stdout, and each test case is reported as a single line of the form `testcase-id: [PASS/FAIL/ABORT]`.
+   Lines of other forms are permissible and will be ignored.
+   We also occasionally extend the concept of _script_ to manual audits.
+4. _Target_:
+   A named collection of test cases.
    Ultimately, the certification of a subject always depends on a designated "main" target; all its test cases must be passed for the certificate to be awarded.
    Further targets can be used to report on optional aspects of the certificate, such as particularly good security and encryption measures.
-7. _(Certificate-scope) version_:
-   A collection of modules and a collection of targets, one of them being "main".
+5. _Module_:
+   A collection of targets.
+   Ultimately, we aim to specify one module for each version of each standard: the module translates the standard into something measurable and, ideally, executable to be used for certification.
+6. _(Certificate-scope) version_:
+   A collection of modules. Like a module, the version has targets, but they are implicitly given by taking the union of the targets of all its modules.
    Note that a collection of modules can again be construed as a (larger) module. We opt to use one module per standard version, as mentioned above, in order to make commonalities between certificate-scope versions explicit.
-8. _Certificate scope_:
+7. _Certificate scope_:
    A list of certificate-scope versions.
 
 Having introduced these concepts, we can now get even more precise by defining the actual specification in YAML format.
@@ -120,6 +121,7 @@ The certification YAML _MUST_ contain the following keys:
 | `uuid`     | String        | Universally unique identifier                        | `d912d0a5-826a-4b01-bafd-b48f65f76f43`                                                          |
 | `name`     | String        | Full name of this certificate scope                  | `SCS-open KaaS`                                                                                 |
 | `url`      | String        | Valid URL to the latest raw version of this document | `https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/scs-open-kaas.yaml` |
+| `scripts`  | Array of maps | List of script descriptors (described below)         | (see below)                                                                                     |
 | `modules`  | Array of maps | List of module descriptors (described below)         | (see below)                                                                                     |
 | `timeline` | Array of maps | List of timeline entries (described below)           | (see below)                                                                                     |
 | `versions` | Array of maps | List of version descriptors (described below)        | (see below)                                                                                     |
@@ -154,7 +156,6 @@ then a certificate of that prerequisite scope has to be presented before the cer
 | --------------- | ------------- | ------------------------------------------------------------------------------- | ------------------ |
 | `version`       | String        | required: version of the particular list of standards                           | `v3`               |
 | `include`       | Array         | required: list of module ids or include descriptors (see below)                 | `[scs-0100-v3]`    |
-| `targets`       | Map of maps   | required: this maps target names to selector expressions (explained below)      | `main: mandatory`  |
 | `stabilized_at` | Date          | ISO formatted date indicating the date after this version is considered stable. | `2022-11-09`       |
 
 The ids of the test cases of all the modules specified via `include` MUST be pairwise different.
@@ -172,38 +173,6 @@ Each include may be specified by means of a module id (i.e., a string) or by an
 
 When the referenced module uses parameters, then these parameters must be assigned values here.
 
-#### Selector expressions
-
-In order to define what a selector expression is, we need to define tags, atoms and terms first.
-
-A _tag_ is a string that does not contain any space, comma, forward slash, or exclamation mark.
-
-Examples: `iaas`, `mandatory`, `recommended`, `encryption`.
-
-An _atom_ is a string that is either (i) a tag or (ii) an exclamation mark followed by tag.
-A list of tags _satisfies_ the atom if
-
-- the atom is of form (i) and the tag is contained in the list, or
-- the atom is of form (ii) and the tag is not contained in the list.
-
-Examples: `mandatory`, `!mandatory`.
-
-A _term_ is a string that is a non-empty list of atoms joined by slashes.
-A list of tags _satisfies_ the term if it satisfies at least one of the atoms.
-
-Examples: `mandatory`, `mandatory/recommended`, `!mandatory/encryption`.
-
-A _selector (expression)_ is a string that is a non-empty list of terms joined by space.
-A list of tags _satisfies_ the selector if it satisfies all the terms.
-
-Examples: `mandatory`, `iaas mandatory`, `iaas !mandatory/encryption`.
-
-In the map `targets` above, it is possible to specify a list of selectors that are joined by comma.
-(Note that this is still a string, not a YAML list.)
-A list of tags satisfies this list of selectors if it satisfies at least one of the selectors.
-
-Examples: `mandatory iaas, recommended kaas` (NOT: `[mandatory iaas, recommended kaas]`)
-
 ### Module descriptor
 
 | Key                      | Type   | Description                                                                 | Example                                                           |
@@ -212,8 +181,7 @@ Examples: `mandatory iaas, recommended kaas` (NOT: `[mandatory iaas, recommended
 | `name`                   | String | name of this module                                                         | `Flavor naming v3`                                                |
 | `url`                    | String | Valid URL to relevant documentation (usually a standard document)           | `https://docs.scs.community/standards/scs-0100-v3-flavor-naming`  |
 | `parameters`             | List   | List of parameters that the checks in this module might use                 | `[image_spec]`                                                    |
-| `run`                    | Array  | List of all checks that should be run; each entry being a check descriptor  | (see below)                                                       |
-| `testcases`              | Array  | List of all test cases; each entry being a test-case descriptor             | (see below)                                                       |
+| `targets`                | Map of lists   | required: this maps target names to lists of testcases      | `main: [tc1, tc2]`  |
 
 The parameters specified here will be added to the variable assignment for all check tools that belong to this module, so they will be substituted in the same way.
 The values to these parameters must be provided in the include descriptor as explained above.
@@ -223,13 +191,14 @@ Using parameters offers two advantages:
 - they may show up in the automatically generated documentation, whereas the check tools themselves probably won't.
 - multiple versions of a standard can be represented using the same module, if everything that changes between versions can be captured by the parameters.
 
-### Check descriptor
+### Script descriptor
 
-The following fields are valid for every check descriptor:
+The following fields are valid for every script descriptor:
 
 | Key               | Type   | Description                                                                                    | Example    |
 | ----------------- | ------ | ---------------------------------------------------------------------------------------------- | ---------- |
 | `section`         | String | _Optional_ what section to associate this check with (sections can be checked in isolation)    | `weekly`   |
+| `testcases`       | Array  | List of all test cases; each entry being a test-case descriptor                                | (see below)  |
 
 Additional fields are valid depending on whether the check is automated or manual.
 
@@ -257,14 +226,12 @@ TBD
 | ----------------- | --------------- | ------------------------------------------------------------------------------------------------- | ----------------- |
 | `id`              | String          | Identifier for this test case (immutable and unique within this module)                           | `image-md-check`  |
 | `lifetime`        | String          | One of: `day`, `week` (_default_), `month`, `quarter`, `year`                                     | `day`             |
-| `tags`            | List of strings | A tag is a keyword that will be used to select this test case using a selector expression         | `[mandatory]`     |
-| `description`     | String          | Short description of the test case                                                                |                   |
+| `description`     | String          | Short description of the test case (markdown allowed, but keep it short for CLI users)            |                   |
+| `url`             | String          | URL pointing to the relevant SCS documentation for the testcase                                   |                   |
 
 A test result is valid until the end of the next period, except when lifetime is `year`: then the result is
 valid until the end of the following month plus one year.
 
-A tag MUST NOT contain any of these characters: space, comma, exclamation mark, forward slash.
-
 The `id` of a test case MUST NOT be changed.
 Exceptions MAY be made if the test case is not referenced by any stable version.
 

…nce/2026-01-project-board-termin-2026.md …nity-Governance/2026-01-project-board.mdCommunity-Governance/2026-01-project-board-termin-2026.md renamed to Community-Governance/2026-01-project-board.md Community-Governance/2026-01-project-board-termin-2026.md renamed to Community-Governance/2026-01-project-board.md

@@ -154,30 +154,12 @@ generation `x` for the time being.
 
 ## Automated tests
 
-### Errors
-
-The following items MUST be detected and reported as an error:
-
-- any syntax error in a name starting with `SCS-`,
-- any mismatch between any immediately discoverable property of a flavor (currently, CPU, RAM and disk size)
-  and the meaning of its name (which is usually a lower bound), such as the CPU generation or hypervisor.
-
-In addition, the following items MAY be reported as an error:
-
-- any mismatch between any non-immediately discoverable property of flavor and the meaning of its name.
-
-### Warnings
-
-None so far.
-
-### Implementation
-
-We implemented two testcases, paralleling the two items in the "Errors" section above:
+The following testcases [are implemented](https://github.com/SovereignCloudStack/standards/tree/main/Tests/iaas/openstack_test.py):
 
 - `scs-0100-syntax-check` ensures that any name starting with `SCS-` adheres to the standard;
-- `scs-0100-semantics-check` ensures that any such name is telling the truth as specified in the standard.
-
-These testcases can be checked using [`openstack_test.py`](https://github.com/SovereignCloudStack/standards/tree/main/Tests/iaas/openstack_test.py).
+- `scs-0100-semantics-check` ensures that any such name is telling the truth as specified in the standard;
+  specifically: any immediately discoverable property of a flavor (currently, CPU, RAM and disk size)
+  matches the meaning of its name (which is usually a lower bound), such as the CPU generation or hypervisor.
 
 ## Manual tests
 

Standards/scs-0003-v1-sovereign-cloud-standards-yaml.md

@@ -19,57 +19,26 @@ Only the flavor and image attributes required by the standard have to be set:
 
 ## Automated tests
 
-### Images sample
+The following testcases [are implemented](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py):
 
-Some checks need to be performed on a live instance. For these checks, it is
-necessary to choose a sample of VM images to test on.
-
-For the time being, the sample MUST contain at least one public image reported
-by OpenStack. This may be extended in the future.
-
-### Errors
-
-For every image in the chosen sample, the following items MUST be detected and
-reported as an error:
-
-- the service `rngd` is not running,
-- the special file `/proc/sys/kernel/random/entropy_avail` does not contain
-  the value 256 (pinned since kernel 5.18),
-- the number of FIPS 140-2 failures exceeds 5 out of 1000 blocks
-  tested, as determined by `cat /dev/random | rngtest -c 1000` .
+- `scs-0101-image-property` ensures that each flavor has the extra spec `hw_rng:allowed=True`,
+- `scs-0101-flavor-property` ensures that each public image has the property `hw_rng_model: virtio`,
+- `scs-0101-rngd` ensures that the service `rngd` is present on a sample VM,
+- `scs-0101-entropy-avail` ensures that the special file `/proc/sys/kernel/random/entropy_avail` contains
+  the value 256 (pinned since kernel 5.18) on a sample VM,
+- `scs-0101-fips-test` ensures that the number of FIPS 140-2 failures is below 5 out of 1000 blocks
+  tested, as determined by `cat /dev/random | rngtest -c 1000` on a sample VM.
 
 Note: The latter two items act as surrogates for the following item, which
 cannot be detected directly:
 
-- CPU instructions for accessing entropy are not available to the VMs.
-
-### Warnings
-
-The following items MUST be detected and reported as a warning:
-
-- any flavor missing the attribute `hw_rng:allowed=True`,
-- any image missing the attribute `hw_rng_model: virtio`,
-- the number of FIPS 140-2 failures exceeds 3 out of 1000 blocks
-  tested (compare with errors).
+- CPU instructions for accessing entropy are available to the VMs.
 
 Note that the requirement regarding the kernel patch level will not be
 checked, because of two reasons: (a) we already check the file `entropy_avail`
 (see subsection on Errors), and (b) users can always choose a recent image,
 as ensured by the image metadata standard.
 
-### Implementation
-
-We implemented the following testcases that reflect the items in the above section
-on automated tests:
-
-- `scs-0101-image-property`,
-- `scs-0101-flavor-property`,
-- `scs-0101-entropy-avail`,
-- `scs-0101-rngd`,
-- `scs-0101-fips-test` (covers both the error and warning case).
-
-These testcases can be checked using [`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).
-
 ## Manual tests
 
 None.

Standards/scs-0100-w1-flavor-naming-implementation-testing.md

@@ -16,8 +16,10 @@ for these images.
 
 ## Automated tests
 
-We implemented a host of testcases to reflect the requirements and recommendations of the standard. The following
-testcases ensure that fields have proper values:
+We [implemented](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py)
+a host of testcases to reflect the requirements and recommendations of the standard.
+
+The following testcases ensure that fields have proper values:
 
 - `scs-0102-prop-architecture`,
 - `scs-0102-prop-hash_algo`,
@@ -43,9 +45,6 @@ The following testcase ensures that each image is as recent as claimed by its `r
 
 - `scs-0102-image-recency`
 
-The script [`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py)
-can be used to check these testcases.
-
 ## Manual tests
 
 None.

Standards/scs-0101-w1-entropy-implementation-testing.md

@@ -17,13 +17,11 @@ You can generate input for it using the tool
 
 ## Automated tests
 
-We implemented a set of testcases corresponding 1:1 to the standard flavors:
+We [implemented](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py)
+a set of testcases corresponding 1:1 to the standard flavors:
 
 - `scs-0103-flavor-X` with varying `X`: ensures that flavor `SCS-X` is present and has the
   required `extra_specs`.
 
 _NOTE_: We still need to add testcases to ensure that the `extra_specs` of non-standard
 flavors are correct as well.
-
-The testcases can be run using the script
-[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).