add standard for CNCF Kubernetes conformance (#1154)

* add draft for new standard scs-0201-v1
* fix Markdown reference link
* Revert "fix Markdown reference link"
* fix Markdown link
* keep reference to CNCF
* remove misleading reference to decision record
* add reference to test configuration
* remove default value
* remove yet another default value
* fix wording
* update link

---------

Signed-off-by: Marvin Frommhold <depressiveRobot@users.noreply.github.com>

Author: depressiveRobot

Standards/scs-0201-v1-cncf-conformance.md

@@ -0,0 +1,27 @@
+---
+title: CNCF Kubernetes conformance
+type: Standard
+status: Draft
+track: KaaS
+description: |
+  SCS-0201 describes conformance testing of Kubernetes environments using the upstream CNCF Kubernetes e2e suite.
+---
+
+## Introduction
+
+Interoperability, consistency, and traceability play a crucial role in the deployment and use of a Kubernetes environment. Testing ensures this conformance across different Kubernetes environments.
+
+The SCS project always intends to reuse existing standards whenever possible. For this reason, the upstream test suite of the [Certified Kubernetes Conformance Program](https://github.com/cncf/k8s-conformance) is used for conformance testing.
+
+## Motivation
+
+As an operator as well as an user, I want to check the conformance of a Kubernetes environment in order to ensure the interoperability, consistency, and traceability of it.
+
+## Regulations
+
+The conformance testing is performed as part of the [SCS compliance check suite](https://github.com/SovereignCloudStack/standards/tree/main/Tests) executing the standard set of conformance tests defined by the `[Conformance]` tag in the [Kubernetes e2e suite](https://github.com/kubernetes/kubernetes/tree/master/test/e2e). All conformance tests MUST be passed successfully.
+
+We allow exceptions from the Kubernetes e2e tests if it is reasonable, e.g. [bugs in certain tests](https://github.com/SovereignCloudStack/standards/blob/74a3197d37cbfae3f8f0ed45644e6611186639da/Tests/kaas/scs-sonobuoy-config.yaml#L12). Please note that exceptions may be added and/or removed in such reasonable cases and thereby allowing the standard to be updated without the need for a new version of it. If the reason for an exception goes away, so does the exception and it will therefore be removed.
+
+The exceptions are listed under the key `okToFail` in
+[Tests/kaas/scs-sonobuoy-config-v1.yaml](https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/kaas/scs-sonobuoy-config-v1.yaml).

Standards/scs-0201-w1-implementation-testing.md

@@ -0,0 +1,19 @@
+---
+title: "CNCF Kubernetes conformance: Implementation and Testing Notes"
+type: Supplement
+track: KaaS
+supplements:
+  - scs-0201-v1-cncf-conformance.md
+---
+
+## Implementation notes
+
+The actual execution of the conformance tests is performed by the [SCS compliance check suite](https://github.com/SovereignCloudStack/standards/tree/main/Tests) using [Sonobuoy](https://sonobuoy.io).
+
+## Automated tests
+
+The script [`run_sonobuoy.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/kaas/sonobuoy_handler/run_sonobuoy.py) connects to an existing K8s cluster and handles both the execution of Sonobuoy and the generation of the results for a test report. See [`scs-compatible-kaas.yaml`](https://github.com/SovereignCloudStack/standards/blob/c1e9a1b761a0bb201e80d9207f4a1dd0a68ff57f/Tests/scs-compatible-kaas.yaml#L15) for the test configuration.
+
+## Manual tests
+
+None.

Standards/scs-0219-w1-kaas-networking.md

@@ -11,7 +11,7 @@ supplements:
 We test _full support_ for network policies using the upstream e2e tests via Sonobuoy with `--e2e-focus NetworkPolicy`.
 
 All testcases of this set must be passed, with some well-founded exceptions (e.g., due to bugs), that are listed in
-[scs-sonobuoy-config.yaml](https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/kaas/scs-sonobuoy-config.yaml).
+[scs-sonobuoy-config-v1.yaml](https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/kaas/scs-sonobuoy-config-v1.yaml).
 This list will be adapted if and when new exceptions become necessary or old ones become obsolete. Since we only react
 to bugs and other well-founded circumstances, these exceptions are not deemed part of the normative text, and therefore
 these changes do not require proportionate changes in the standard document.

Tests/kaas/scs-sonobuoy-config.yaml Tests/kaas/scs-sonobuoy-config-v1.yamlTests/kaas/scs-sonobuoy-config.yaml renamed to Tests/kaas/scs-sonobuoy-config-v1.yaml

@@ -20,7 +20,7 @@ def cli():
 @click.option("-k", "--kubeconfig", "kubeconfig", required=True, type=click.Path(exists=True), help="path/to/kubeconfig_file.yaml",)
 @click.option("-r", "--result_dir_name", "result_dir_name", type=str, default="sonobuoy_results", help="directory name to store results at",)
 @click.option("-c", "--check", "check_name", type=str, default="sonobuoy_executor", help="this MUST be the same name as the id in 'scs-compatible-kaas.yaml'",)
-@click.option("--scs-sonobuoy-config", "scs_sonobuoy_config_yaml", type=click.Path(exists=True), default="kaas/sonobuoy-config.yaml", help="Configuration for Sonobuoy (yaml format)")
+@click.option("--scs-sonobuoy-config", "scs_sonobuoy_config_yaml", required=True, type=click.Path(exists=True), help="Configuration for Sonobuoy (YAML format)")
 @click.option('--execution-mode', 'mode', type=click.Choice(('serial', 'parallel')), default='serial')
 @click.option("-a", "--arg", "args", multiple=True)
 def sonobuoy_run(kubeconfig, result_dir_name, check_name, scs_sonobuoy_config_yaml, mode, args):
@@ -33,7 +33,7 @@ def sonobuoy_run(kubeconfig, result_dir_name, check_name, scs_sonobuoy_config_ya
 
 
 @cli.command("check-results")
-@click.option("--scs-sonobuoy-config", "scs_sonobuoy_config_yaml", type=click.Path(exists=True), default="kaas/scs-sonobuoy-config.yaml", help="Configuration for Sonobuoy (yaml format)")
+@click.option("--scs-sonobuoy-config", "scs_sonobuoy_config_yaml", required=True, type=click.Path(exists=True), help="Configuration for Sonobuoy (YAML format)")
 @click.argument("sonobuoy_result_yaml", type=click.Path(exists=True))
 def check_results(scs_sonobuoy_config_yaml, sonobuoy_result_yaml):
     check_sonobuoy_result(scs_sonobuoy_config_yaml, sonobuoy_result_yaml)

Tests/kaas/sonobuoy_handler/run_sonobuoy.py

@@ -39,10 +39,10 @@ class SonobuoyHandler:
 
     def __init__(
         self,
+        scs_sonobuoy_config_yaml,
         check_name="sonobuoy_handler",
         kubeconfig=None,
         result_dir_name="sonobuoy_results",
-        scs_sonobuoy_config_yaml="kaas/sonobuoy-config.yaml",
         args=(),
     ):
         self.check_name = check_name

Tests/kaas/sonobuoy_handler/sonobuoy_handler.py

@@ -12,14 +12,14 @@ variables:
   - execution_mode: parallel  # alternatively: serial
 scripts:
 - executable: ./kaas/sonobuoy_handler/run_sonobuoy.py
-  args: run -k {kubeconfig} --scs-sonobuoy-config kaas/scs-sonobuoy-config.yaml -r {subject_root}/sono-results-e2e -c 'cncf-k8s-conformance'  -a '--mode=certified-conformance' --execution-mode {execution_mode}
+  args: run -k {kubeconfig} --scs-sonobuoy-config kaas/scs-sonobuoy-config-v1.yaml -r {subject_root}/sono-results-e2e -c 'cncf-k8s-conformance'  -a '--mode=certified-conformance' --execution-mode {execution_mode}
   #~ args: -k {kubeconfig} -r {subject_root}/sono-results -c 'cncf-k8s-conformance'  -a '--plugin-env e2e.E2E_DRYRUN=true'
   testcases:
   - id: cncf-k8s-conformance
     lifetime: year
     section: heavy
-    description: Must fulfill all requirements of _CNCF Kubernetes conformance_.
-    url: https://github.com/cncf/k8s-conformance/tree/master
+    description: Must fulfill all requirements of scs-0201-v1 (CNCF Kubernetes conformance).
+    url: https://docs.scs.community/standards/scs-0201-v1-cncf-conformance#regulations
 - executable: ./kaas/k8s-version-policy/k8s_version_policy.py
   args: -k {kubeconfig}
   testcases:
@@ -35,7 +35,7 @@ scripts:
     description: Must fulfill all requirements of scs-0214-v2.
     url: https://docs.scs.community/standards/scs-0214-v2-k8s-node-distribution#decision
 - executable: ./kaas/sonobuoy_handler/run_sonobuoy.py
-  args: run -k {kubeconfig} --scs-sonobuoy-config kaas/scs-sonobuoy-config.yaml -r {subject_root}/sono-results-0219 -c 'kaas-networking-check' -a '--e2e-focus "NetworkPolicy"' --execution-mode {execution_mode}
+  args: run -k {kubeconfig} --scs-sonobuoy-config kaas/scs-sonobuoy-config-v1.yaml -r {subject_root}/sono-results-0219 -c 'kaas-networking-check' -a '--e2e-focus "NetworkPolicy"' --execution-mode {execution_mode}
   testcases:
   - id: kaas-networking-check
     lifetime: month
@@ -44,8 +44,8 @@ scripts:
     url: https://docs.scs.community/standards/scs-0219-w1-kaas-networking#automated-tests
 modules:
   - id: cncf-k8s-conformance
-    name: CNCF Kubernetes conformance
-    url: https://github.com/cncf/k8s-conformance/tree/master
+    name: scs-0201-v1: CNCF Kubernetes conformance
+    url: https://docs.scs.community/standards/scs-0201-v1-cncf-conformance
     targets:
       main:
       - cncf-k8s-conformance