|
| 1 | +--- |
| 2 | +title: Sovereign Cloud Standards Testing |
| 3 | +type: Procedural |
| 4 | +status: Draft |
| 5 | +track: Global |
| 6 | +description: | |
| 7 | + SCS-0003 defines concepts central to testing SCS standards and regulates how results may |
| 8 | + be obtained and aggregated. |
| 9 | +--- |
| 10 | + |
| 11 | +## Introduction |
| 12 | + |
| 13 | +Standards |
| 14 | + |
| 15 | +## Concept definitions |
| 16 | + |
| 17 | +A standard can be viewed as a collection of propositions that "must" (or "should") be satisfied |
| 18 | +by a test subject (cloud or cluster). The standard is satisfied if all "must" propositions are |
| 19 | +satisfied. |
| 20 | + |
| 21 | +A _testcase_ is a collection of propositions that coincide with respect to the following properties: |
| 22 | + |
| 23 | +- whether they can be tested automatically with normal user permissions; |
| 24 | +- how often they need to be tested (daily, weekly, monthly, annually); |
| 25 | +- whether they are required ("must"). |
| 26 | + |
| 27 | +We unambiguously refer to a testcase using a composite identifier consisting of two parts: |
| 28 | + |
| 29 | +- the _scope_: in the context of this standard, just an identifier of a namespace; |
| 30 | + for example: `scs-compatible-iaas` (or a UUID) |
| 31 | +- the _testcase id_, for example `scs-0100-syntax-check` or `scs-0101-fips-test`. |
| 32 | + |
| 33 | +The scope part is usually clear from the context and therefore omitted. |
| 34 | + |
| 35 | +A _test_ is a testcase or a collection of tests. We refer to a test using the same kind of |
| 36 | +composite identifier as for a testcase. |
| 37 | + |
| 38 | +For instance, if we have testcases `scs-0100-syntax-check` and `scs-0100-semantics-check`, |
| 39 | +we could define the test `scs-0100-v3` as the collection consisting of these two testcases. |
| 40 | +Given further tests `scs-0101-v1`, `scs-0102-v1` etc., we could define the test `scs-0501-v4` |
| 41 | +as the collection consisting of `scs-0100-v3`, `scs-0101-v1`, `scs-0102-v1` etc. |
| 42 | + |
| 43 | +A test can be viewed as a collection of propositions; namely, all propositions of all testcases |
| 44 | +that are part of the test. |
| 45 | + |
| 46 | +The _result_ of a test is one of the following values: |
| 47 | + |
| 48 | +- `FAIL`: it could be verified that at least one of its propositions is not satisfied; |
| 49 | +- `MISS` (missing): for at least one of its propositions, it was not (recently) attempted to verify it; |
| 50 | +- `DNF` (did not finish): for at least one of its propositions, it could not be verified whether it is satisfied; |
| 51 | +- `PASS`: it could be verified that all its propositions are satisfied. |
| 52 | + |
| 53 | +If multiple items apply, we always opt for the topmost one. |
| 54 | + |
| 55 | +A _score card_ is a data structure that contains the following information: |
| 56 | + |
| 57 | +- Subject: the name of the test subject, |
| 58 | +- Scope: the scope of all the testcases referred to in this score card, |
| 59 | +- Test results: a mapping that maps testcase ids to results. |
| 60 | + |
| 61 | +A _test report_ is a data structure that contains the information of a score card, plus: |
| 62 | + |
| 63 | +- Creator: who created the report (name of person or version of test suite), |
| 64 | +- Check date: when the test was performed, |
| 65 | +- Log: free-form text that details the test run. |
| 66 | + |
| 67 | +A _check script_ is a computer program that tests one or more testcases and reports the result per testcase. |
| 68 | + |
| 69 | +The precise form of the data structures (score card, report), as well as the input-output formats of the check scripts is up to the implementation, but it must be well documented. |
| 70 | + |
| 71 | +A _test suite_ is a collection of interoperable check scripts, i.e., check scripts that use the same data structures and I/O formats. |
| 72 | + |
| 73 | +## Regulation |
| 74 | + |
| 75 | +Each standard must be decomposed into testcases. Each testcase should be "atomic" in the following two senses: |
| 76 | + |
| 77 | +- it's clear what specific part of the standard is satisfied or not satisfied; |
| 78 | +- the testcase can be reused for multiple versions of the standards. |
| 79 | + |
| 80 | +The latter criterion is a matter of engineering judgment, because it cannot be known in advance how a standard might evolve. |
| 81 | + |
| 82 | +Each testcase id must be prefixed by `scs-XXXX-` where `XXXX` is the document id of the standard; an exception is possible in the rare case when a testcase applies to multiple standards. |
| 83 | + |
| 84 | +A check script that can test multiple testcases should provide the option to select which testcases to run. |
| 85 | + |
| 86 | +A list of reports from a certain time frame can be merged into an aggregate score card, provided that the following conditions are satisfied: |
| 87 | + |
| 88 | +- for each testcase, the result must be taken from the most recent report containing that testcase, |
| 89 | +- if an additional report from the same time frame is known to exist, all its testcases must be contained in more recent reports from the list. |
0 commit comments