Skip to content

Commit ff0ce9a

Browse files
mbuechsembuechse
authored
Replace yaml scopes by regular standards (#1199)
* Fix chk_adrs.py: Stable need not precede Deprecated * Replace yaml scopes by regular standards * Add scs-0003-v2 Signed-off-by: Matthias Büchse <matthias.buechse@alasca.cloud>
1 parent 384d0c4 commit ff0ce9a

13 files changed

Lines changed: 369 additions & 10 deletions

‎Standards/scs-0001-v1-sovereign-cloud-standards.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,7 @@ The second digit in `XXXX` describes the track where the document belongs:
100100
| KaaS | 2 |
101101
| IAM | 3 |
102102
| Ops | 4 |
103+
| Scopes | 5 |
103104

104105
In addition to the number, each document has the following metadata,
105106
embedded in the markdown header.

‎Standards/scs-0003-v2-sovereign-cloud-standards-testing.md‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
---
2+
title: Sovereign Cloud Standards Testing
3+
type: Procedural
4+
status: Draft
5+
track: Global
6+
replaces:
7+
- scs-0003-v1-sovereign-cloud-standards-yaml.md
8+
description: |
9+
SCS-0003 defines concepts central to testing SCS standards and regulates how results may
10+
be obtained and aggregated.
11+
---
12+
13+
## Introduction
14+
15+
This standard defines concepts central to testing SCS standards and regulates how results may
16+
be obtained and aggregated.
17+
18+
## Concept definitions
19+
20+
A standard can be viewed as a collection of propositions that "must" (or "should") be satisfied
21+
by a test subject (cloud or cluster). The standard is satisfied if all "must" propositions are
22+
satisfied.
23+
24+
A _testcase_ is a collection of propositions that coincide with respect to the following properties:
25+
26+
- whether they can be tested automatically with normal user permissions;
27+
- how often they need to be tested (daily, weekly, monthly, annually);
28+
- whether they are required ("must").
29+
30+
We unambiguously refer to a testcase using a composite identifier consisting of two parts:
31+
32+
- the _scope_: in the context of this standard, just an identifier of a namespace;
33+
for example: `scs-compatible-iaas` (or a UUID)
34+
- the _testcase id_, for example `scs-0100-syntax-check` or `scs-0101-fips-test`.
35+
36+
The scope part is usually clear from the context and therefore omitted.
37+
38+
A _test_ is a testcase or a collection of tests. We refer to a test using the same kind of
39+
composite identifier as for a testcase.
40+
41+
For instance, if we have testcases `scs-0100-syntax-check` and `scs-0100-semantics-check`,
42+
we could define the test `scs-0100-v3` as the collection consisting of these two testcases.
43+
Given further tests `scs-0101-v1`, `scs-0102-v1` etc., we could define the test `scs-0501-v4`
44+
as the collection consisting of `scs-0100-v3`, `scs-0101-v1`, `scs-0102-v1` etc.
45+
46+
A test can be viewed as a collection of propositions; namely, all propositions of all testcases
47+
that are part of the test.
48+
49+
The _result_ of a test is one of the following values:
50+
51+
- `FAIL`: it could be verified that at least one of its propositions is not satisfied;
52+
- `MISS` (missing): for at least one of its propositions, it was not (recently) attempted to verify it;
53+
- `DNF` (did not finish): for at least one of its propositions, it could not be verified whether it is satisfied;
54+
- `PASS`: it could be verified that all its propositions are satisfied.
55+
56+
If multiple items apply, we always opt for the topmost one.
57+
58+
A _test report_ is a data structure that contains the following information:
59+
60+
- UUID,
61+
- Subject: the name of the test subject,
62+
- Scope: the scope of all the testcases referred to in this score card,
63+
- Creator: who created the report (name of person or version of the software),
64+
- Check date: when the test was performed,
65+
- Test results: a mapping that maps testcase ids to results.
66+
- Log: free-form text that details the test run.
67+
68+
A _check script_ is a computer program that tests one or more testcases and produces a test report.
69+
70+
A _score card_ is a data structure that contains the following information:
71+
72+
- Subject: the name of the test subject,
73+
- Scope: the scope of all the testcases referred to in this score card,
74+
- Results: a mapping that maps testcase ids to results, where each result is additionally
75+
equipped with validity timespan (beginning at the time of the check and expiring according to
76+
the lifetime of the result).
77+
78+
## Regulation
79+
80+
Each standard must be decomposed into testcases. Each testcase should be "atomic" in the following two senses:
81+
82+
- it's clear what specific part of the standard is satisfied or not satisfied;
83+
- the testcase can be reused for multiple versions of the standards.
84+
85+
The latter criterion is a matter of engineering judgment, because it cannot be known in advance how a standard might evolve.
86+
87+
Each testcase id must be prefixed by `scs-XXXX-` where `XXXX` is the document id of the standard; an exception is possible in the rare case when a testcase applies to multiple standards.
88+
89+
A check script that can test multiple testcases should provide the option to select which testcases to run.
90+
91+
A list of reports from a certain time frame can be merged into an aggregate score card, provided that the following conditions are satisfied:
92+
93+
- for each testcase, the result must be taken from the most recent report containing that testcase,
94+
- if an additional report from the same time frame is known to exist, all its testcases must be contained in more recent reports from the list.

‎Standards/scs-0004-v1-achieving-certification.md‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -26,12 +26,6 @@ As operator, I want to obtain a certificate with the scope SCS-compatible IaaS o
2626

2727
For public clouds, the SCS certification assessment body can take on this task provided that suitable access to test subject is supplied.
2828

29-
The test suite is partitioned according to resource usage; the required test intervals depend on this classification:
30-
31-
- _light_: at least daily,
32-
- _medium_: at least weekly,
33-
- _heavy_: at least monthly.
34-
3529
3. If the desired certificate requires manual checks, then the operator MUST offer suitable documentation to the SCS certification assessment body. The operator MUST ensure that these checks are repeated once every quarter. In addition, the SCS certification assessment body reserves the right to occasionally verify documentation on premises.
3630

3731
4. Details on the standards achieved, as well as the current state and the history of all test and check results of the past 18 months will be displayed on a public webpage (henceforth, _certificate status page_) owned by SCS.

‎Standards/scs-0500-v1-scopes.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
---
2+
title: SCS Certificate Scopes
3+
type: Standard
4+
track: Scopes
5+
status: Draft
6+
description: |
7+
Basic definitions and regulations regarding SCS certificate scopes.
8+
---
9+
10+
## Definition
11+
12+
In essence, a certificate scope is a standard that has to be fulfilled in order for a certificate to be awarded.
13+
It usually refers to other standards in regards to actual subject matter — what properties the subject
14+
has to satisfy —, and it contains regulations regarding the testing regime.

‎Standards/scs-0501-v1-scs-compatible-iaas.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
---
2+
title: SCS-compatible IaaS
3+
type: Standard
4+
track: Scopes
5+
status: Deprecated
6+
stabilized_at: 2021-01-01
7+
deprecated_at: 2023-10-31
8+
description: |
9+
The certificate scope for level SCS-compatible and layer IaaS.
10+
---
11+
12+
## Introduction
13+
14+
This is v1 of the certificate scope _SCS-compatible IaaS_.
15+
16+
## Subject Matter
17+
18+
The following standards are included:
19+
20+
- OpenStack-powered Compute v2020.11
21+
- [scs-0100-v1: Flavor naming](https://docs.scs.community/standards/scs-0100-v1-flavor-naming)
22+
- [scs-0102-v1: Image metadata](https://docs.scs.community/standards/scs-0102-v1-image-metadata)
23+
24+
## Testing Regime
25+
26+
The following regulations apply for those certifications where regular testing is required.
27+
28+
In general, automated tests (where applicable) shall be performed at least weekly.
29+
30+
- Tests for OpenStack-powered Compute may be performed less frequently, but no fewer than once a year.

‎Standards/scs-0501-v2-scs-compatible-iaas.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
---
2+
title: SCS-compatible IaaS
3+
type: Standard
4+
track: Scopes
5+
status: Deprecated
6+
stabilized_at: 2023-03-23
7+
deprecated_at: 2023-11-30
8+
replaces:
9+
- scs-0501-v2-scs-compatible-iaas.md
10+
description: |
11+
The certificate scope for level SCS-compatible and layer IaaS.
12+
---
13+
14+
## Introduction
15+
16+
This is v2 of the certificate scope _SCS-compatible IaaS_.
17+
18+
## Subject Matter
19+
20+
The following standards are included:
21+
22+
- OpenStack-powered Compute v2022.11
23+
- [scs-0100-v2: Flavor naming](https://docs.scs.community/standards/scs-0100-v2-flavor-naming)
24+
- [scs-0102-v1: Image metadata](https://docs.scs.community/standards/scs-0102-v1-image-metadata)
25+
26+
## Testing Regime
27+
28+
The following regulations apply for those certifications where regular testing is required.
29+
30+
In general, automated tests (where applicable) shall be performed at least weekly.
31+
32+
- Tests for OpenStack-powered Compute may be performed less frequently, but no fewer than once a year.
33+
34+
## Version history
35+
36+
- v2 amends v1 by bumping the versions of OpenStack-powered Compute and of scs-0100.

‎Standards/scs-0501-v3-scs-compatible-iaas.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
---
2+
title: SCS-compatible IaaS
3+
type: Standard
4+
track: Scopes
5+
status: Deprecated
6+
stabilized_at: 2023-06-15
7+
deprecated_at: 2024-07-31
8+
replaces:
9+
- scs-0501-v2-scs-compatible-iaas.md
10+
description: |
11+
The certificate scope for level SCS-compatible and layer IaaS.
12+
---
13+
14+
## Introduction
15+
16+
This is v3.1 of the certificate scope _SCS-compatible IaaS_.
17+
18+
Change log:
19+
20+
- v3.1 bumps [scs-0100-v3](https://docs.scs.community/standards/scs-0100-v3-flavor-naming) from v3.0 to v3.1
21+
22+
## Subject Matter
23+
24+
The following standards are included:
25+
26+
- OpenStack-powered Compute v2022.11
27+
- [scs-0102-v1: Image metadata](https://docs.scs.community/standards/scs-0102-v1-image-metadata)
28+
- [scs-0100-v3: Flavor naming (v3.1)](https://docs.scs.community/standards/scs-0100-v3-flavor-naming)
29+
30+
## Testing Regime
31+
32+
The following regulations apply for those certifications where regular testing is required.
33+
34+
In general, automated tests (where applicable) shall be performed at least weekly.
35+
36+
- Tests for OpenStack-powered Compute may be performed less frequently, but no fewer than once a year.
37+
38+
## Previous Versions
39+
40+
- v3 amends v2 by bumping the version of scs-0100.

‎Standards/scs-0501-v4-scs-compatible-iaas.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
---
2+
title: SCS-compatible IaaS
3+
type: Standard
4+
track: Scopes
5+
status: Deprecated
6+
stabilized_at: 2024-02-28
7+
deprecated_at: 2025-07-01
8+
replaces:
9+
- scs-0501-v3-scs-compatible-iaas.md
10+
description: |
11+
The certificate scope for level SCS-compatible and layer IaaS.
12+
---
13+
14+
## Introduction
15+
16+
This is v4 of the certificate scope _SCS-compatible IaaS_.
17+
18+
## Subject Matter
19+
20+
Standards that were already included in v3.1:
21+
22+
- OpenStack-powered Compute v2022.11
23+
- [scs-0100-v3: Flavor naming (v3.1)](https://docs.scs.community/standards/scs-0100-v3-flavor-naming)
24+
- [scs-0102-v1: Image metadata](https://docs.scs.community/standards/scs-0102-v1-image-metadata)
25+
26+
Standards new with v4:
27+
28+
- [scs-0101-v1: Entropy](https://docs.scs.community/standards/scs-0102-v1-image-metadata)
29+
- [scs-0103-v1: Standard flavors](https://docs.scs.community/standards/scs-0103-v1-standard-flavors)
30+
- [scs-0104-v1: Standard images](https://docs.scs.community/standards/scs-0104-v1-standard-images) with [spec file](https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/iaas/scs-0104-v1-images.yaml)
31+
32+
## Testing Regime
33+
34+
The following regulations apply for those certifications where regular testing is required.
35+
36+
In general, automated tests (where applicable) shall be performed at least weekly.
37+
38+
- Tests for OpenStack-powered Compute may be performed less frequently, but no fewer than once a year.
39+
40+
## Previous Versions
41+
42+
- v4 extends v3 by including new standards as listed above.

‎Standards/scs-0501-v5-scs-compatible-iaas.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
---
2+
title: SCS-compatible IaaS
3+
type: Standard
4+
track: Scopes
5+
status: Stable
6+
stabilized_at: 2024-12-19
7+
replaces:
8+
- scs-0501-v4-scs-compatible-iaas.md
9+
description: |
10+
The certificate scope for level SCS-compatible and layer IaaS.
11+
---
12+
13+
## Introduction
14+
15+
This is v5.1 of the certificate scope _SCS-compatible IaaS_.
16+
17+
Change log:
18+
19+
- v5.1 adds [scs-0123-v1: Mandatory and Supported IaaS Services](https://docs.scs.community/standards/scs-0123-v1-mandatory-and-supported-IaaS-services), which was intended to be included (as witnessed by the meeting minutes), but was inadvertently omitted.
20+
21+
## Subject Matter
22+
23+
Standards that were already included in v4:
24+
25+
- [scs-0128-v1: SCS end-to-end testing (formerly OpenStack-powered Compute)](https://docs.scs.community/standards/scs-0128-v1-e2e-testing)
26+
- [scs-0100-v3: Flavor naming (v3.1)](https://docs.scs.community/standards/scs-0100-v3-flavor-naming)
27+
- [scs-0101-v1: Entropy](https://docs.scs.community/standards/scs-0102-v1-image-metadata)
28+
- [scs-0102-v1: Image metadata](https://docs.scs.community/standards/scs-0102-v1-image-metadata)
29+
- [scs-0103-v1: Standard flavors](https://docs.scs.community/standards/scs-0103-v1-standard-flavors)
30+
31+
Standards changed with v5:
32+
33+
- [scs-0104-v1: Standard images](https://docs.scs.community/standards/scs-0104-v1-standard-images) with [new spec file](https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/iaas/scs-0104-v1-images-v5.yaml) instead of [former spec file](https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/iaas/scs-0104-v1-images.yaml)
34+
35+
Standards new with v5:
36+
37+
- [scs-0114-v1: Volume Types](https://docs.scs.community/standards/scs-0114-v1-volume-type-standard)
38+
- [scs-0115-v1: Default rules for security groups](https://docs.scs.community/standards/scs-0115-v1-default-rules-for-security-groups)
39+
- [scs-0116-v1: Key manager](https://docs.scs.community/standards/scs-0116-v1-key-manager-standard)
40+
- [scs-0117-v1: Volume backup](https://docs.scs.community/standards/scs-0117-v1-volume-backup-service)
41+
- [scs-0121-v1: Availability Zones](https://docs.scs.community/standards/scs-0121-v1-Availability-Zones-Standard)
42+
- [scs-0123-v1: Mandatory and Supported IaaS Services](https://docs.scs.community/standards/scs-0123-v1-mandatory-and-supported-IaaS-services)
43+
- [scs-0302-v1: Domain Manager Role](https://docs.scs.community/standards/scs-0302-v1-domain-manager-role)
44+
45+
## Testing Regime
46+
47+
The following regulations apply for those certifications where regular testing is required.
48+
49+
In general, automated tests (where applicable) shall be performed at least weekly.
50+
51+
- Tests for scs-0128-v1 may be performed less frequently, but no fewer than once a year.
52+
53+
## Previous Versions
54+
55+
- v5 extends v4 by including new standards as listed above,
56+
and it changes the spec for scs-0104-v1, updating standard images.

‎Standards/scs-0501-w1-scs-compatible-iaas-testing.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
title: "SCS-compatible IaaS: Testing Notes"
3+
type: Supplement
4+
track: Scopes
5+
supplements:
6+
- scs-0501-v5-scs-compatible-iaas.md
7+
---
8+
9+
## Testing notes
10+
11+
See [documentation of official test suite](https://github.com/SovereignCloudStack/standards/blob/main/Tests/README.md).

0 commit comments

Comments
 (0)