Skip to content

Commit ad94af6

Browse files
liulinCliulinC
authored
sync master to feature branch (#7053)
2 parents e7f51b6 + 7ca7274 commit ad94af6

57 files changed

Lines changed: 2658 additions & 585 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎doc/content/design/pool-certificates.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ revision: 2
66
status: released (22.6.0)
77
---
88

9+
This design is modified by [trusted-certificates.md](trusted-certificates.md).
10+
911
## Overview
1012

1113
Xenserver has used TLS-encrypted communications between xapi daemons in a pool since its first release.
@@ -353,22 +355,30 @@ This feature needs clients to behave differently when initiating pool joins, to
353355
Several alerts are introduced:
354356
* POOL_CA_CERTIFICATE_EXPIRING_30, POOL_CA_CERTIFICATE_EXPIRING_14, POOL_CA_CERTIFICATE_EXPIRING_07, POOL_CA_CERTIFICATE_EXPIRED: Similar to host certificates, now the user-installable pool's CA certificates are monitored for expiry dates and alerts are generated about them. The body for this type of message is:
355357

358+
```
356359
<body><message>The trusted TLS server certificate {is expiring soon|has expired}.</message><date>20210302T02:00:01Z</date></body>
360+
```
357361

358362
* HOST_INTERNAL_CERTIFICATE_EXPIRING_30, HOST_INTERNAL_CERTIFICATE_EXPIRING_14, HOST_INTERNAL_CERTIFICATE_EXPIRING_07, HOST_INTERNAL_CERTIFICATE_EXPIRED: Similar to host certificates, the newly-introduced hosts' internal server certificates are monitored for expiry dates and alerts are generated about them. The body for this type of message is:
359363

364+
```
360365
<body><message>The TLS server certificate for internal communications {is expiring soon|has expired}.</message><date>20210302T02:00:01Z</date></body>
366+
```
361367

362368
* TLS_VERIFICATION_EMERGENCY_DISABLED: The host is in emergency mode and is not enforcing tls verification anymore, the situation that forced the disabling must be fixed and the verification enabled ASAP.
363369

370+
```
364371
<body><host>HOST-UUID</host></body>
372+
```
365373

366374
* FAILED_LOGIN_ATTEMPTS: An hourly alert that contains the number of failed attempts and the 3 most common origins for these failed alerts. The body for this type of message is:
367375

376+
```
368377
<body>
369378
<total>35</total>
370379
<known><username>usr5</username><originator>origin5</originator><ip>5.4.3.2</ip><number>10</number><date>20200922T15:03:13Z</date></known>
371380
<known><username>usr4</username><useragent>UA</useragent><number>6</number><date>20200922T15:03:13Z</date></known>
372381
<known><useragent>UA</useragent><ip>4.3.2.1</ip><number>4</number><date>20200922T14:57:11Z</date></known>
373382
<unknown>10</unknown>
374383
</body>
384+
```

‎doc/content/design/trusted-certificates.md‎

Lines changed: 111 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -2,16 +2,16 @@
22
title: Trusted certificates for identity validation in TLS connections
33
layout: default
44
design_doc: true
5-
revision: 1
5+
revision: 2
66
status: draft
77
---
88

99
# Overview
1010

1111
In various use cases, TLS connections are established on the host on which XAPI runs.
1212
When establishing a TLS connection, the peer identity needs to be validated.
13-
This is done using either a root CA certificate to perform certificate chain validation, or a known peer certificate for validation with certificate pinning.
14-
The root CA certificates and peer certificates involved in this process are referred to as trusted certificates.
13+
This is done using either a root CA certificate to perform certificate chain validation, or a pinned certificate for validation with certificate pinning.
14+
The root CA certificates and pinned certificates involved in this process are referred to as trusted certificates.
1515
When a trusted certificate is installed, the local endpoint can validate the peer identity during TLS connection establishment.
1616
Certificate chain validation is a general-purpose, standards-based approach but requires additional steps, such as getting the peer's certificate signed by a CA.
1717
In contrast, certificate pinning offers a quicker way to set up trust in some cases without the overhead of CA signing.
@@ -21,13 +21,13 @@ This allows the use case to start in quicker and easier way without prior CA sig
2121

2222
As the unified API for the whole system, XAPI also exposes interfaces for users to install and manage trusted certificates that are used by system components for different purposes.
2323

24-
The base design described in [pool-certificates.md](https://github.com/minglumlu/xen-api/blob/5d1ea1520825d502c57a90a02db476cd7d6a9132/doc/content/design/pool-certificates.md) defines the database, API, and trust store in the filesystem for managing trusted certificates.
24+
The base design described in [pool-certificates.md](pool-certificates.md) defines the database, API, and trust store in the filesystem for managing trusted certificates.
2525
This document introduces the following enhancements to that design:
2626

27-
* Explicit separation of root CA certificates and peer certificates:
27+
* Explicit separation of root CA certificates and pinned certificates:
2828
In the base design, both certificate types share the same database schema, APIs, and are stored together in a single bundle file.
2929
This makes it difficult to determine the appropriate validation approach based on the certificate type.
30-
The improvement introduces a type value to separate root CA certificates and peer certificates explicitly.
30+
The improvement introduces a type value to separate root CA certificates and pinned certificates explicitly.
3131

3232
* Add a "purpose" attribute for trusted certificates:
3333
According to the base design, only certificates used for internal TLS connections among XAPI processes within a pool are stored separately.
@@ -49,13 +49,13 @@ This case benefits from the improvements introduced in this design as well.
4949
## Database schema
5050
The *Certificate* class in database is defined to represent general certificates, including trusted certificates.
5151
One existing class field "type" supports the following enumeration values:
52-
* "ca": trusted certificates including both root CA and peer.
52+
* "ca": trusted certificates including both root CA and pinned.
5353
* "host": identity certificate of a host for communication with entities outside the pool.
5454
* "host_internal": identity certificate of a host for communication with other pool members.
5555

5656
Two improvements in this design:
57-
* A new value "peer" is introduced in this design so that the existing "ca" now represents trusted root CA only.
58-
The new "peer" will represent trusted peer certificates.
57+
* A new value "pinned" is introduced in this design so that the existing "ca" now represents trusted root CA only.
58+
The new "pinned" will represent trusted pinned certificates.
5959

6060
* A new enumeration type "purpose" is introduced to indicate the intended usage of a trusted certificate.
6161
A new *Certificate* class field "purpose" (a set of values of enumeration type "purpose") will be added to represent all applicable purposes of a trusted certificate.
@@ -79,27 +79,122 @@ For the same reason, "pool.uninstall_ca_certificate" will also be deprecated.
7979
This is a new API introduced in this design with its arguments being defined as:
8080
* session (ref session_id): reference to a valid session;
8181
* self (ref Pool): reference to the pool;
82-
* ca (boolean): the trusted certificate is a root CA certificate used to verify a chain (true), or a peer certificate used for certificate pinning (false);
82+
* ca (boolean): the trusted certificate is a root CA certificate used to verify a chain (true), or a pinned certificate used for certificate pinning (false);
8383
* cert (string): the trusted certificate in PEM format;
8484
* purpose (string list): the purposes of the trusted certificate.
8585

8686
This new API is used to install trusted certificate.
8787
When *purpose* is an empty set, it stands for a root CA certificate for general purpose.
88-
The *purpose* can not be an empty set when the *ca* is false, because each peer certificate is specific to a single server and therefore unsuitable for a shared trusted certificate for general purpose.
88+
The *purpose* can not be an empty set when the *ca* is false, because each pinned certificate is specific to a single server and therefore unsuitable for a shared trusted certificate for general purpose.
8989

9090
It returns *void* when succeed. Otherwise, return corresponding API error.
9191

9292
### pool.uninstall_trusted_certificate
9393
This is a new API introduced in this design to uninstall a trusted certificate with its arguments being defined as:
9494
* session (ref session_id): reference to a valid session;
9595
* certificate (ref Certificate): reference to the trusted certificate;
96-
* force (bool): remove the database entry even if the file doesn't exist.
9796

9897
It returns *void* when succeed. Otherwise, return corresponding API error.
9998

10099
### pool.join
101-
Prior to this design, trusted certificates are exchanged between the pool and the joining host during the pre‑join phase.
102-
This design preserves that behavior to ensure the joiner works correctly both before and after joining the pool.
100+
According to the base design, trusted certificates are exchanged between the pool and the joining host during the pre‑join phase.
101+
This design basically preserves that behavior to ensure the joiner works correctly both before and after joining the pool.
102+
However, a number of modifications have been introduced compared with the base design.
103+
104+
~~~mermaid
105+
106+
sequenceDiagram
107+
participant clnt as Client
108+
participant join as Joiner
109+
participant coor as Coordinator
110+
participant memb as Member
111+
clnt->>join: Pool.join coordinator_ip coordinator_username coordinator_password
112+
join->>coor:login_with_password rpc_no_verify coordinator_ip coordinator_username coordinator_password
113+
coor-->>join:
114+
115+
Note over join: pre_join_checks
116+
rect rgba(0,0,0,0.05)
117+
join->>join: assert_tls_verification_matches
118+
alt fails
119+
Note over join: interrupt join, raise error
120+
end
121+
end
122+
123+
Note over join: exchnage trusted intra-pool host identity certificates
124+
rect rgba(0,0,0,0.05)
125+
join->>coor: Pool.exchange_certificates_on_join <Joiner's trusted host identity cert>
126+
coor->>coor: Cert_distrib.exchange_certificates_with_joiner start
127+
coor->>memb: Host.cert_distrib_atom Write
128+
memb-->>coor:
129+
coor->>coor: Cert_distrib.get_local_pool_certs
130+
coor-->>coor: Cert_distrib.exchange_certificates_with_joiner done
131+
coor-->>join: <trusted host identity certs in pool>
132+
join->>join: Cert_distrib.import_joining_pool_certs <trusted host identity certs in pool>
133+
end
134+
135+
join->>coor:login_with_password rpc_verify coordinator_ip coordinator_username coordinator_password
136+
coor-->>join:
137+
138+
Note over join: exchange legacy ca certificates
139+
rect rgba(0,0,0,0.05)
140+
join->>coor: Pool.exchange_ca_certificates_on_join <Joiner's legacy ca certs>
141+
coor->>coor: Cert_distrib.exchange_ca_certificates_with_joiner
142+
coor-->>join: <legacy ca certs in pool>
143+
join->>join: Cert_distrib.import_joining_pool_ca_certificates <legacy ca certs in pool>
144+
end
145+
146+
Note over join: exchange trusted certificates
147+
rect rgba(0,0,0,0.05)
148+
join->>coor: Pool.exchange_trusted_certificates_on_join <Joiner's trusted certs>
149+
loop for every <trusted> in Joiner
150+
coor->>coor: Pool.install_trusted_certificate start
151+
coor->>memb: Host.cert_distrib_atom Write
152+
memb-->>coor:
153+
coor->>memb: Host.certificate_sync
154+
memb-->>coor:
155+
coor-->>coor: Pool.install_trusted_certificate done
156+
end
157+
coor->>coor: Cert_distrib.collect_trusted_certs
158+
coor-->>join: <trusted certs in pool>
159+
loop for every <trusted> in pool
160+
join->>join: Pool.install_trusted_certificate
161+
end
162+
end
163+
164+
Note over join: exchange CRLs
165+
rect rgba(0,0,0,0.05)
166+
join->>coor: Pool.exchange_crls_on_join <Joiner's CRLs>
167+
loop for every <CRL> in Joiner
168+
coor->>coor: Pool.crl_install start
169+
coor->>memb: Host.cert_distrib_atom Write
170+
memb-->>coor:
171+
coor->>memb: Host.certificate_sync
172+
memb-->>coor:
173+
coor-->>coor: Pool.crl_install done
174+
end
175+
coor->>coor: Cert_distrib.collect_crls
176+
coor-->>join: <CLRs in pool>
177+
loop for every <CRL> in pool
178+
join->>join: Pool.crl_install
179+
end
180+
end
181+
182+
join->>coor: Host.add joiner
183+
coor-->>join:
184+
185+
join->>join: restart_as_slave
186+
187+
Note over join: Copy all certificates from coordinator
188+
rect rgba(0,0,0,0.05)
189+
join->>coor: Host.copy_primary_host_certs
190+
coor->>join: Host.cert_distrib_atom Write
191+
join-->>coor:
192+
coor->>join: Host.cert_distrib_atom GenBundle
193+
join-->>coor:
194+
coor-->>join: return from Host.copy_primary_host_certs
195+
end
196+
197+
~~~
103198

104199
### pool.eject
105200
The trusted certificates will be removed from any host which is being eject from the pool.
@@ -130,10 +225,10 @@ The stores for the certificates installed via "pool.install_trusted_certificate"
130225
| Name | Filesystem location | Used for |
131226
| ---- | ------------------- | -------- |
132227
| Trusted General CA | /etc/trusted-certs/ca-general/ | Trusted root CA certificates that users can install to validate a peer’s identity when establishing a TLS connection for general purpose.
133-
| Trusted Peer | /etc/trusted-certs/peer-\<PURPOSE\>/ | Trusted peer certificates that users can install to validate a peer’s identity when establishing a TLS connection for \<PURPOSE\>.
228+
| Trusted Peer | /etc/trusted-certs/pinned-\<PURPOSE\>/ | Trusted pinned certificates that users can install to validate a peer’s identity when establishing a TLS connection for \<PURPOSE\>.
134229
| Trusted CA | /etc/trusted-certs/ca-\<PURPOSE\>/ | Trusted root CA certificates that users can install to validate a peer’s identity when establishing a TLS connection for \<PURPOSE\>.
135230
| General Bundle | /etc/trusted-certs/ca-bundle-general.pem | Bundle of trusted root CA certificates under /etc/trusted-certs/ca-general/ to verify a peer's identity when establishing a TLS connection for general purpose.
136-
| Peer Bundle | /etc/trusted-certs/peer-bundle-\<PURPOSE\>.pem | Bundle of trusted peer certificates under /etc/trusted-certs/peer-\<PURPOSE\>/ to verify a peer's identity when establishing a TLS connection for \<PURPOSE\>.
231+
| Peer Bundle | /etc/trusted-certs/pinned-bundle-\<PURPOSE\>.pem | Bundle of trusted pinned certificates under /etc/trusted-certs/pinned-\<PURPOSE\>/ to verify a peer's identity when establishing a TLS connection for \<PURPOSE\>.
137232
| CA Bundle | /etc/trusted-certs/ca-bundle-\<PURPOSE\>.pem | Bundle of trusted root CA certificates under /etc/trusted-certs/ca-\<PURPOSE\>/ to verify a peer's identity when establishing a TLS connection for \<PURPOSE\>.
138233

139234
The filesystem location is derived from the \<PURPOSE\>. Each \<PURPOSE\> string corresponds to a predefined value of the "purpose" type in the database, implemented as predefined constants.

‎ocaml/alerts/certificate/certificate_check.ml‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ type cert =
55
| CA of API.ref_Certificate * API.datetime
66
| Host of API.ref_host * API.datetime
77
| Internal of API.ref_host * API.datetime
8+
| Pinned of API.ref_Certificate * API.datetime
89

910
let get_certificates rpc session_id =
1011
XenAPI.Certificate.get_all_records ~rpc ~session_id
@@ -22,14 +23,18 @@ let get_certificates rpc session_id =
2223
)
2324
| `ca ->
2425
CA (cert_ref, certificate.API.certificate_not_after)
26+
| `pinned ->
27+
Pinned (cert_ref, certificate.API.certificate_not_after)
2528

2629
let certificate_description = function
2730
| Host _ ->
2831
"The TLS server certificate"
2932
| Internal _ ->
3033
"The internal TLS server certificate"
3134
| CA _ ->
32-
"The CA pool certificate"
35+
"The pool-wide trusted root CA certificate"
36+
| Pinned _ ->
37+
"The pool-wide trusted pinned leaf certificate"
3338

3439
let alert_conditions = function
3540
| Host _ ->
@@ -53,16 +58,25 @@ let alert_conditions = function
5358
; (14, Api_messages.pool_ca_certificate_expiring_14)
5459
; (30, Api_messages.pool_ca_certificate_expiring_30)
5560
]
61+
| Pinned _ ->
62+
[
63+
(0, Api_messages.pool_pinned_certificate_expired)
64+
; (7, Api_messages.pool_pinned_certificate_expiring_07)
65+
; (14, Api_messages.pool_pinned_certificate_expiring_14)
66+
; (30, Api_messages.pool_pinned_certificate_expiring_30)
67+
]
5668

5769
let alert_message_cls_and_obj_uuid rpc session_id cert =
5870
match cert with
5971
| Host (host, _) | Internal (host, _) ->
6072
(`Host, XenAPI.Host.get_uuid ~rpc ~session_id ~self:host)
6173
| CA (cert, _) ->
6274
(`Certificate, XenAPI.Certificate.get_uuid ~rpc ~session_id ~self:cert)
75+
| Pinned (cert, _) ->
76+
(`Certificate, XenAPI.Certificate.get_uuid ~rpc ~session_id ~self:cert)
6377

6478
let get_expiry = function
65-
| Host (_, exp) | Internal (_, exp) | CA (_, exp) ->
79+
| Host (_, exp) | Internal (_, exp) | CA (_, exp) | Pinned (_, exp) ->
6680
exp
6781

6882
let alert rpc session_id =

‎ocaml/alerts/certificate/certificate_check.mli‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ type cert =
2121
| CA of API.ref_Certificate * API.datetime
2222
| Host of API.ref_host * API.datetime
2323
| Internal of API.ref_host * API.datetime
24+
| Pinned of API.ref_Certificate * API.datetime
2425

2526
val certificate_description : cert -> string
2627

‎ocaml/idl/datamodel_certificate.ml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,9 +30,16 @@ let certificate_type =
3030
; ( "host_internal"
3131
, "Certificate that identifies a single host to other pool members"
3232
)
33+
; ("pinned", "Pinned leaf certificate that is trusted by the whole pool")
3334
]
3435
)
3536

37+
let certificate_purpose =
38+
Enum
39+
( "certificate_purpose"
40+
, [("licensing", "Trusted certificates that are for licensing purpose.")]
41+
)
42+
3643
let t =
3744
create_obj ~name:_certificate
3845
~descr:"An X509 certificate used for TLS connections" ~doccomments:[]
@@ -75,5 +82,8 @@ let t =
7582
; field ~qualifier:StaticRO ~lifecycle:[] ~ty:String "fingerprint_sha1"
7683
~default_value:(Some (VString ""))
7784
"The certificate's SHA1 fingerprint / hash"
85+
; field ~qualifier:StaticRO ~lifecycle:[] ~ty:(Set certificate_purpose)
86+
"purpose" ~default_value:(Some (VSet []))
87+
"The purposes of the certificate"
7888
]
7989
~messages:[] ()

‎ocaml/idl/datamodel_common.ml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ open Datamodel_roles
1010
to leave a gap for potential hotfixes needing to increment the schema version.*)
1111
let schema_major_vsn = 5
1212

13-
let schema_minor_vsn = 901
13+
let schema_minor_vsn = 902
1414

1515
(* Historical schema versions just in case this is useful later *)
1616
let rio_schema_major_vsn = 5

‎ocaml/idl/datamodel_errors.ml‎

Lines changed: 28 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -895,6 +895,13 @@ let _ =
895895
"The host joining the pool has different CA certificates from the pool \
896896
coordinator while using the same name, uninstall them and try again."
897897
() ;
898+
error Api_errors.pool_joining_host_trusted_certificates_conflict
899+
["ref_in_pool"; "ref_on_host"]
900+
~doc:
901+
"The joining host has a trusted certificate identical to one on the pool \
902+
coordinator but with different purpose. Uninstall it then install it on \
903+
the host again with the pool-compatible purpose, and try again."
904+
() ;
898905
error Api_errors.pool_joining_sm_features_incompatible
899906
["pool_sm_ref"; "candidate_sm_ref"]
900907
~doc:
@@ -1685,6 +1692,8 @@ let _ =
16851692
~doc:"The specified certificate does not exist." () ;
16861693
error Api_errors.certificate_already_exists ["name"]
16871694
~doc:"A certificate already exists with the specified name." () ;
1695+
error Api_errors.trusted_certificate_already_exists ["fingerprint"]
1696+
~doc:"A trusted certificate already exists with the same purpose." () ;
16881697
error Api_errors.certificate_name_invalid ["name"]
16891698
~doc:"The specified certificate name is invalid." () ;
16901699
error Api_errors.certificate_corrupt ["name"]
@@ -1711,7 +1720,7 @@ let _ =
17111720
() ;
17121721

17131722
error Api_errors.server_certificate_invalid []
1714-
~doc:"The provided certificate is not in a PEM-encoded X509." () ;
1723+
~doc:"The provided certificate is not in a PEM-encoded X509 format." () ;
17151724
error Api_errors.server_certificate_key_mismatch []
17161725
~doc:
17171726
"The provided key does not match the provided certificate's public key."
@@ -1727,9 +1736,26 @@ let _ =
17271736
() ;
17281737

17291738
error Api_errors.server_certificate_chain_invalid []
1730-
~doc:"The provided intermediate certificates are not in a PEM-encoded X509."
1739+
~doc:
1740+
"The provided intermediate certificates are not in a PEM-encoded X509 \
1741+
format."
17311742
() ;
17321743

1744+
error Api_errors.not_trusted_certificate ["ref"]
1745+
~doc:"The provided certificate is not a trusted certificate." () ;
1746+
1747+
error Api_errors.certificate_lacks_purpose []
1748+
~doc:"No purpose is specified for the provided certificate." () ;
1749+
1750+
error Api_errors.trusted_certificate_expired ["now"; "not_after"]
1751+
~doc:"The provided certificate has expired." () ;
1752+
1753+
error Api_errors.trusted_certificate_not_valid_yet ["now"; "not_before"]
1754+
~doc:"The provided certificate is not valid yet." () ;
1755+
1756+
error Api_errors.trusted_certificate_invalid []
1757+
~doc:"The provided certificate is not in a PEM-encoded X509 format." () ;
1758+
17331759
error Api_errors.vmpp_has_vm []
17341760
~doc:"There is at least one VM assigned to this protection policy." () ;
17351761
error Api_errors.vmpp_archive_more_frequent_than_backup []

0 commit comments

Comments
 (0)