diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 489b5cb53..1c2cc655e 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -8,6 +8,14 @@ updates: time: "13:00" open-pull-requests-limit: 10 +# Raise PRs for npm (mermaid CDN version tracking) +- package-ecosystem: npm + directory: "/" + schedule: + interval: weekly + time: "13:00" + open-pull-requests-limit: 5 + # Maintain dependencies for GitHub Actions - package-ecosystem: github-actions directory: "/" diff --git a/_data/nav/openvox_8x.yml b/_data/nav/openvox_8x.yml index 1f7038321..d278590b2 100644 --- a/_data/nav/openvox_8x.yml +++ b/_data/nav/openvox_8x.yml @@ -418,7 +418,7 @@ - text: Details about OpenVox's internals items: - text: Agent/server HTTPS communications - link: "subsystem_agent_master_comm.html" + link: "subsystem_agent_server_comm.html" - text: Catalog compilation link: "subsystem_catalog_compilation.html" - text: Experimental features diff --git a/_includes/jekyll_vitepress/head_end.html b/_includes/jekyll_vitepress/head_end.html index ed704f827..bacba237c 100644 --- a/_includes/jekyll_vitepress/head_end.html +++ b/_includes/jekyll_vitepress/head_end.html @@ -10,3 +10,23 @@ }); }); + + diff --git a/docs/_openvox_8x/architecture.markdown b/docs/_openvox_8x/architecture.markdown index 9803c0979..9a880fcb6 100644 --- a/docs/_openvox_8x/architecture.markdown +++ b/docs/_openvox_8x/architecture.markdown @@ -3,43 +3,54 @@ layout: default title: "Overview of OpenVox's architecture" --- -[agent_unix]: ./services_agent_unix.html -[agent_win]: ./services_agent_windows.html -[https_walkthrough]: ./subsystem_agent_master_comm.html -[server_http]: ./http_api/http_api_index.html -[auth.conf]: ./config_file_auth.html -[catalog_compilation]: ./subsystem_catalog_compilation.html -[report handlers]: ./report.html -[lang_basics]: ./lang_summary.html -[apply]: ./services_apply.html -[openvox-db]: /openvoxdb/latest/ -[resource]: ./lang_resources.html +[agent_unix]: services_agent_unix.html +[agent_win]: services_agent_windows.html +[https_walkthrough]: subsystem_agent_server_comm.html +[server_http]: http_api/http_api_index.html +[auth.conf]: config_file_auth.html +[catalog_compilation]: subsystem_catalog_compilation.html +[report handlers]: report.html +[lang_basics]: lang_summary.html +[apply]: services_apply.html +[OpenVoxDB]: /openvoxdb/latest/ +[resource]: lang_resources.html [OpenVox Server]: /openvox-server/latest/ -You can configure systems with OpenVox either in a client-server architecture, using the **OpenVox agent** and **OpenVox server** applications, or in a stand-alone architecture, using the **OpenVox apply** application. +You can configure systems with OpenVox either in a client-server architecture, using the +**OpenVox agent** and **OpenVox server** applications, or in a stand-alone architecture, using +the **OpenVox apply** application. -{:.concept} ## Catalogs -A catalog is a document that describes the desired system state for one specific computer. It lists all of the resources that need to be managed, as well as any dependencies between those resources. +A catalog is a document that describes the desired system state for one specific computer. It +lists all of the resources that need to be managed, as well as any dependencies between those +resources. OpenVox configures systems in two stages: 1. Compile a catalog. 2. Apply the catalog. -To compile a catalog, OpenVox uses several sources of information. For more info, see the pages on [basics of the OpenVox language][lang_basics] and [catalog compilation][catalog_compilation]. +To compile a catalog, OpenVox uses several sources of information. For more info, see the pages +on [basics of the OpenVox language][lang_basics] and [catalog compilation][catalog_compilation]. -{:.concept} ## The agent-server architecture -When set up as an agent-server architecture, an OpenVox server controls the configuration information, and each managed agent node requests its own configuration catalog from the server. +When set up as an agent-server architecture, an OpenVox server controls the configuration +information, and each managed agent node requests its own configuration catalog from the server. -In this architecture, managed nodes run the **OpenVox agent** application, usually as a background or scheduled service. One or more servers run the **OpenVox server** application, [OpenVox Server][]. +In this architecture, managed nodes run the **OpenVox agent** application, usually as a +background or scheduled service. One or more servers run the **OpenVox server** application, +[OpenVox Server][]. -Periodically, each OpenVox agent sends facts to the OpenVox server, and requests a catalog. The server compiles and returns that node's catalog, using several sources of information it has access to. +Periodically, each OpenVox agent sends facts to the OpenVox server, and requests a catalog. The +server compiles and returns that node's catalog, using several sources of information it has +access to. -Once it receives a catalog, OpenVox agent applies it to the node by checking each [resource][] the catalog describes. If it finds any resources that are not in their desired state, it makes the changes necessary to correct them. Or, in no-op mode, it reports on what changes would have been done. +Once it receives a catalog, OpenVox agent applies it to the node by checking each [resource][] +the catalog describes. If it finds any resources that are not in their desired state, it makes +the changes necessary to correct them. Or, in no-op mode, it reports on what changes would have +been done. After applying the catalog, the agent sends a report to the OpenVox server. @@ -49,16 +60,22 @@ For more information, see: * [OpenVox Agent on Windows Systems][agent_win] * [OpenVox Server][] -{:.section} ### Communications and security OpenVox agent nodes and OpenVox servers communicate by HTTPS with client SSL verification. -The OpenVox Server provides an HTTP interface, with [various endpoints][server_http] available. When requesting or submitting anything to the server, the agent makes an HTTPS request to one of those endpoints. +The OpenVox Server provides an HTTP interface, with [various endpoints][server_http] available. +When requesting or submitting anything to the server, the agent makes an HTTPS request to one +of those endpoints. -Client-verified HTTPS means each server or agent must have an identifying SSL certificate. They each examine their counterpart's certificate to decide whether to allow an exchange of information. +Client-verified HTTPS means each server or agent must have an identifying SSL certificate. They +each examine their counterpart's certificate to decide whether to allow an exchange of +information. -OpenVox includes a built-in certificate authority for managing certificates. Agents can automatically request certificates through the server's HTTP API. You can use the **puppet ssl** command on an agent to inspect local certificates. The **puppetserver ca** command on the server manages new and existing certificates. Agents can then download the signed certificates. +OpenVox includes a built-in certificate authority for managing certificates. Agents can +automatically request certificates through the server's HTTP API. You can use the **puppet ssl** +command on an agent to inspect local certificates. The **puppetserver ca** command on the server +manages new and existing certificates. Agents can then download the signed certificates. For more information, see: @@ -67,32 +84,64 @@ For more information, see: * [The OpenVox server's auth.conf file][auth.conf] * [Background reference on SSL and HTTPS.](/docs/background/ssl/) -{:.concept} ## The stand-alone architecture -Alternatively, OpenVox can run in a stand-alone architecture, where each managed node has its own complete copy of your configuration info and compiles its own catalog. +Alternatively, OpenVox can run in a stand-alone architecture, where each managed node has its +own complete copy of your configuration info and compiles its own catalog. -In this architecture, managed nodes run the **OpenVox apply** application, usually as a scheduled task or cron job. You can also run it on demand for initial configuration of a server or for smaller configuration tasks. +In this architecture, managed nodes run the **OpenVox apply** application, usually as a +scheduled task or cron job. You can also run it on demand for initial configuration of a server +or for smaller configuration tasks. -Like the OpenVox server application, OpenVox apply needs access to several sources of configuration data, which it uses to compile a catalog for the node it is managing. +Like the OpenVox server application, OpenVox apply needs access to several sources of +configuration data, which it uses to compile a catalog for the node it is managing. -After OpenVox apply compiles the catalog, it immediately applies it by checking each [resource][] the catalog describes. If it finds any resources that are not in their desired state, it makes the changes necessary to correct them. Or, in no-op mode, it reports on what changes would have been needed. +After OpenVox apply compiles the catalog, it immediately applies it by checking each [resource][] +the catalog describes. If it finds any resources that are not in their desired state, it makes +the changes necessary to correct them. Or, in no-op mode, it reports on what changes would have +been needed. -After applying the catalog, OpenVox apply stores a report on disk. You can configure it to send reports to a central service. +After applying the catalog, OpenVox apply stores a report on disk. You can configure it to send +reports to a central service. For more information, see the documentation for [the OpenVox apply application][apply]. -{:.concept} ## Differences between agent-server and stand-alone -In general, OpenVox apply can do the same things as the combination of OpenVox agent and OpenVox server, but there are several trade-offs around security and the ease of certain tasks. - -If you don't have a preference, you should select the agent-server architecture. If you have questions, considering these trade-offs helps you make your decision. - -* **Principle of least privilege.** In agent-server OpenVox, each agent only gets its own configuration, and is unable to see how other nodes are configured. With OpenVox apply, it's impractical to do this, so every node has access to complete knowledge about how your site is configured. Depending on how you're configuring your systems, this can potentially raise the risks of horizontal privilege escalation. -* **Ease of centralized reporting and inventory.** Agents send reports to the OpenVox server by default, and the server can be configured with any number of [report handlers][] to pass these on to other services. You can also connect the master to [OpenVoxDB][], a powerful tool for querying inventory and activity data. OpenVox apply nodes handle their own information, so if you're using OpenVoxDB or sending reports to another service, _each_ node needs to be configured and authorized to connect to it. -* **Ease of updating configurations.** Only OpenVox servers have the Puppet modules, main manifests, and other data necessary for compiling catalogs. This means that when you need to update your systems' configurations, you only need to update content on one (or a few) OpenVox servers. In a decentralized `puppet apply` deployment, you'll need to sync new configuration code and data to every node. -* **CPU and memory usage on managed machines.** Since OpenVox agent doesn't compile its own catalogs, it uses fewer resources on the machines it manages, leaving them with more capacity for their designated tasks. -* **Need for a dedicated OpenVox server.** The OpenVox server takes on the performance load of compiling all catalogs, and it should usually be a dedicated machine with a fast processor, lots of RAM, and a fast disk. Not everybody wants to (or is able to) allocate that, and OpenVox apply can get around the need for it. -* **Need for good network connectivity.** Agents need to be able to reach the OpenVox server at a reliable hostname in order to configure themselves. If a system lives in a degraded or isolated network environment, you might want it to be more self-sufficient. -* **Security overhead.** Agents and servers use HTTPS to secure their communications and authenticate each other, and every system involved needs an SSL certificate. OpenVox includes a built-in CA to easily manage certificates, but it's even easier to not manage them at all. (Of course, you'll still need to manage security somehow, since you're probably using Rsync or something to update OpenVox content on every node.) +In general, OpenVox apply can do the same things as the combination of OpenVox agent and OpenVox +server, but there are several trade-offs around security and the ease of certain tasks. + +If you don't have a preference, you should select the agent-server architecture. If you have +questions, considering these trade-offs helps you make your decision. + +* **Principle of least privilege.** In agent-server OpenVox, each agent only gets its own + configuration, and is unable to see how other nodes are configured. With OpenVox apply, it's + impractical to do this, so every node has access to complete knowledge about how your site is + configured. Depending on how you're configuring your systems, this can potentially raise the + risks of horizontal privilege escalation. +* **Ease of centralized reporting and inventory.** Agents send reports to the OpenVox server by + default, and the server can be configured with any number of [report handlers][] to pass these + on to other services. You can also connect the server to [OpenVoxDB][], a powerful tool for + querying inventory and activity data. OpenVox apply nodes handle their own information, so if + you're using [OpenVoxDB][] or sending reports to another service, _each_ node needs to be + configured and authorized to connect to it. +* **Ease of updating configurations.** Only OpenVox servers have the Puppet modules, main + manifests, and other data necessary for compiling catalogs. This means that when you need to + update your systems' configurations, you only need to update content on one (or a few) OpenVox + servers. In a decentralized `puppet apply` deployment, you'll need to sync new configuration + code and data to every node. +* **CPU and memory usage on managed machines.** Since OpenVox agent doesn't compile its own + catalogs, it uses fewer resources on the machines it manages, leaving them with more capacity + for their designated tasks. +* **Need for a dedicated OpenVox server.** The OpenVox server takes on the performance load of + compiling all catalogs, and it should usually be a dedicated machine with a fast processor, + lots of RAM, and a fast disk. Not everybody wants to (or is able to) allocate that, and OpenVox + apply can get around the need for it. +* **Need for good network connectivity.** Agents need to be able to reach the OpenVox server at + a reliable hostname in order to configure themselves. If a system lives in a degraded or + isolated network environment, you might want it to be more self-sufficient. +* **Security overhead.** Agents and servers use HTTPS to secure their communications and + authenticate each other, and every system involved needs an SSL certificate. OpenVox includes a + built-in CA to easily manage certificates, but it's even easier to not manage them at all. (Of + course, you'll still need to manage security somehow, since you're probably using Rsync or + something to update OpenVox content on every node.) diff --git a/docs/_openvox_8x/images/agent-master-https-sequence-large.gif b/docs/_openvox_8x/images/agent-master-https-sequence-large.gif deleted file mode 100644 index c86990385..000000000 Binary files a/docs/_openvox_8x/images/agent-master-https-sequence-large.gif and /dev/null differ diff --git a/docs/_openvox_8x/images/agent-master-https-sequence-small.gif b/docs/_openvox_8x/images/agent-master-https-sequence-small.gif deleted file mode 100644 index 34cd5b478..000000000 Binary files a/docs/_openvox_8x/images/agent-master-https-sequence-small.gif and /dev/null differ diff --git a/docs/_openvox_8x/ssl_attributes_extensions.markdown b/docs/_openvox_8x/ssl_attributes_extensions.markdown index 5ea3e3d41..ac0334170 100644 --- a/docs/_openvox_8x/ssl_attributes_extensions.markdown +++ b/docs/_openvox_8x/ssl_attributes_extensions.markdown @@ -3,39 +3,52 @@ layout: default title: "SSL configuration: CSR attributes and certificate extensions" --- -[cert_request]: ./subsystem_agent_master_comm.html#check-for-keys-and-certificates -[csr_attributes]: ./configuration.html#csrattributes -[confdir]: ./configuration.html#confdir -[autosign_policy]: ./ssl_autosign.html#policy-based-autosigning -[autosign_basic]: ./ssl_autosign.html#basic-autosigning-autosignconf +[cert_request]: subsystem_agent_server_comm.html#check-for-keys-and-certificates +[csr_attributes]: configuration.html#csrattributes +[confdir]: configuration.html#confdir +[autosign_policy]: ssl_autosign.html#policy-based-autosigning +[autosign_basic]: ssl_autosign.html#basic-autosigning-autosignconf [puppet_oids]: #puppet-specific-registered-ids -[trusted_hash]: ./lang_facts_and_builtin_vars.html#trusted-facts -[oid_map]: ./config_file_oid_map.html +[trusted_hash]: lang_facts_and_builtin_vars.html#trusted-facts +[oid_map]: config_file_oid_map.html -When OpenVox agent nodes request their certificates, the certificate signing request (CSR) usually contains only their certname and the necessary cryptographic information. However, agents can also embed more data in their CSR. This extra data can be useful for [policy-based autosigning][autosign_policy] and adding new [trusted facts][trusted_hash]. +When OpenVox agent nodes request their certificates, the certificate signing request (CSR) usually +contains only their certname and the necessary cryptographic information. However, agents can also +embed more data in their CSR. This extra data can be useful for +[policy-based autosigning][autosign_policy] and adding new [trusted facts][trusted_hash]. Embedding additional data into CSRs is mostly useful when: * Large numbers of nodes are regularly created and destroyed as part of an elastic scaling system. * You are willing to build custom tooling to make certificate autosigning more secure and useful. -It might also be useful in deployments where Puppet is used to deploy private keys or other sensitive information, and you want extra control over which nodes receive this data. +It might also be useful in deployments where Puppet is used to deploy private keys or other +sensitive information, and you want extra control over which nodes receive this data. If your deployment doesn't match one of these descriptions, you might not need this feature. ## Timing: When data can be added to CSRs and certificates -When OpenVox agent starts the process of requesting a catalog, it first checks whether it has a valid signed certificate. If it does not, it generates a key pair, crafts a CSR, and submits it to the certificate authority (CA) OpenVox Server. The steps are [covered in more detail in the reference page about agent/master HTTPS traffic][cert_request]. +When OpenVox agent starts the process of requesting a catalog, it first checks whether it has a +valid signed certificate. If it does not, it generates a key pair, crafts a CSR, and submits it +to the certificate authority (CA) OpenVox Server. The steps are +[covered in more detail in the reference page about agent/server HTTPS traffic][cert_request]. -For all practical purposes, a certificate is locked and immutable as soon as it is signed. For any data to persist in the certificate, it has to be added to the CSR _before_ the CA signs the certificate. +For all practical purposes, a certificate is locked and immutable as soon as it is signed. For +any data to persist in the certificate, it has to be added to the CSR _before_ the CA signs the +certificate. -This means **any desired extra data must be present _before_ OpenVox agent attempts to request its catalog for the first time.** +This means **any desired extra data must be present _before_ OpenVox agent attempts to request +its catalog for the first time.** -Practically speaking, you should populate any extra data when provisioning the node. If you mess up, see [Recovering From Failed Data Embedding](#recovering-from-failed-data-embedding) below. +Practically speaking, you should populate any extra data when provisioning the node. If you mess +up, see [Recovering From Failed Data Embedding](#recovering-from-failed-data-embedding) below. ## Data location and format -Extra data for the CSR is read from the `csr_attributes.yaml` file in Puppet's [`confdir`][confdir]. (The location of this file can be changed with [the `csr_attributes` setting][csr_attributes].) +Extra data for the CSR is read from the `csr_attributes.yaml` file in Puppet's +[`confdir`][confdir]. (The location of this file can be changed with +[the `csr_attributes` setting][csr_attributes].) The `csr_attributes` file must be a YAML hash containing one or both of the following keys: @@ -44,92 +57,130 @@ The `csr_attributes` file must be a YAML hash containing one or both of the foll The value of each key must also be a hash, where: -* Each key is a valid [object identifier (OID)](http://en.wikipedia.org/wiki/Object_identifier) --- [Puppet-specific OIDs][puppet_oids] can optionally be referenced by short name instead of by numeric ID. -* Each value is an object that can be cast to a string (that is, numbers are allowed but arrays are not). +* Each key is a valid [object identifier (OID)](http://en.wikipedia.org/wiki/Object_identifier) + --- [Puppet-specific OIDs][puppet_oids] can optionally be referenced by short name instead of + by numeric ID. +* Each value is an object that can be cast to a string (that is, numbers are allowed but arrays + are not). -See the respective sections below for information about how each hash is used and recommended OIDs for each hash. +See the respective sections below for information about how each hash is used and recommended +OIDs for each hash. ## Custom attributes (transient CSR data) -**Custom Attributes** are pieces of data that are _only_ embedded in the CSR. The CA can use them when deciding whether to sign the certificate, but they are discarded after that and aren't transferred to the final certificate. +**Custom Attributes** are pieces of data that are _only_ embedded in the CSR. The CA can use +them when deciding whether to sign the certificate, but they are discarded after that and aren't +transferred to the final certificate. ### Default behavior -The `puppetserver ca list` command doesn't display custom attributes for pending CSRs, and [basic autosigning (autosign.conf)][autosign_basic] doesn't check them before signing. +The `puppetserver ca list` command doesn't display custom attributes for pending CSRs, and +[basic autosigning (autosign.conf)][autosign_basic] doesn't check them before signing. ### Configurable behavior -If you use [policy-based autosigning][autosign_policy], your policy executable receives the complete CSR in PEM format. The executable can extract and inspect the custom attributes, and use them when deciding whether to sign the certificate. +If you use [policy-based autosigning][autosign_policy], your policy executable receives the +complete CSR in PEM format. The executable can extract and inspect the custom attributes, and use +them when deciding whether to sign the certificate. -The simplest use is to embed a pre-shared key of some kind in the custom attributes. A policy executable can compare it to a list of known keys and autosign certificates for any pre-authorized nodes. +The simplest use is to embed a pre-shared key of some kind in the custom attributes. A policy +executable can compare it to a list of known keys and autosign certificates for any +pre-authorized nodes. -A more complex use might be to embed an instance-specific ID and write a policy executable that can check it against a list of your recently requested instances on a public cloud, like EC2 or GCE. +A more complex use might be to embed an instance-specific ID and write a policy executable that +can check it against a list of your recently requested instances on a public cloud, like EC2 or +GCE. -You can also sign requests using authorization extensions and the `--allow-authorization-extensions` flag for `puppetserver ca sign`. +You can also sign requests using authorization extensions and the `--allow-authorization-extensions` +flag for `puppetserver ca sign`. ### Manually checking for custom attributes in CSRs -You can check for custom attributes by using OpenSSL to dump a PEM-format CSR to text format. Do this by running: +You can check for custom attributes by using OpenSSL to dump a PEM-format CSR to text format. +Do this by running: ``` bash openssl req -noout -text -in .pem ``` -In the output, look for a section called "Attributes," which generally appears below the "Subject Public Key Info" block: +In the output, look for a section called "Attributes," which generally appears below the +"Subject Public Key Info" block: -``` +``` text Attributes: challengePassword :342thbjkt82094y0uthhor289jnqthpc2290 ``` ### Recommended OIDs for attributes -Custom attributes can use any public or site-specific OID, **with the exception of the OIDs used for core X.509 functionality.** This means you can't re-use existing OIDs for things like subject alternative names. +Custom attributes can use any public or site-specific OID, **with the exception of the OIDs used +for core X.509 functionality.** This means you can't re-use existing OIDs for things like +subject alternative names. -One useful OID is the "challengePassword" attribute --- `1.2.840.113549.1.9.7`. This is a rarely-used corner of X.509 that can easily be repurposed to hold a pre-shared key. The benefit of using this instead of an arbitrary OID is that it appears by name when using OpenSSL to dump the CSR to text; OIDs that `openssl req` can't recognize are displayed as numerical strings. +One useful OID is the "challengePassword" attribute --- `1.2.840.113549.1.9.7`. This is a +rarely-used corner of X.509 that can easily be repurposed to hold a pre-shared key. The benefit +of using this instead of an arbitrary OID is that it appears by name when using OpenSSL to dump +the CSR to text; OIDs that `openssl req` can't recognize are displayed as numerical strings. -You can also use the Puppet-specific OIDs [referenced below][puppet_oids] in the section on extension requests. +You can also use the Puppet-specific OIDs [referenced below][puppet_oids] in the section on +extension requests. ## Extension requests (permanent certificate data) -**Extension requests** are pieces of data that are _transferred to the final certificate_ (as **extensions**) when the CA signs the CSR. They persist as trusted, immutable data, that cannot be altered after the certificate is signed. +**Extension requests** are pieces of data that are _transferred to the final certificate_ (as +**extensions**) when the CA signs the CSR. They persist as trusted, immutable data, that cannot +be altered after the certificate is signed. They can also be used by the CA when deciding whether or not to sign the certificate. ### Default behavior -When signing a certificate, Puppet's CA tools transfer any extension requests into the final certificate. +When signing a certificate, Puppet's CA tools transfer any extension requests into the final +certificate. You can access certificate extensions in manifests as `$trusted[extensions][]`. -Any OIDs in the ppRegCertExt range ([see below][puppet_oids]) appear using their short names. By default, any other OIDs appear as plain dotted numbers, but you can use [the `custom_trusted_oid_mapping.yaml` file][oid_map] to assign short names to any other OIDs you use at your site. If you do, those OIDs will appear in `$trusted` as their short names instead of their full numerical OID. +Any OIDs in the ppRegCertExt range ([see below][puppet_oids]) appear using their short names. +By default, any other OIDs appear as plain dotted numbers, but you can use +[the `custom_trusted_oid_mapping.yaml` file][oid_map] to assign short names to any other OIDs +you use at your site. If you do, those OIDs will appear in `$trusted` as their short names +instead of their full numerical OID. See [the page on facts and special variables][trusted_hash] for more information about `$trusted`. Visibility of extensions is somewhat limited: -* The `puppetserver ca list` command _does not_ display custom attributes for any pending CSRs, and - [basic autosigning (autosign.conf)][autosign_basic] doesn't check them before signing. Either use - [policy-based autosigning][autosign_policy] or inspect CSRs manually with the `openssl` command (see below). -* The `puppet ssl show` command displays any extensions in the local node's signed certificate, under the "X509v3 extensions" section. +* The `puppetserver ca list` command _does not_ display custom attributes for any pending CSRs, + and [basic autosigning (autosign.conf)][autosign_basic] doesn't check them before signing. + Either use [policy-based autosigning][autosign_policy] or inspect CSRs manually with the + `openssl` command (see below). +* The `puppet ssl show` command displays any extensions in the local node's signed certificate, + under the "X509v3 extensions" section. -Puppet's authorization system (`auth.conf`) does not use certificate extensions, but [Puppet Server's authorization system](/puppetserver/latest/config_file_auth.html), which is based on `trapperkeeper-authorization`, can use extensions in the ppAuthCertExt OID range, and requires them for requests to write access rules. +Puppet's authorization system (`auth.conf`) does not use certificate extensions, but +[Puppet Server's authorization system](/puppetserver/latest/config_file_auth.html), which is +based on `trapperkeeper-authorization`, can use extensions in the ppAuthCertExt OID range, and +requires them for requests to write access rules. ### Configurable behavior -If you use [policy-based autosigning][autosign_policy], your policy executable receives the complete CSR in PEM format. The executable can extract and inspect the extension requests, and use them when deciding whether to sign the certificate. +If you use [policy-based autosigning][autosign_policy], your policy executable receives the +complete CSR in PEM format. The executable can extract and inspect the extension requests, and +use them when deciding whether to sign the certificate. ### Manually checking for extensions in CSRs and certificates -You can check for extension requests in a CSR by using OpenSSL to dump a PEM-format CSR to text format. Do this by running: +You can check for extension requests in a CSR by using OpenSSL to dump a PEM-format CSR to text +format. Do this by running: ``` bash openssl req -noout -text -in .pem ``` -In the output, look for a section called "Requested Extensions," which generally appears below the "Subject Public Key Info" and "Attributes" blocks: +In the output, look for a section called "Requested Extensions," which generally appears below +the "Subject Public Key Info" and "Attributes" blocks: -``` +``` text Requested Extensions: pp_uuid: .$ED803750-E3C7-44F5-BB08-41A04433FE2E @@ -139,15 +190,17 @@ Requested Extensions: .$342thbjkt82094y0uthhor289jnqthpc2290 ``` -Note that every extension is preceded by any combination of two characters (`.$` and `..` in the above example) that contain ASN.1 encoding information. Since OpenSSL is unaware of Puppet's custom extensions OIDs, it's unable to properly display the values. +Note that every extension is preceded by any combination of two characters (`.$` and `..` in the +above example) that contain ASN.1 encoding information. Since OpenSSL is unaware of Puppet's +custom extensions OIDs, it's unable to properly display the values. Any Puppet-specific OIDs (see below) appear as numeric strings when using OpenSSL. -You can check for extensions in a signed certificate by running `puppet ssl show` on the agent node -that holds the certificate. In the output, look for the "X509v3 extensions" section. Any of the -Puppet-specific registered OIDs (see below) appear as their descriptive names: +You can check for extensions in a signed certificate by running `puppet ssl show` on the agent +node that holds the certificate. In the output, look for the "X509v3 extensions" section. Any +of the Puppet-specific registered OIDs (see below) appear as their descriptive names: -``` +``` text X509v3 extensions: Netscape Comment: Puppet Ruby/OpenSSL Internal Certificate @@ -169,29 +222,45 @@ X509v3 extensions: ### Recommended OIDs for extensions -Extension request OIDs **must** be under the "ppRegCertExt" (`1.3.6.1.4.1.34380.1.1`), "ppPrivCertExt" (`1.3.6.1.4.1.34380.1.2`), or "ppAuthCertExt" (`1.3.6.1.4.1.34380.1.3`) OID arcs. +Extension request OIDs **must** be under the "ppRegCertExt" (`1.3.6.1.4.1.34380.1.1`), +"ppPrivCertExt" (`1.3.6.1.4.1.34380.1.2`), or "ppAuthCertExt" (`1.3.6.1.4.1.34380.1.3`) OID +arcs. -Puppet provides several registered OIDs (under "ppRegCertExt") for the most common kinds of extension information, a private OID range ("ppPrivCertExt") for site-specific extension information, and an OID range for safe authorization to Puppet Server ("ppAuthCertExt"). +Puppet provides several registered OIDs (under "ppRegCertExt") for the most common kinds of +extension information, a private OID range ("ppPrivCertExt") for site-specific extension +information, and an OID range for safe authorization to Puppet Server ("ppAuthCertExt"). There are several benefits to using the registered OIDs: -* You can reference them in `csr_attributes.yaml` with their short names instead of their numeric IDs. -* You can access them in `$trusted[extensions]` with their short names instead of their numeric IDs. -* When using Puppet tools to print certificate info, they will appear using their descriptive names instead of their numeric IDs. +* You can reference them in `csr_attributes.yaml` with their short names instead of their + numeric IDs. +* You can access them in `$trusted[extensions]` with their short names instead of their numeric + IDs. +* When using Puppet tools to print certificate info, they will appear using their descriptive + names instead of their numeric IDs. -The private range is available for any information you want to embed into a certificate that isn't already in wide use elsewhere. It is completely unregulated, and its contents are expected to be different in every Puppet deployment. +The private range is available for any information you want to embed into a certificate that +isn't already in wide use elsewhere. It is completely unregulated, and its contents are expected +to be different in every Puppet deployment. -You can use [the `custom_trusted_oid_mapping.yaml` file][oid_map] to set short names for any private extension OIDs you use. Note that this only enables the short names in the `$trusted[extensions]` hash. +You can use [the `custom_trusted_oid_mapping.yaml` file][oid_map] to set short names for any +private extension OIDs you use. Note that this only enables the short names in the +`$trusted[extensions]` hash. #### Puppet-specific registered IDs + {% include_relative _registered_oids.md %} + ## AWS attributes and extensions population example -You can use an automated script (possibly using cloud-init or a similar tool) to populate the `csr_attributes.yaml` file when you provision a node. +You can use an automated script (possibly using cloud-init or a similar tool) to populate the +`csr_attributes.yaml` file when you provision a node. -As an example, you can enter the following script into the "Configure Instance Details —> Advanced Details" section when provisioning a new node from the AWS EC2 dashboard: +As an example, you can enter the following script into the +"Configure Instance Details —> Advanced Details" section when provisioning a new node from the +AWS EC2 dashboard: ``` bash #!/bin/sh @@ -207,26 +276,33 @@ extension_requests: YAML ``` -Assuming your image has the `erb` binary available, this populates the attributes file with the AWS instance ID, image name, and a pre-shared key to use with policy-based autosigning. +Assuming your image has the `erb` binary available, this populates the attributes file with the +AWS instance ID, image name, and a pre-shared key to use with policy-based autosigning. ## Troubleshooting ### Recovering from failed data embedding -When first testing this feature, you might fail to embed the right information in a CSR or certificate and want to start over for your test nodes. (This is not really a problem once your provisioning system is changed to populate the data, but it can happen pretty easily when doing things manually.) +When first testing this feature, you might fail to embed the right information in a CSR or +certificate and want to start over for your test nodes. (This is not really a problem once your +provisioning system is changed to populate the data, but it can happen pretty easily when doing +things manually.) To start over, do the following: **On the test node:** * Turn off OpenVox agent, if it's running. -* Check whether a CSR is present; it will be at `$ssldir/certificate_requests/.pem`. If it exists, delete it. -* Check whether a certificate is present; it will be at `$ssldir/certs/.pem`. If it exists, delete it. +* Check whether a CSR is present; it will be at `$ssldir/certificate_requests/.pem`. + If it exists, delete it. +* Check whether a certificate is present; it will be at `$ssldir/certs/.pem`. + If it exists, delete it. **On the CA OpenVox Server:** -* Check whether a signed certificate exists; use `puppetserver ca list --all` to see the complete list. - If it exists, revoke and delete it with `puppetserver ca clean --certname `. -* Check whether a CSR for the node exists; it will be in `$ssldir/ca/requests/.pem`. If it exists, delete it. +* Check whether a signed certificate exists; use `puppetserver ca list --all` to see the + complete list. If it exists, revoke and delete it with `puppetserver ca clean --certname `. +* Check whether a CSR for the node exists; it will be in `$ssldir/ca/requests/.pem`. + If it exists, delete it. After you've done that, you can start over. diff --git a/docs/_openvox_8x/subsystem_agent_master_comm.markdown b/docs/_openvox_8x/subsystem_agent_master_comm.markdown deleted file mode 100644 index 21e775021..000000000 --- a/docs/_openvox_8x/subsystem_agent_master_comm.markdown +++ /dev/null @@ -1,118 +0,0 @@ ---- -title: "Subsystems: Agent/master HTTPS communications" -layout: default ---- - -[http_api]: ./http_api/http_api_index.html -[authconf]: ./config_file_auth.html -[facts]: ./lang_variables.html#facts-and-built-in-variables -[catalog]: ./lang_summary.html#compilation-and-catalogs -[file]: ./type.html#file -[static]: ./indirection.html#catalog -[keepalive_setting]: ./configuration.html#httpkeepalivetimeout - - - -OpenVox agent and OpenVox Server communicate via host-verified HTTPS. - -> **Note on verification:** If the agent does not yet have its own certificate, it will make several unverified requests before it can switch to verified mode. In these requests, the agent doesn't identify itself to the master and doesn't check the master's cert against the CA. In the descriptions below, assume every request is host-verified unless stated otherwise. - -The HTTPS endpoints Puppet uses are detailed in the [HTTP API reference][http_api]. Note that access to each individual endpoint is controlled by [auth.conf][authconf] on the master. - -## Persistent connections/Keep-alive - -When acting as an HTTPS client, Puppet will try to re-use connections in order to reduce TLS overhead, by sending `Connection: Keep-Alive` in the HTTP request. This helps improve performance for runs with dozens of HTTPS requests. - -Puppet will only cache verified HTTPS connections, so it excludes the unverified connections a new agent makes to request a new certificate. Puppet also will not cache connections when a custom HTTP connection class has been specified. (This is an esoteric use case that most users will never see.) - -You can use [the `http_keepalive_timeout` setting][keepalive_setting] to configure the keepalive duration. It must be shorter than the maximum keepalive allowed by the OpenVox Server web server. - -An HTTP server can disable persistent connections ([Apache example](http://httpd.apache.org/docs/current/mod/core.html#keepalive)). If so, Puppet will request that the connection be kept open as usual, but the server will decline by sending `Connection: close` in the HTTP response and Puppet will start a new connection for its next request. - - -## Diagram - -This flow diagram illustrates the pattern of agent-side checks and HTTPS requests to the OpenVox Server during a single Puppet run. - -[See below the image for a textual description of this process](#check-for-keys-and-certificates), which explains the illustrated steps in more detail. - -> **Note:** This diagram is out of date; it uses the HTTPS endpoints from Puppet 3.x. We will update this diagram sometime in the future. - -[![An illustration of the process described below -- this diagram contains no new content, but is simply a visual interpretation of everything from "check for keys and certificates" on down.](./images/agent-master-https-sequence-small.gif)](./images/agent-master-https-sequence-large.gif) - -## Check for keys and certificates - -1. Does the agent have a private key at `$ssldir/private_keys/.pem`? - * If no, generate one. -2. Does the agent have a copy of the CA certificate at `$ssldir/certs/ca.pem`? - * If no, fetch it. (Unverified GET request to `/certificate/ca`. Since the agent is retrieving the foundation for all future trust over an untrusted link, this could be vulnerable to MITM attacks, but it's also just a convenience; you can make this step unnecessary by distributing the CA cert as part of your server provisioning process, so that agents never ask for a CA cert over the network. If you do this, an attacker could temporarily deny Puppet service to brand new nodes, but would be unable to take control of them with a rogue OpenVox Server.) -3. Does the agent have a signed certificate at `$ssldir/certs/.pem`? - * If yes, skip the following section and continue to "request node object." - * (If it has a cert but it doesn't match the private key, bail with an error.) - -## Obtain a certificate (if necessary) - -Note that if the agent has submitted a certificate signing request, an admin user will need to run -`puppetserver ca sign --certname ` on the Puppet Server before the agent can fetch a signed -certificate. (Unless autosign is enabled.) Since incoming CSRs are unverified, you can use fingerprints -to prove them, by comparing `puppet agent --fingerprint` on the agent to `puppetserver ca list` on the Puppet Server. - -1. Try to fetch an already-signed certificate from the master. (Unverified GET request to `/puppet-ca/v1/certificate/`.) - * If it gets one, skip the rest of this section and continue to "request node object." - * (If it gets one that doesn't match the private key, bail with an error.) -2. Determine whether the agent has already requested a certificate signing: Look for `$ssldir/certificate_requests/.pem`. - * If this file exists, the agent will bail, assuming it needs user intervention on the master. If `waitforcert` is enabled, it will wait a few seconds and start this section over. -3. Double-check with the master whether the agent has already requested a certificate signing; the agent might have just lost the local copy of the request. (Unverified GET request to `/puppet-ca/v1/certificate_request/`.) - * If this request doesn't 404, the agent will bail, assuming it needs user intervention on the master. If `waitforcert` is enabled, it will wait a few seconds and start this section over. -4. If the agent has reached this step, it has never requested a certificate, so request one now. (Unverified PUT request to `/puppet-ca/v1/certificate_request/`.) -5. Return to the first step of this section, in case the master has autosign enabled; if it doesn't, the agent will end up bailing at step 2. - -## Request node object and switch environments - -1. Do a GET request to `/puppet/v3/node/`. - * If successful, read the environment from the node object. - * If the node object has one: In all subsequent requests during this run, use this environment instead of the one in the agent's config file. - * If unsuccessful, or if the node object had no environment set, continue using the environment from the agent's config file. - -> **Note:** This step was added in Puppet 3.0.0, to allow an ENC to centrally assign nodes to environments. The lenient failure mode is because many users' [auth.conf][authconf] files didn't allow access to node objects, since that rule was only added to the default auth.conf in Puppet 2.7.0 and many people don't update their config files for every upgrade. - -## Fetch plugins - -If `pluginsync` is enabled on the agent: - -1. Do a GET request to `/puppet/v3/file_metadatas/plugins` with `recurse=true` and `links=manage`. This is a special file server mountpoint that scans the `lib` directory of every module. Note the funky Rails-esque endpoint pluralization on `file_metadata`. -2. Check whether any of the discovered plugins need to be downloaded. - * If so, do a GET request to `/puppet/v3/file_content/plugins/` for each one. - -## Request catalog while submitting facts - -1. Do a POST request to `/puppet/v3/catalog/`, where the post data is all of the node's [facts][] encoded as JSON. Receive a compiled [catalog][] in return. - -> **Note:** This used to be a GET request with facts encoded as base64-encoded zlib-compressed JSON submitted as a URL parameter. This would sometimes cause failures with certain web servers and a large amount of facts, and was changed to a POST in Puppet 2.7.0. GETs should still work in Puppet 3 if something other than an agent tries one, but agents will use POSTs. -> -> Note also that submitting facts isn't necessarily logically bound to requesting a catalog, and could be done out of band on a different schedule; this is just the way Puppet happens to do it, because the original design assumptions were that relevant facts could change at any moment and you'd always want to guarantee the most recent data. - -## Make file source requests while applying catalog - -[File][] resources can specify file contents as either a `content` or `source` attribute. Content attributes go into the catalog, and OpenVox agent needs no additional data. Source attributes only put references into the catalog, and might require additional HTTPS requests. - -If you are using the normal compiler, then for each file source, OpenVox agent will: - -1. Do a GET request to `/puppet/v3/file_metadata/`. -2. Compare the metadata to the state of the file on disk. - * If it is in sync, move on to the next file source. - * If it is out of sync, do a GET request to `/puppet/v3/file_content/` for the current content. - -If you are using the [static compiler][static], all file metadata is embedded in the catalog. For each file source, OpenVox agent will: - -1. Compare the embedded metadata to the state of the file on disk. - * If it is in sync, move on to the next file source. - * If it is out of sync, do a GET request to `/puppet/v3/file_bucket_file/md5/` for the current content. - -Note that this is cheaper in terms of network traffic, but potentially more expensive during catalog compilation. Large amounts of files, especially recursive directories, will amplify the effect in both directions. - -## Submit report - -If `report` is enabled on the agent: - -1. Do a PUT request to `/puppet/v3/report/`. The content of the PUT should be a Puppet report object in YAML format. diff --git a/docs/_openvox_8x/subsystem_agent_server_comm.markdown b/docs/_openvox_8x/subsystem_agent_server_comm.markdown new file mode 100644 index 000000000..b3260436d --- /dev/null +++ b/docs/_openvox_8x/subsystem_agent_server_comm.markdown @@ -0,0 +1,149 @@ +--- +title: "Subsystems: Agent/server HTTPS communications" +layout: default +--- + +[http_api]: http_api/http_api_index.html +[authconf]: config_file_auth.html +[facts]: lang_variables.html#facts-and-built-in-variables +[catalog]: lang_summary.html#compilation-and-catalogs +[file]: type.html#file +[static]: indirection.html#catalog +[keepalive_setting]: configuration.html#httpkeepalivetimeout + +OpenVox agent and OpenVox Server communicate via mutually authenticated HTTPS using client +certificates. + +The available HTTPS endpoints are detailed in the [HTTP API reference][http_api]. Access to each +endpoint is controlled by [auth.conf][authconf] settings. + +## Persistent HTTP and HTTPS connections and Keep-Alive + +When acting as an HTTPS client, OpenVox reuses connections by sending `Connection: Keep-Alive` +in HTTP requests. This reduces transport layer security (TLS) overhead, improving performance +for runs with dozens of HTTPS requests. + +You can configure the Keep-Alive duration using the [`http_keepalive_timeout` +setting][keepalive_setting], but it must be shorter than the maximum keepalive allowed by the +OpenVox Server web server. + +OpenVox caches HTTP connections and verified HTTPS connections. If you specify a custom HTTP +connection class, OpenVox does not cache the connection. + +OpenVox always requests that a connection be kept open, but the server can choose to close the +connection by sending `Connection: close` in the HTTP response. If that occurs, OpenVox does not +cache the connection and starts a new connection for its next request. + +## Diagram + +This sequence diagram illustrates the HTTPS requests an OpenVox agent makes to the OpenVox Server +during a single agent run. See the sections below for a detailed description of each step. + +```mermaid +sequenceDiagram + participant A as OpenVox Agent + participant S as OpenVox Server + + Note over A: 1. Check for keys and certificates + + A->>S: GET /puppet/v3/node/NAME + S-->>A: 2. Node object (environment) + + A->>S: GET /puppet/v3/file_metadatas/plugins + S-->>A: 3. Plugin metadata + A->>S: GET /puppet/v3/file_content/plugins/FILE + S-->>A: Plugin files + + A->>S: POST /puppet/v3/catalog/NAME (facts as JSON) + S-->>A: 4. Compiled catalog + + A->>S: GET /puppet/v3/file_metadata/PATH + S-->>A: 5. File metadata + A->>S: GET /puppet/v3/file_content/PATH + S-->>A: File content + + A->>S: PUT /puppet/v3/report/NAME + Note over S: 6. Report stored +``` + +## The process of Agent-side checks and HTTPS requests during a single agent run + +1. **Check for keys and certificates:** + + a. The agent downloads the CA (Certification Authority) bundle. + + b. If certificate revocation is enabled, the agent loads or downloads the Certificate + Revocation List (CRL) bundle using the previous CA bundle to verify the connection. + + c. The agent loads or generates a private key. If the agent needs a certificate, it generates + a Certificate Signing Request (CSR), including any `dns_alt_names` and `csr_attributes`, + and submits the request using `PUT /puppet-ca/v1/certificate_request/:certname`. + + d. The agent attempts to download the signed certificate using + `GET /puppet-ca/v1/certificate/:certname`. + + * If there is a conflict that must be resolved on the OpenVox Server, such as cleaning the + old CSR or certificate, the agent sleeps for `waitforcert` seconds, or exits with `1` if + waiting is not allowed, such as when running `puppet agent -t`. + + > **Tip:** This can happen if the agent's SSL directory is deleted, as the OpenVox Server + > still has the valid, unrevoked certificate. + + * If the downloaded certificate fails verification — for example, if it does not match its + private key — OpenVox discards the certificate. The agent sleeps for `waitforcert` + seconds, or exits with `1` if waiting is not allowed, such as when running + `puppet agent -t`. + +2. **Request a node object and switch environments:** + + Do a GET request to `/puppet/v3/node/`. + + * If the request is successful, read the environment from the node object. If the node object + has an environment, use that environment instead of the one in the agent's config file in all + subsequent requests during this run. + * If the request is unsuccessful, or if the node object had no environment set, use the + environment from the agent's config file. + +3. **If `pluginsync` is enabled on the agent, fetch plugins** from a file server mountpoint that + scans the `lib` directory of every module: + + * Do a GET request to `/puppet/v3/file_metadatas/plugins` with `recurse=true` and + `links=manage`. + * Check whether any of the discovered plugins need to be downloaded. If so, do a GET request + to `/puppet/v3/file_content/plugins/` for each one. + +4. **Request catalog while submitting facts:** + + Do a POST request to `/puppet/v3/catalog/`, where the post data is all of the node's + [facts][] encoded as JSON. Receive a compiled [catalog][] in return. + + > **Note:** Submitting facts isn't logically bound to requesting a catalog. For more + > information about facts, see [Language: Facts and built-in variables][facts]. + +5. **Make file source requests while applying the catalog:** + + [File][] resources can specify file contents as either a `content` or `source` attribute. + Content attributes go into the catalog, and the agent needs no additional data. Source + attributes put only references into the catalog and might require additional HTTPS requests. + + * If you are using the normal compiler, then for each file source, the agent makes a GET + request to `/puppet/v3/file_metadata/` and compares the metadata to the state + of the file on disk. + * If it is in sync, it continues on to the next file source. + * If it is out of sync, it does a GET request to `/puppet/v3/file_content/` for + the content. + * If you are using the [static compiler][static], all file metadata is embedded in the + catalog. For each file source, the agent compares the embedded metadata to the state of the + file on disk. + * If it is in sync, it continues on to the next file source. + * If it is out of sync, it does a GET request to + `/puppet/v3/file_bucket_file/md5/` for the current content. + + The static compiler is cheaper in terms of network traffic, but potentially more expensive + during catalog compilation. Large amounts of files, especially recursive directories, amplify + the effect in both directions. + +6. **Submit report:** + + If `report` is enabled on the agent, do a PUT request to `/puppet/v3/report/`. The + content of the PUT should be a Puppet report object in YAML format. diff --git a/package.json b/package.json new file mode 100644 index 000000000..4bc3848f1 --- /dev/null +++ b/package.json @@ -0,0 +1,7 @@ +{ + "name": "openvox-docs", + "private": true, + "dependencies": { + "mermaid": "11.15.0" + } +}