Skip to content

Commit 5c72aae

Browse files
tuxmeatuxmea
authored
Merge pull request #158 from miharp/feat/modernize-scaling-puppet-server
Modernize scaling page: OpenVox branding, compilers, Vox Pupuli load-balancing link
2 parents cde1b87 + a08e93c commit 5c72aae

2 files changed

Lines changed: 87 additions & 72 deletions

File tree

‎_data/nav/openvox-server_8x.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,8 @@
6868
link: ssl_server_certificate_change_and_virtual_ips.html
6969
- text: OpenVox Server tuning guide
7070
link: tuning_guide.html
71+
- text: Scaling OpenVox Server with compilers
72+
link: scaling_puppet_server.html
7173
- text: Migrating to the HOCON auth.conf format
7274
link: config_file_auth_migration.html
7375
- text: Administrative API endpoints

‎docs/_openvox-server_8x/scaling_puppet_server.markdown‎

Lines changed: 85 additions & 72 deletions
Original file line numberDiff line numberDiff line change
@@ -1,53 +1,53 @@
11
---
22
layout: default
3-
title: "Scaling Puppet Server with compilers"
4-
canonical: "/puppetserver/latest/scaling_puppet_server.html"
3+
title: "Scaling OpenVox Server with compilers"
54
---
65

7-
To scale Puppet Server for many thousands of nodes, you'll need to add Puppet masters dedicated to catalog compilation. These Servers are known as **compilers**, and are simply additional load-balanced Puppet
8-
Servers that receive catalog requests from agents and synchronize the results with each other.
6+
To scale OpenVox Server for many thousands of nodes, you'll need to add servers dedicated to catalog compilation.
7+
These are known as **compilers**, and are simply additional load-balanced OpenVox Servers that independently
8+
compile catalogs for agents from a shared codebase.
99

10-
> **If you're using Puppet Enterprise (PE),** consult its documentation instead of this guide for PE-specific requirements, settings, and instructions:
11-
>
12-
> - [Large environment installations (LEI)](https://puppet.com/docs/pe/latest/installing/hardware_requirements.html#large-environment-hardware-requirements)
13-
> - [Installing compilers](https://puppet.com/docs/pe/latest/installing/installing_compile_masters.html)
14-
> - [Load balancing](https://puppet.com/docs/pe/latest/installing/installing_compile_masters.html#using-load-balancers-with-compile-masters)
15-
> - [High availability](https://puppet.com/docs/pe/latest/high_availability/high_availability_overview.html)
16-
> - [Code Manager](https://puppet.com/docs/pe/latest/code_management/code_mgr_how_it_works.html)
10+
For a reference architecture covering the full load-balanced stack (HAProxy, r10k, Foreman, and automation),
11+
see [Load balanced architecture](https://voxpupuli.org/docs/arch_load_balanced) on the Vox Pupuli site.
1712

1813
## Planning your load-balancing strategy
1914

20-
The rest of your configuration depends on how you plan on distributing the agent load. Determine what your deployment will look like before you add any compilers, but **implement load balancing as the last
21-
step** only after you have the infrastructure in place to support it.
15+
The rest of your configuration depends on how you plan on distributing the agent load. Determine what your
16+
deployment will look like before you add any compilers, but **implement load balancing as the last step** only
17+
after you have the infrastructure in place to support it.
2218

2319
### Using round-robin DNS
2420

25-
Leave all of your agents pointed at the same Puppet Server hostname, then configure your site's DNS to arbitrarily route all requests directed at that hostname to the pool of available masters.
21+
Leave all of your agents pointed at the same OpenVox Server hostname, then configure your site's DNS to
22+
arbitrarily route all requests directed at that hostname to the pool of available servers.
2623

2724
For instance, if all of your agent nodes are configured with `server = puppet.example.com`, configure a DNS name such as:
2825

2926
```text
30-
# IP address of master 1:
27+
# IP address of server 1:
3128
puppet.example.com. IN A 192.0.2.50
32-
# IP address of master 2:
29+
# IP address of server 2:
3330
puppet.example.com. IN A 198.51.100.215
3431
```
3532

36-
For this option, configure your masters with `dns_alt_names` before their certificate request is made.
33+
For this option, configure your servers with `dns_alt_names` before their certificate request is made.
3734

3835
### Using a hardware load balancer
3936

40-
You can also use a hardware load balancer or a load-balancing proxy webserver to redirect requests more intelligently. Depending on your configuration (for instance, SSL using either raw TCP proxying or acting
41-
as its own SSL endpoint), you might also need to use other procedures in this document.
37+
You can also use a hardware load balancer or a load-balancing proxy webserver to redirect requests more
38+
intelligently. Depending on your configuration (for instance, SSL using either raw TCP proxying or acting as
39+
its own SSL endpoint), you might also need to use other procedures in this document.
4240

4341
Configuring a load balancer depends on the product, and is beyond the scope of this document.
4442

4543
### Using DNS `SRV` Records
4644

47-
You can use DNS `SRV` records to assign a pool of puppet masters for agents to communicate with. This requires a DNS service capable of `SRV` records, which includes all major DNS software.
45+
You can use DNS `SRV` records to assign a pool of OpenVox Servers for agents to communicate with. This requires
46+
a DNS service capable of `SRV` records, which includes all major DNS software.
4847

49-
> **Note:** This method makes a large number of DNS requests. Request timeouts are completely under the DNS server's control and agents cannot cancel requests early. SRV records don't interact well with static
50-
> servers set in the config file. Please keep these potential pitfalls in mind when configuring your DNS!
48+
> **Note:** This method makes a large number of DNS requests. Request timeouts are completely under the DNS
49+
> server's control and agents cannot cancel requests early. SRV records don't interact well with static servers
50+
> set in the config file. Please keep these potential pitfalls in mind when configuring your DNS!
5151
5252
Configure each of your agents with a `srv_domain` instead of a `server` in `puppet.conf`:
5353

@@ -57,104 +57,117 @@ use_srv_records = true
5757
srv_domain = example.com
5858
```
5959

60-
Agents will then lookup a `SRV` record at `_x-puppet._tcp.example.com` when they need to talk to a Puppet master.
60+
Agents will then look up a `SRV` record at `_x-puppet._tcp.example.com` when they need to talk to an OpenVox Server.
6161

6262
```text
63-
# Equal-weight load balancing between master-a and master-b:
64-
_x-puppet._tcp.example.com. IN SRV 0 5 8140 master-a.example.com.
65-
_x-puppet._tcp.example.com. IN SRV 0 5 8140 master-b.example.com.
63+
# Equal-weight load balancing between server-a and server-b:
64+
_x-puppet._tcp.example.com. IN SRV 0 5 8140 server-a.example.com.
65+
_x-puppet._tcp.example.com. IN SRV 0 5 8140 server-b.example.com.
6666
```
6767

68-
You can also implement more complex configurations. For instance, if all devices in site A are configured with a `srv_domain` of `site-a.example.com`, and all nodes in site B are configured to
69-
`site-b.example.com`, you can configure them to prefer a master in the local site but fail over to the remote site:
68+
You can also implement more complex configurations. For instance, if all devices in site A are configured with
69+
a `srv_domain` of `site-a.example.com`, and all nodes in site B are configured to `site-b.example.com`, you
70+
can configure them to prefer a server in the local site but fail over to the remote site:
7071

7172
```text
72-
# Site A has two masters - master-1 is beefier, give it 75% of the load:
73-
_x-puppet._tcp.site-a.example.com. IN SRV 0 75 8140 master-1.site-a.example.com.
74-
_x-puppet._tcp.site-a.example.com. IN SRV 0 25 8140 master-2.site-a.example.com.
75-
_x-puppet._tcp.site-a.example.com. IN SRV 1 5 8140 master.site-b.example.com.
76-
77-
# For site B, prefer the local master unless it's down, then fail back to site A
78-
_x-puppet._tcp.site-b.example.com. IN SRV 0 5 8140 master.site-b.example.com.
79-
_x-puppet._tcp.site-b.example.com. IN SRV 1 75 8140 master-1.site-a.example.com.
80-
_x-puppet._tcp.site-b.example.com. IN SRV 1 25 8140 master-2.site-a.example.com.
73+
# Site A has two servers - server-1 is beefier, give it 75% of the load:
74+
_x-puppet._tcp.site-a.example.com. IN SRV 0 75 8140 server-1.site-a.example.com.
75+
_x-puppet._tcp.site-a.example.com. IN SRV 0 25 8140 server-2.site-a.example.com.
76+
_x-puppet._tcp.site-a.example.com. IN SRV 1 5 8140 server.site-b.example.com.
77+
78+
# For site B, prefer the local server unless it's down, then fail back to site A
79+
_x-puppet._tcp.site-b.example.com. IN SRV 0 5 8140 server.site-b.example.com.
80+
_x-puppet._tcp.site-b.example.com. IN SRV 1 75 8140 server-1.site-a.example.com.
81+
_x-puppet._tcp.site-b.example.com. IN SRV 1 25 8140 server-2.site-a.example.com.
8182
```
8283

8384
## Centralizing the Certificate Authority
8485

85-
Additional Puppet Servers should only share the burden of compiling and serving catalogs, which is why they're typically referred to as "compilers". Any certificate authority functions should be delegated to a
86-
single server.
86+
Additional OpenVox Servers should only share the burden of compiling and serving catalogs, which is why they're
87+
typically referred to as "compilers". Any certificate authority functions should be delegated to a single server.
8788

88-
Before you centralize this functionality, ensure that the single server that you want to use as the central CA is reachable at a unique hostname other than (or in addition to) `puppet`. Next, point all agent
89-
requests to the centralized CA master, either by configuring each agent or through DNS `SRV` records.
89+
Before you centralize this functionality, ensure that the single server that you want to use as the central CA
90+
is reachable at a unique hostname other than (or in addition to) `puppet`. Next, point all agent requests to
91+
the centralized CA server, either by configuring each agent or through DNS `SRV` records.
9092

9193
### Directing individual agents to a central CA
9294

93-
On every agent, set the [`ca_server`](https://puppet.com/docs/puppet/latest/configuration.html#caserver) setting in [`puppet.conf`](https://puppet.com/docs/puppet/latest/config_file_main.html) (in the `[main]`
94-
configuration block) to the hostname of the server acting as the certificate authority. If you have a large number of existing nodes, it is easiest to do this by managing `puppet.conf` with a Puppet module and
95-
a template.
95+
On every agent, set the `ca_server` setting in `puppet.conf` (in the `[main]` configuration block) to the
96+
hostname of the server acting as the certificate authority. If you have a large number of existing nodes, it is
97+
easiest to do this by managing `puppet.conf` with an OpenVox module and a template.
9698

9799
> **Note:** Set this setting _before_ provisioning new nodes, or they won't be able to complete their initial agent run.
98100
99101
### Pointing DNS `SRV` records at a central CA
100102

101-
If you [use `SRV` records for agents](#using-dns-srv-records), you can use the `_x-puppet-ca._tcp.$srv_domain` DNS name to point clients to one specific CA server, while the `_x-puppet._tcp.$srv_domain` DNS
102-
name handles most of their requests to masters and can point to a set of compilers.
103+
If you [use `SRV` records for agents](#using-dns-srv-records), you can use the `_x-puppet-ca._tcp.$srv_domain`
104+
DNS name to point clients to one specific CA server, while the `_x-puppet._tcp.$srv_domain` DNS name handles
105+
most of their requests and can point to a set of compilers.
103106

104107
## Creating and configuring compilers
105108

106-
To add a compiler to your deployment, begin by [installing and configuring Puppet Server](./install_from_packages.html) on it.
109+
To add a compiler to your deployment, begin by [installing and configuring OpenVox Server](./install_from_packages.html) on it.
107110

108-
Before running `puppet agent` or `puppet master` on the new server:
111+
Before running `puppet agent` or starting `puppetserver` on the new compiler:
109112

110-
1. In the compiler's [`puppet.conf`](https://puppet.com/docs/puppet/latest/config_file_main.html), in the `[main]` configuration block, set the
111-
[`ca_server`](https://puppet.com/docs/puppet/latest/configuration.html#caserver) setting to the hostname of the server acting as the certificate authority.
113+
1. In the compiler's `puppet.conf`, in the `[main]` configuration block, set `ca_server` to the hostname of
114+
the server acting as the certificate authority.
112115

113116
1. In the compiler's `webserver.conf` file, add and set the following SSL settings:
114117
- ssl-cert
115118
- ssl-key
116119
- ssl-ca-cert
117120
- ssl-crl-path
118121

119-
1. [Disable Puppet Server's certificate authority services](./configuration.html#service-bootstrapping).
122+
1. [Disable OpenVox Server's certificate authority services](./configuration.html#service-bootstrapping).
123+
If an `ssldir` is configured, make sure it's set in the `[main]` block of `puppet.conf` only.
120124

121-
If you're using the [individual agent configuration method of CA centralization](#directing-individual-agents-to-a-central-ca), set `ca_server` in `puppet.conf` to the hostname of your CA server in the
122-
`[main]` config block. If an `ssldir` is configured, make sure it's configured in the `[main]` block only.
125+
1. If you're using the [DNS round robin method](#using-round-robin-dns) of agent load balancing, or a
126+
[load balancer](#using-a-hardware-load-balancer) in TCP proxying mode, provide compilers with certificates
127+
using DNS Subject Alternative Names.
123128

124-
1. If you're using the [DNS round robin method](#using-round-robin-dns) of agent load balancing, or a [load balancer](#using-a-hardware-load-balancer) in TCP proxying mode, provide compilers with
125-
certificates using DNS Subject Alternative Names.
126-
127-
Configure `dns_alt_names` in the `[main]` block of `puppet.conf` to cover every DNS name that might be used by an agent to access this master.
129+
Configure `dns_alt_names` in the `[main]` block of `puppet.conf` to cover every DNS name that might be
130+
used by an agent to access this server.
128131

129132
```text
130133
dns_alt_names = puppet,puppet.example.com,puppet.site-a.example.com
131134
```
132135
133-
If the agent or master has been run and already created a certificate, remove it by running `sudo puppet ssl clean`. If an agent has requested a certificate from the master, delete it there to re-issue a
134-
new one with the alt names: `puppetserver ca clean master-2.example.com`.
136+
If the agent or server has been run and already created a certificate, remove it by running
137+
`sudo puppet ssl clean`. If an agent has requested a certificate from the server, delete it there to
138+
re-issue a new one with the alt names: `puppetserver ca clean server-2.example.com`.
135139
136140
1. Request a new certificate by running `puppet agent --test --waitforcert 10`.
137141
138-
1. Log into the CA server and run `puppetserver ca sign master-2.example.com`.
142+
1. Log into the CA server and run `puppetserver ca sign server-2.example.com`.
139143
140-
## Centralizing reports, inventory service, and catalog searching (storeconfigs)
144+
## Centralizing reports and exported resources
141145
142-
If you use an HTTP report processor, point your master and all of your Puppet compilers at the same shared report server in order to see all of your agents' reports.
146+
If you use an HTTP report processor, point your primary server and all compilers at the same shared report
147+
server in order to see all of your agents' reports.
143148
144-
If you use the inventory service or exported resources, use PuppetDB and point your master and all of your Puppet compilers at a shared PuppetDB instance. A reasonably robust PuppetDB server can handle many
145-
Puppet compilers and many thousands of agents.
149+
If you use exported resources, use OpenVoxDB and point your primary server and all compilers at a shared
150+
OpenVoxDB instance. A reasonably robust OpenVoxDB server can handle many compilers and many thousands of agents.
146151
147-
See the [PuppetDB documentation](https://puppet.com/docs/puppetdb/latest/) for instructions on deploying a PuppetDB server, then configure every Puppet compiler to use it. Note that every Puppet master and
148-
compiler must have its own [whitelist entry](https://puppet.com/docs/puppetdb/latest/configure.html#certificate-whitelist) if you're using HTTPS certificates for authorization.
152+
See the [OpenVoxDB documentation](../../../openvoxdb/latest/) for instructions on deploying an OpenVoxDB
153+
server, then configure every compiler to use it. Note that every server and compiler must have its own
154+
[certificate allowlist entry](../../../openvoxdb/latest/configure.html) if you're using HTTPS certificates
155+
for authorization.
149156
150157
## Keeping manifests and modules synchronized across compilers
151158
152-
You must ensure that all Puppet compilers have identical copies of your manifests, modules, and [external node classifier](https://puppet.com/docs/puppet/latest/nodes_external.html) data. Examples include:
159+
You must ensure that all compilers have identical copies of your manifests, modules, and external node
160+
classifier data.
153161
154-
- Using a version control system such as [r10k](https://github.com/puppetlabs/r10k), Git, Mercurial, or Subversion to manage and sync your manifests, modules, and other data.
155-
- Running an out-of-band `rsync` task via `cron`.
156-
- Configuring `puppet agent` on each compiler to point to a designated model Puppet master, then use Puppet itself to distribute the modules.
162+
The recommended approach is to use [r10k](https://github.com/voxpupuli/r10k) with a webhook to trigger
163+
automatic code deployment to all compilers on every push to your control repository. The Vox Pupuli
164+
[webhook-go](https://github.com/voxpupuli/webhook-go) service provides a lightweight webhook receiver
165+
that calls r10k on each compiler. See the [load balanced architecture guide](https://voxpupuli.org/docs/arch_load_balanced)
166+
for a full reference setup.
157167
158-
## Implementing load distribution
168+
Other options include:
159169
160-
Now that your other compilers are ready, you can implement your [agent load-balancing strategy](#planning-your-load-balancing-strategy).
170+
- Running r10k manually or via `cron` on each compiler.
171+
- Running an out-of-band `rsync` task via `cron`.
172+
- Configuring `puppet agent` on each compiler to point to a designated primary server, then use OpenVox itself
173+
to distribute the modules.

0 commit comments

Comments
 (0)