Skip to content

operations_and_monitoring.rst

Latest commit

 

History

History
826 lines (537 loc) · 25.1 KB

operations_and_monitoring.rst

File metadata and controls

826 lines (537 loc) · 25.1 KB

Operations and Monitoring

Access to Kibana

OpenStack control plane logs are aggregated from all servers by Fluentd and stored in ElasticSearch. The control plane logs can be accessed from ElasticSearch using Kibana, which is available at the following URL: |kibana_url|

To log in, use the kibana user. The password is auto-generated by Kolla-Ansible and can be extracted from the encrypted passwords file (|kolla_passwords|):

kayobe# ansible-vault view ${KAYOBE_CONFIG_PATH}/kolla/passwords.yml --vault-password-file |vault_password_file_path| | grep ^kibana

Access to Grafana

Control plane metrics can be visualised in Grafana dashboards. Grafana can be found at the following address: |grafana_url|

To log in, use the |grafana_username| user. The password is auto-generated by Kolla-Ansible and can be extracted from the encrypted passwords file (|kolla_passwords|):

kayobe# ansible-vault view ${KAYOBE_CONFIG_PATH}/kolla/passwords.yml --vault-password-file |vault_password_file_path| | grep ^grafana_admin_password

Access to Prometheus Alertmanager

Control plane alerts can be visualised and managed in Alertmanager, which can be found at the following address: |alertmanager_url|

To log in, use the admin user. The password is auto-generated by Kolla-Ansible and can be extracted from the encrypted passwords file (|kolla_passwords|):

kayobe# ansible-vault view ${KAYOBE_CONFIG_PATH}/kolla/passwords.yml --vault-password-file |vault_password_file_path| | grep ^prometheus_alertmanager_password

Migrating virtual machines

To see where all virtual machines are running on the hypervisors:

admin# openstack server list --all-projects --long

To move a virtual machine with shared storage or booted from volume from one hypervisor to another, for example to |hypervisor_hostname|:

admin# openstack --os-compute-api-version 2.30 server migrate --live-migration --host |hypervisor_hostname| 6a35592c-5a7e-4da3-9ab9-6765345641cb

To move a virtual machine with local disks:

admin# openstack --os-compute-api-version 2.30 server migrate --live-migration --block-migration --host |hypervisor_hostname| 6a35592c-5a7e-4da3-9ab9-6765345641cb

OpenStack Reconfiguration

Disabling a Service

Ansible is oriented towards adding or reconfiguring services, but removing a service is handled less well, because of Ansible's imperative style.

To remove a service, it is disabled in Kayobe's Kolla config, which prevents other services from communicating with it. For example, to disable cinder-backup, edit ${KAYOBE_CONFIG_PATH}/kolla.yml:

-enable_cinder_backup: true
+enable_cinder_backup: false

Then, reconfigure Cinder services with Kayobe:

kayobe# kayobe overcloud service reconfigure --kolla-tags cinder

However, the service itself, no longer in Ansible's manifest of managed state, must be manually stopped and prevented from restarting.

On each controller:

kayobe# docker rm -f cinder_backup

Some services may store data in a dedicated Docker volume, which can be removed with docker volume rm.

Installing TLS Certificates

|tls_setup|

To configure TLS for the first time, we write the contents of a PEM file to the secrets.yml file as secrets_kolla_external_tls_cert. Use a command of this form:

kayobe# ansible-vault edit ${KAYOBE_CONFIG_PATH}/secrets.yml --vault-password-file=|vault_password_file_path|

Concatenate the contents of the certificate and key files to create secrets_kolla_external_tls_cert. The certificates should be installed in this order:

In ${KAYOBE_CONFIG_PATH}/kolla.yml, set the following:

kolla_enable_tls_external: True
kolla_external_tls_cert: "{{ secrets_kolla_external_tls_cert }}"

To apply TLS configuration, we need to reconfigure all services, as endpoint URLs need to be updated in Keystone:

kayobe# kayobe overcloud service reconfigure

Alternative Configuration

As an alternative to writing the certificates as a variable to secrets.yml, it is also possible to write the same data to a file, etc/kayobe/kolla/certificates/haproxy.pem. The file should be vault-encrypted in the same manner as secrets.yml. In this instance, variable kolla_external_tls_cert does not need to be defined.

See Kolla-Ansible TLS guide for further details.

Updating TLS Certificates

Check the expiry date on an installed TLS certificate from a host that can reach the |project_name| OpenStack APIs:

openstack# openssl s_client -connect |public_endpoint_fqdn|:443 2> /dev/null | openssl x509 -noout -dates

NOTE: Prometheus Blackbox monitoring can check certificates automatically and alert when expiry is approaching.

To update an existing certificate, for example when it has reached expiration, change the value of secrets_kolla_external_tls_cert, in the same order as above. Run the following command:

kayobe# kayobe overcloud service reconfigure --kolla-tags haproxy

Taking a Hypervisor out of Service

To take a hypervisor out of Nova scheduling, for example |hypervisor_hostname|:

admin# openstack compute service set --disable \
       |hypervisor_hostname| nova-compute

Running instances on the hypervisor will not be affected, but new instances will not be deployed on it.

A reason for disabling a hypervisor can be documented with the --disable-reason flag:

admin# openstack compute service set --disable \
       --disable-reason "Broken drive" |hypervisor_hostname| nova-compute

Details about all hypervisors and the reasons they are disabled can be displayed with:

admin# openstack compute service list --long

And then to enable a hypervisor again:

admin# openstack compute service set --enable \
       |hypervisor_hostname| nova-compute

Managing Space in the Docker Registry

If the Docker registry becomes full, this can prevent container updates and (depending on the storage configuration of the seed host) could lead to other problems with services provided by the seed host.

To remove container images from the Docker Registry, follow this process:

  • Reconfigure the registry container to allow deleting containers. This can be done in docker-registry.yml with Kayobe:
docker_registry_env:
  REGISTRY_STORAGE_DELETE_ENABLED: "true"
  • For the change to take effect, run:
kayobe seed host configure
for repo in `./docker_reg_tool http://registry-ip:4000 list`; do
     ./docker_reg_tool http://registry-ip:4000 delete $repo $tag
done
  • Deleting the tag does not actually release the space. To actually free up space, run garbage collection:
seed# docker exec docker_registry bin/registry garbage-collect /etc/docker/registry/config.yml

The seed host can also accrue a lot of data from building container images. The images stored locally in the seed host can be seen using docker image ls.

Old and redundant images can be identified from their names and tags, and removed using docker image rm.

Octavia

Octavia uses mutual TLS to secure communication between the amphorae and Octavia services. It uses a private CA to sign both client and server certificates. We use the kolla-ansible built-in support for generating these certificates.

We ensure that duplicate subject names are allowed, by setting unique_subject=no in etc/kayobe/kolla/octavia-certificates/client_ca/index.txt.attr. This should already be set.

Checking certificate expiry
kayobe# ansible-vault decrypt client.cert-and-key.pem --vault-password-file  ~/vault
Decryption successful
kayobe# openssl x509 -enddate -noout -in client.cert-and-key.pem
notAfter=Aug 12 10:45:35 2022 GMT

There is also support in Kolla-Ansible to check if certs will expire within a given number of days:

kayobe# kayobe kolla ansible run "octavia-certificates <number-of-days>
Rotating client certificates

Octavia client certificates expire after a year, and should be rotated before they expire.

Note

Only the client certificate needs to be rotated annually. The server CA and client CA both have 10 year expiries, and should not be rotated before this since it would break the trust chain.

Decrypt all certificate files:

git grep -l VAULT etc/kayobe/kolla/octavia-certificates | xargs ansible-vault decrypt --vault-password-file

Remove the existing client certificate and keys to ensure they are regenerated:

rm etc/kayobe/kolla/octavia-certificates/client_ca/client.*

Generate new certificates:

kayobe kolla ansible run octavia-certificates -ke node_custom_config=$KAYOBE_CONFIG_PATH/kolla/config -ke node_config=$KAYOBE_CONFIG_PATH/kolla

This is likely to modify several files, and possibly add some new ones. Add the new files:

git add etc/kayobe/kolla/octavia-certificates/client_ca/

Encrypt the changed files that we are interested in:

git grep -L VAULT etc/kayobe/kolla/octavia-certificates/client_ca | xargs ansible-vault encrypt --vault-password-file
git grep -L VAULT etc/kayobe/kolla/config/octavia/*.pem | xargs ansible-vault encrypt --vault-password-file

Add and commit:

git add etc/kayobe/kolla/octavia-certificates/client_ca/
git add etc/kayobe/kolla/config/octavia/client.cert-and-key.pem etc/kayobe/kolla/config/octavia/client_ca.cert.pem
git commit -m "Octavia certs: rotate client certs"

Discard other unnecessary changes:

git status # check for anything unexpected!
git checkout etc/kayobe/kolla/octavia-certificates/ etc/kayobe/kolla/config/octavia

Deploy the new client certs:

kayobe overcloud service deploy --kolla-tags octavia
Rotating the CAs

The CAs have a 10 year lifetime. When rotating these, you will also need to generate new certificates.

Decrypt all certificate files:

git grep -l VAULT etc/kayobe/kolla/octavia-certificates | xargs ansible-vault decrypt --vault-password-file

Remove the existing CAs where appropriate:

rm -rf etc/kayobe/kolla/octavia-certificates/client_ca/
rm -rf etc/kayobe/kolla/octavia-certificates/server_ca/

Generate new CAs and certificates:

kayobe kolla ansible run octavia-certificates -ke node_custom_config=$KAYOBE_CONFIG_PATH/kolla/config -ke node_config=$KAYOBE_CONFIG_PATH/kolla

Encrypt the changed files that we are interested in:

git grep -L VAULT etc/kayobe/kolla/octavia-certificates/client_ca | xargs ansible-vault encrypt --vault-password-file
git grep -L VAULT etc/kayobe/kolla/config/octavia/*.pem | xargs ansible-vault encrypt --vault-password-file

Add and commit:

git add etc/kayobe/kolla/octavia-certificates/client_ca/
git add etc/kayobe/kolla/octavia-certificates/server_ca/
git add etc/kayobe/kolla/config/octavia/client.cert-and-key.pem etc/kayobe/kolla/config/octavia/client_ca.cert.pem
git commit -m "Octavia certs: rotate CAs"

Discard other unnecessary changes:

git status # check for anything unexpected!
git checkout etc/kayobe/kolla/octavia-certificates/ etc/kayobe/kolla/config/octavia

Deploy the new client certs:

kayobe overcloud service deploy --kolla-tags octavia

Follow any instructions in the upstream docs.

Backup of the OpenStack Control Plane

As the backup procedure is constantly changing, it is normally best to check the upstream documentation for an up to date procedure. Here is a high level overview of the key things you need to backup:

Controllers

Compute

The compute nodes can largely be thought of as ephemeral, but you do need to make sure you have migrated any instances and disabled the hypervisor before decommissioning or making any disruptive configuration change.

Monitoring

Seed

Ansible control host

  • Back up service VMs such as the seed VM

Control Plane Monitoring

The control plane has been configured to collect logs centrally using the EFK stack (Elasticsearch, Fluentd and Kibana).

Telemetry monitoring of the control plane is performed by Prometheus. Metrics are collected by Prometheus exporters, which are either running on all hosts (e.g. node exporter), on specific hosts (e.g. controllers for the memcached exporter or monitoring hosts for the OpenStack exporter). These exporters are scraped by the Prometheus server.

Configuring Prometheus Alerts

Alerts are defined in code and stored in Kayobe configuration. See *.rules files in ${KAYOBE_CONFIG_PATH}/kolla/config/prometheus as a model to add custom rules.

Silencing Prometheus Alerts

Sometimes alerts must be silenced because the root cause cannot be resolved right away, such as when hardware is faulty. For example, an unreachable hypervisor will produce several alerts:

  • InstanceDown from Node Exporter
  • OpenStackServiceDown from the OpenStack exporter, which reports status of the nova-compute agent on the host
  • PrometheusTargetMissing from several Prometheus exporters

Rather than silencing each alert one by one for a specific host, a silence can apply to multiple alerts using a reduced list of labels. :ref:`Log into Alertmanager <prometheus-alertmanager>`, click on the Silence button next to an alert and adjust the matcher list to keep only instance=<hostname> label. Then, create another silence to match hostname=<hostname> (this is required because, for the OpenStack exporter, the instance is the host running the monitoring service rather than the host being monitored).

Note

After creating the silence, you may get redirected to a 404 page. This is a known issue when running several Alertmanager instances behind HAProxy.

Generating Alerts from Metrics

Alerts are defined in code and stored in Kayobe configuration. See *.rules files in ${KAYOBE_CONFIG_PATH}/kolla/config/prometheus as a model to add custom rules.

Control Plane Shutdown Procedure

Overview

  • Verify integrity of clustered components (RabbitMQ, Galera, Keepalived). They should all report a healthy status.
  • Put node into maintenance mode in bifrost to prevent it from automatically powering back on
  • Shutdown down nodes one at a time gracefully using systemctl poweroff

Controllers

If you are restarting the controllers, it is best to do this one controller at a time to avoid the clustered components losing quorum.

Checking Galera state

On each controller perform the following:

[stack@|controller0_hostname| ~]$ docker exec -i mariadb mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_local_state_comment'"
Variable_name   Value
wsrep_local_state_comment       Synced

The password can be found using:

kayobe# ansible-vault view ${KAYOBE_CONFIG_PATH}/kolla/passwords.yml \
        --vault-password-file |vault_password_file_path| | grep ^database

Checking RabbitMQ

RabbitMQ health is determined using the command rabbitmqctl cluster_status:

[stack@|controller0_hostname| ~]$ docker exec rabbitmq rabbitmqctl cluster_status
Cluster status of node rabbit@|controller0_hostname| ...
[{nodes,[{disc,['rabbit@|controller0_hostname|','rabbit@|controller1_hostname|',
                'rabbit@|controller2_hostname|']}]},
 {running_nodes,['rabbit@|controller1_hostname|','rabbit@|controller2_hostname|',
                 'rabbit@|controller0_hostname|']},
 {cluster_name,<<"rabbit@|controller2_hostname|">>},
 {partitions,[]},
 {alarms,[{'rabbit@|controller1_hostname|',[]},
          {'rabbit@|controller2_hostname|',[]},
          {'rabbit@|controller0_hostname|',[]}]}]

Checking Keepalived

On (for example) three controllers:

[stack@|controller0_hostname| ~]$ docker logs keepalived

Two instances should show:

VRRP_Instance(kolla_internal_vip_51) Entering BACKUP STATE

and the other:

VRRP_Instance(kolla_internal_vip_51) Entering MASTER STATE

Ansible Control Host

The Ansible control host is not enrolled in bifrost. This node may run services such as the seed virtual machine which will need to be gracefully powered down.

Compute

If you are shutting down a single hypervisor, to avoid down time to tenants it is advisable to migrate all of the instances to another machine. See :ref:`evacuating-all-instances`.

.. ifconfig:: deployment['ceph_managed']

   Ceph
   ----

   The following guide provides a good overview:
   https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/8/html/director_installation_and_usage/sect-rebooting-ceph

Shutting down the seed VM

kayobe# virsh shutdown |seed_name|

Full shutdown

In case a full shutdown of the system is required, we advise to use the following order:

  • Perform a graceful shutdown of all virtual machine instances
  • Shut down compute nodes
  • Shut down monitoring node
  • Shut down network nodes (if separate from controllers)
  • Shut down controllers
  • Shut down Ceph nodes (if applicable)
  • Shut down seed VM
  • Shut down Ansible control host

Rebooting a node

Example: Reboot all compute hosts apart from |hypervisor_hostname|:

kayobe# kayobe overcloud host command run --limit 'compute:!|hypervisor_hostname|' -b --command "shutdown -r"

References

Control Plane Power on Procedure

Overview

  • Remove the node from maintenance mode in bifrost
  • Bifrost should automatically power on the node via IPMI
  • Check that all docker containers are running
  • Check Kibana for any messages with log level ERROR or equivalent

Controllers

If all of the servers were shut down at the same time, it is necessary to run a script to recover the database once they have all started up. This can be done with the following command:

kayobe# kayobe overcloud database recover

Ansible Control Host

The Ansible control host is not enrolled in Bifrost and will have to be powered on manually.

Seed VM

The seed VM (and any other service VM) should start automatically when the seed hypervisor is powered on. If it does not, it can be started with:

kayobe# virsh start seed-0

Full power on

Follow the order in :ref:`full-shutdown`, but in reverse order.

Shutting Down / Restarting Monitoring Services

Shutting down

Log into the monitoring host(s):

kayobe# ssh stack@|monitoring_host|

Stop all Docker containers:

|monitoring_host|# for i in `docker ps -q`; do docker stop $i; done

Shut down the node:

|monitoring_host|# sudo shutdown -h

Starting up

The monitoring services containers will automatically start when the monitoring node is powered back on.

Software Updates

Update Packages on Control Plane

OS packages can be updated with:

kayobe # kayobe overcloud host package update --limit |hypervisor_hostname| --packages '*'
kayobe # kayobe overcloud seed package update --packages '*'

See https://docs.openstack.org/kayobe/latest/administration/overcloud.html#updating-packages

Minor Upgrades to OpenStack Services

  • Pull latest changes from upstream stable branch to your own kolla fork (if applicable)
  • Update kolla_openstack_release in etc/kayobe/kolla.yml (unless using default)
  • Update tags for the images in etc/kayobe/kolla/globals.yml to use the new value of kolla_openstack_release
  • Rebuild container images
  • Pull container images to overcloud hosts
  • Run kayobe overcloud service upgrade

For more information, see: https://docs.openstack.org/kayobe/latest/upgrading.html

Troubleshooting

Deploying to a Specific Hypervisor

To test creating an instance on a specific hypervisor, as an admin-level user you can specify the hypervisor name as part of an extended availability zone description.

To see the list of hypervisor names:

admin# openstack hypervisor list

To boot an instance on a specific hypervisor, for example on |hypervisor_hostname|:

admin# openstack server create --flavor |flavor_name| --network |network_name| --key-name <key> --image CentOS8.2 --availability-zone nova::|hypervisor_hostname| vm-name

Cleanup Procedures

OpenStack services can sometimes fail to remove all resources correctly. This is the case with Magnum, which fails to clean up users in its domain after clusters are deleted. A patch has been submitted to stable branches. Until this fix becomes available, if Magnum is in use, administrators can perform the following cleanup procedure regularly:

admin# for user in $(openstack user list --domain magnum -f value -c Name | grep -v magnum_trustee_domain_admin); do
         if openstack coe cluster list -c uuid -f value | grep -q $(echo $user | sed 's/_[0-9a-f]*$//'); then
           echo "$user still in use, not deleting"
         else
           openstack user delete --domain magnum $user
         fi
       done

Elasticsearch indexes retention

To enable and alter default rotation values for Elasticsearch Curator, edit ${KAYOBE_CONFIG_PATH}/kolla/globals.yml:

# Allow Elasticsearch Curator to apply a retention policy to logs
enable_elasticsearch_curator: true

# Duration after which index is closed
elasticsearch_curator_soft_retention_period_days: 90

# Duration after which index is deleted
elasticsearch_curator_hard_retention_period_days: 180

Reconfigure Elasticsearch with new values:

kayobe overcloud service reconfigure --kolla-tags elasticsearch

For more information see the upstream documentation.