Skip to content

design-decisions.md

Latest commit

 

History

History
965 lines (816 loc) · 32.9 KB

design-decisions.md

File metadata and controls

965 lines (816 loc) · 32.9 KB
Raw

Design decisions

Umbrella Glance Service (Split API deployment)

Note: Starting with RHOSO 19, the split API layout is deprecated. The security vulnerability described in OSSN-0090 has been resolved with Glance's new location API, which is now supported by Nova and Cinder. For new deployments, use the default single layout. For migration guidance, see Location API Migration Guide.

As mentioned in OSSN-0090, when deploying Glance in a popular configuration where Glance shares a common storage backend with Nova and/or Cinder, it was possible to open some known attack vectors by which malicious data modification could occur. The split API deployment was implemented as a security workaround before the location API was available.

  • A user facing glance-api service, accessible via the Public and Admin keystone endpoints.
  • An internal facing only service, accessible via the Internal keystone endpoint.

The user facing service is configured to not expose image locations, namely by setting the following options in glance-api.conf:

[DEFAULT]
show_image_direct_url = False
show_multiple_locations = False

The internal service, operating on a different port (e.g. 9293), is configured identically to the public facing service, except for the following:

[DEFAULT]
show_image_direct_url = True
show_multiple_locations = True

OpenStack services that use glance (cinder and nova) is configured to access it via the new internal service. That way both cinder and nova will have access to the image location data.

Enable Per Tenant Quotas

Glance supports resource consumption quotas on tenants through the use of Keystone’s unified limits functionality. In order to enable this feature, as per the official documentation, the Glance CRD exposes the quotas structure, where the limits defined in the documentation can be defined. As the full Glance example CR shows, when the following is added in the top level definition, all the GlanceAPI Pods will enable per tenant quotas via use_keystone_limits = True in their config, that will be automatically generated by the Glance operator.

apiVersion: glance.openstack.org/v1beta1
kind: Glance
metadata:
  name: glance
spec:
  ...
  ...
  quotas:
    imageSizeTotal: 1000
    imageStageTotal: 1000
    imageCountUpload: 100
    imageCountTotal: 100

As a follow up work, tha Glance operator might rely on the existing webhooks to enforce the Quota Limits values.

Cinder backend

When using the Cinder backend the Glance containers need to run with higher privileges than with other backends, this is due to the usage of os-brick to connect to volumes.

Some of the things that the operator does when using Cinder as a backend are:

  • Run containers as privileged
  • Mount for r/w more directories
  • Pod shares the PID namespace with the host
  • Run CLI commands with the binaries present in the host

OpenShift deployments are different from TripleO deployments. For example, in TripleO we run iscsid and multipathd daemons in containers and their client side commands are in sync (version-wise) in the service containers. It is also required that there are no other daemons running in the host or other containers.

In the OpenShift world this is different, iscsid and multipathd run on the host and therefore their version won't be in sync with the one in the openstack service containers, which means that they won't work correctly.

The reason why OpenShift runs iscsid and multipathd in the host is so that they can be shared between the different things that may need it: OpenShift itself, CSI plugins, and in this case also OpenStack services.

The way to use the host iscsi and multipath services is by exiting the pod namespace and running the commands on the host namespace using nsenter. Even if this may look nasty/weird, this is the recommended way by the storage OpenShift team and what CSI plugins do.

We are using nsenter through the templates/glance/bin/run-on-host script. This run-on-host script supports 2 ways of replacing commands: being copied or symlinked. In our case we instruct kolla to use the copying mechanism in templates/glance/config/glance-api-config.json.

To be able to use nsenter the pod needs to share the PID namespace with the host.

GlanceAPI deployment

It's possible to deploy multiple GlanceAPI to serve different workloads. When a Glance CR is created, if not explicitly specified in the top level CR, the mutate webhook invocation creates an instance of a glanceAPI object called "default", and it results in two GlanceAPI deployments.

apiVersion: glance.openstack.org/v1beta1
kind: Glance
...
spec:
  serviceUser: glance
  databaseInstance: openstack
  databaseUser: glance
  glanceAPIs:
    ..
  secret: osp-secret
  storageClass: ""
...
...

Both GlanceAPI internal and external sub-CR(s) are generated, resulting in two different k8s StatefuSet objects. The Status and the definition of the deployed GlanceAPI services can be retrieved with the following commands:

$ oc get Glance

NAME     STATUS   MESSAGE
glance   True     Setup complete

$ oc get GlanceAPI

NAME                      NETWORKATTACHMENTS   STATUS   MESSAGE
glance-default-external                        True     Setup complete
glance-default-internal                        True     Setup complete

The resulting CRs can be inspected as regular k8s objects. The following diagram describes how k8s resources/object are connected.

                                                         [statefulset(s)]
                                                   +-------------------------+
+------------+       +---------------------+ |---> | glance-default-internal |
| glance CR  | |---> | glanceAPI-default CR|       +-------------------------+
+------------+       +---------------------+ |---> | glance-default-external |
                                                   +-------------------------------+
 (headless SVC to allow pod2pod communication)     | glance-default-internal-api   |
 (it must match with the StatefulSet ServiceName)  +-------------------------------+
                                                   | glance-default-external-api   |
                                                   +-------------------------------+

By default, if no option is specified, a glanceAPI deployment is split between internal and external. It's possible to not split the deployment and reduce everything to a single API instance. To do that, a Glance CR spec like the following should be created:

apiVersion: glance.openstack.org/v1beta1
kind: Glance
...
spec:
  serviceUser: glance
  databaseInstance: openstack
  databaseUser: glance
  glanceAPIs:
    api0:
      replicas=1
      type: "single"
  secret: osp-secret
  storageClass: ""
...
...

As per the example above, by passing type: single to the api0 instance, defined in the glanceAPIs section, let the main Glance controller to only generate a single api0 StatefulSet that is not split between internal and external.

                                                                               [k8s Service(s)]
                                                   [statefulset(s)]        +---------------------+
+------------+       +------------------+       +-------------------+      | glance-api-internal |
| glance CR  | |---> | glance-default CR| |---> | glance-api-single | |--> | --------------------+
+------------+       +------------------+       +-------------------+      | glance-api-public   |
                                                                           +-----------------------+
                                                       (headless Svc) |--> | glance-api-single-api |
                                                                           +-----------------------+

Both internal and public k8s Services are created, and they point to the single StatefuSet generated by the glanceAPI controller.

$ oc get glanceapi
NAME                NETWORKATTACHMENTS   STATUS   MESSAGE
glance-api-single                        True     Setup complete

The generic GlanceAPI StatefuSet, generated by the process above, deploys three containers within the same Pod:

+-------------------------------------+
| GlanceAPI::api0:Pod                 |
|                                     |
|   +-----------------------+         |
|   |   +------------+      |         |
|   |   | glance-log | -------------------> It streams the glance-api0 logs
|   |   +------------+      |         |     in /var/log/glance
|   +-----------------------+         |
|   +-----------------------+         |
|   |   +--------------+    |         |
|   |   | glance-httpd | ---|-------------> It intercepts requests on 9292 and
|   |   +-+------------+    |         |     do the ProxyPass to the process
|   |     |                 |         |     behind on 127.0.0.1:9293
|   |   +-+----------+      |         |
|   |   | glance-api | -----|-------------> Runs the glance-api process on 9293
|   |   +------------+      |         |
|   +-----+-----------------+         |
+---------|-----------------------------+
          |
          +-> GlanceAPI:api0 = (httpd + glance-api)

The reason about having an additional httpd layer relies on the fact that TLS is not natively supported in Glance. Even though Glance it's not supported in production as regular wsgi process run by httpd, this scenario sees httpd configured in a way that executes ProxyPass for the requests coming from the associated endpoint to the glance-api service.

More details about this can be found in the upstream documentation.

Deploy multiple GlanceAPI instances

The previous section describes the deployment flow and how many layouts are supported by a glanceAPI: having single or split might change according to the configured backend. An additional feature provided by the glance-operator is the ability to deploy multiple GlanceAPI that are supposed to serve different workloads. A real use case scenario where it makes sense to have multiple GlanceAPI is an Edge deployment (known as DCN). This is a scenario where multiple glanceAPI instances are orchestrated by the same operator, but they're connected with different backends.

spec:
  customServiceConfig: |
    [DEFAULT]
    enabled_backends = default_backend:rbd
    [glance_store]
    default_backend = default_backend
    [default_backend]
    rbd_store_ceph_conf = /etc/ceph/ceph.conf
    store_description = "RBD backend"
    rbd_store_pool = images
    rbd_store_user = openstack
  databaseInstance: openstack
  databaseUser: glance
  keystoneEndpoint: central
  glanceAPIs:
    central:
      replicas: 1
    edge0:
      replicas: 1
    edge1:
      replicas: 1
...
...
...
  extraMounts:
    - name: central
      region: r1
      extraVol:
        - propagation:
          - central
          volumes:
          - name: ceph0
            secret:
              secretName: ceph-conf-files-0
          mounts:
          - name: ceph0
            mountPath: "/etc/ceph"
            readOnly: true
    - name: edge0
      region: r1
      extraVol:
        - propagation:
          - edge0
          volumes:
          - name: ceph1
            secret:
              secretName: ceph-conf-files-1
          mounts:
          - name: ceph1
            mountPath: "/etc/ceph"
            readOnly: true
    - name: edge1
      region: r1
      extraVol:
        - propagation:
          - edge1
          volumes:
          - name: ceph2
            secret:
              secretName: ceph-conf-files-2
          mounts:
          - name: ceph2
            mountPath: "/etc/ceph"
            readOnly: true

In the example above, all the GlanceAPI instances share the same configuration, which is inherited by the main customServiceConfig, whileextraMounts are added to connect each instance to a different Ceph cluster. For each instance, you can configure the layout parameter (either split or single) based on the specific backend requirements. However, keep in mind that webhooks will prevent any modifications to the layout after it's been set.

It's also possible to deploy multiple glanceAPI services within the same availability zone (AZ) to handle different workloads. Here's an example configuration:

  keystoneEndpoint: api0
  glanceAPIs:
    api0:
      customServiceConfig: |
        [DEFAULT]
        enabled_backends = default_backend:rbd
      replicas: 1
      type: split
    api1:
      customServiceConfig: |
        [DEFAULT]
        enabled_backends = default_backend:swift
      replicas: 1
      type: split

In this setup:

  • api0 is registered in the Keystone catalog and used as the default endpoint for CLI operations.
  • api1, while not the default, remains an active API that can still be utilized for image uploads by specifying the --os-image-url parameter.

Switching the Keystone catalog's active API is straightforward and can be done by updating the keystoneEndpoint parameter.

Manage KeystoneEndpoint

Even though multiple instances can be deployed and be reachable through the generated k8s Services, only one of them can be registered in the keystone catalog at a given time. There's a 1:1 relation between the image service in keystone, associated to Glance, and the Endpoint (internal and public) that exist in Keystone. For this reason, in the top-level Glance CR, a keystoneEndpoint parameter is defined and exposed. Unless a single instance is deployed, the human operator is supposed to choose, before the main OpenStackControlPlane CR is applied, which instance should be registered in keystone.

As an example, let's consider a generic Glance spec like the following:

spec:
  customServiceConfig: |
  ...
  ...
  databaseInstance: openstack
  databaseUser: glance
  keystoneEndpoint: api0
  glanceAPIs:
    api0:
      replicas: 1
    api1:
      replicas: 1
    api2:
      replicas: 1
...
...

The keystoneEndpoint parameter points to api0: it means that api0 will be registered in the Keystone catalog. An OpenStack administrator will be able to reach that instance via openstack-cli and perform any normal image related operation. At the same time, api1 and api2 are not reachable via cli, and they will only be available through their k8s Services by applications that are able to discover their Endpoints. Once a GlanceAPI is deployed, its Endpoints are reflected in both the generated sub-CR, as well as in the top level CR. For each deployed API instance, an entry is added in the Status.Endpoints map, and each endpoint is prefixed with the name of the API it refers to.

apiVersion: glance.openstack.org/v1beta1
kind: Glance
...
status:
  apiEndpoint:
    api0-internal: http://glance-api0-internal.openstack.svc:9292
    api0-public: https://glance-api0-public.apps.crc.openstack.dev:9292
    api1-internal: http://glance-api1-internal.openstack.svc:9292
    api1-public: https://glance-api1-public.apps.crc.openstack.dev:9292
    ...
...

When an API is decommissioned, the associated Endpoint is removed from the top-level apiEndpoint map. Other than decommissioning an instance (delete from the main CR), it's possible, at any point in time, to update the keystoneEndpoint value and select a different instance that should be registered in Keystone. An update to the keystoneEndpoint parameter results in a reconciliation execution that switches the two affected instances (the one registered in keystone and the one proposed by the new update).

Add a new GlanceAPI as day2 operation

Adding an additional GlanceAPI to OpenStack serves various purposes beyond just supporting multiple workloads. For instance, to maintain the lifecycle of a GlanceAPI and its backend services, switching between a split layout and a single layout is generally not feasible. Consider a scenario where maintenance requires switching from a backend with a split layout (e.g., Ceph, Swift, Cinder, S3) to one that uses a single layout (e.g., File, NFS). The glance-operator webhooks prevent updates to the layout, as such changes impact critical configuration elements like the number of PersistentVolumeClaims (PVCs), their size, the image cache setup, and other components. For further details, refer to the storage requirements documentation. Adding a new GlanceAPI assumes an OpenStackControlPlane is already configured and can be done with the following steps:

  1. Create a glance-api.yaml spec file. In this example, there are two GlanceAPI instances in the OpenStackControlPlane: the existing API uses Swift as its backend, while the new API uses File with a single layout and one replica.
spec:
  glance:
    template:
      databaseInstance: openstack
      keystoneEndpoint: default
      glanceAPIs:
        default:
          customServiceConfig: |
            [DEFAULT]
            enabled_backends = default_backend:swift
            [glance_store]
            default_backend = default_backend
            [default_backend]
            swift_store_create_container_on_put = True
            swift_store_auth_version = 3
            swift_store_auth_address = {{ .KeystoneInternalURL }}
            swift_store_endpoint_type = internalURL
            swift_store_user = service:glance
            swift_store_key = {{ .ServicePassword }}
            swift_store_region = {{ .Region }}
          preserveJobs: false
          replicas: 3
        default1:
          type: single
          replicas: 1
      storage:
        storageRequest: 10G

Note: Retrieve the initial Glance spec from an existing control plane using the following command:

$ oc get oscp openstack-galera-network-isolation -o yaml | yq '.spec.glance'

Clean up the output to include only relevant details for updating the OpenStackControlPlane.

  1. Apply the new spec file:
$ oc -n $NAMESPACE patch openstackcontrolplane $(oc get oscp -o custom-columns=NAME:.metadata.name --no-headers)  --type=merge --patch-file=glance_spec.yaml
  1. Verify the new GlanceAPI deployment:
oc -n $NAMESPACE get pods -l service=glance
  • Replace $NAMESPACE with the actual namespace where the OpenStackControlPlane is deployed.

Additional Notes

You can perform sequential updates to patch customServiceConfig or adjust keystoneEndpoint if you want a new API to serve as the main OpenStack CLI entry point.

Alternative deployment

For simpler requirements, skip Steps 1 and 2 and deploy an additional API instance directly with:

$ oc -n $NAMESPACE patch openstackcontrolplane $(oc get oscp -o custom-columns=NAME:.metadata.name --no-headers) --type=merge -p='{"spec": {"glance": {"template": {"keystoneEndpoint":"default1", "glanceAPIs": {"default1": {"type": "single", "networkAttachments":["storage"]}}}}}}'

This command adds the following structure to the control plane:

spec:
  glance:
    template:
      keystoneEndpoint: default
      glanceAPIs:
        default1:
          type: single
          replicas: 1

Decommissioning a GlanceAPI

Decommissioning a GlanceAPI requires patching the OpenStackControlPlane and perform at least two actions:

  1. Delete the GlanceAPI CR and the kubernetes associated objects (Pods, StatefulSets)
  2. Update the keystoneEndpoint to point to an active API

Note: You cannot delete a GlanceAPI if it is the only API in the OpenStackControlPlane, nor can you point keystoneEndpoint to a non-existent API under the glanceAPIs section.

Removing an existing GlanceAPI can be achieved with the following procedure:

  1. Verify that more that one GlanceAPI is deployed in the OpenStackControlPlane:
oc -n $NAMESPACE get oscp  $(oc get oscp -o custom-columns=NAME:.metadata.name --no-headers) -o jsonpath='{.spec.glance.template.glanceAPIs}' | jq

{
  "default": {
    "apiTimeout": 60,
    "imageCache": {
      "cleanerScheduler": "*/30 * * * *",
      "prunerScheduler": "1 0 * * *",
      "size": ""
    },
    "networkAttachments": [
      "storage"
    ],
    "override": {},
    "replicas": 1,
    "resources": {},
    "storage": {},
    "tls": {
      "api": {
        "internal": {},
        "public": {}
      }
    },
    "type": "split"
  },
  "default1": {
    "apiTimeout": 60,
    "imageCache": {
      "cleanerScheduler": "*/30 * * * *",
      "prunerScheduler": "1 0 * * *",
      "size": ""
    },
    "override": {},
    "replicas": 1,
    "resources": {},
    "storage": {},
    "tls": {
      "api": {
        "internal": {},
        "public": {}
      }
    },
    "type": "split"
  }
}
  1. Identify the current keystone catalog GlanceAPI:
$ oc -n $NAMESPACE get oscp $(oc get oscp -o custom-columns=NAME:.metadata.name --no-headers) -o jsonpath='{.spec.glance.template.keystoneEndpoint}'

default
  1. (Optional): Update the Keystone endpoint. If the API to be removed is currently associated with the keystone catalog, switch the KeystoneEndpoint to another API:
$ oc -n $NAMESPACE patch openstackcontrolplane $(oc get oscp -o custom-columns=NAME:.metadata.name --no-headers) --type=merge -p='{"spec": {"glance": {"template": {"keystoneEndpoint":"default1"}}}}'
  1. Verify the new API has an associated keystone endpoint:
$ oc exec -it openstackclient bash -- openstack endpoint list | grep image
| a32dfa92fafd40588f3fef699744511f | regionOne | glance       | image        | True    | internal  | https://glance-default1-internal.openstack.svc:9292                   |
| c4804456ee37464ea48e5258e7f29d7a | regionOne | glance       | image        | True    | public    | https://glance-default1-public-openstack.apps-crc.testing             |
  1. Remove the GlanceAPI called default:
$ oc -n $NAMESPACE patch openstackcontrolplane $(oc get oscp -o custom-columns=NAME:.metadata.name --no-headers) --type=json -p="[{'op': 'remove', 'path': '/spec/glance/template/glanceAPIs/default'}]"

and observe the Pods and the associated resources are deleted.

Note: The above process does not delete any PersistentVolumeClaims (PVCs) associated with the decommissioned API. This is by design, allowing re-adding the API with previous settings without data loss.

  • Replace $NAMESPACE with the actual namespace where the OpenStackControlPlane is deployed.

How Conditions are managed

Conditions represent a critical aspect for reconciliation because they allow:

  1. to evaluate the status of a particular component validating a set of conditions
  2. give a feedback to the end user about the current state of the Deployment
  3. identify the status of the underlying GlanceAPIs and report their healhy to the upper level CR

The document docs/conditions describes the meaning of the k8s-operators shared Conditions. Conditions are re-evaluated when a new Reconcile loop starts, and if one of them is different from what was previously registered, an update is performed, and it allows to give immediate feedback to the user. Conditions are initially marked as Unknown, while the general ReadyCondition is marked as False because we assume that the Deployment is in progress but not Ready at the same time, until the entire set of conditions are evaluated. Conditions can be seen as a checklist or steps within the reconcile loop, and while the loop proceed to the end, each of them is evaluated. If a particular component with an associated condition is not Ready, an error is returned from the operator, and its failing condition is marked as False with an appropriate error message. If the end of the loop is reached, it means we passed through all the steps and the Conditions are marked to True: it is possible, at this point, to mark the overall ReadyCondition to Status=True as well and Mirror the result to the top-level CR. Conditions are always Mirrored using the defer function, so that we always have an information to the ReadyCondition that reflects the point where the loop is exiting/failing. The general IsReady function is used as a wrapper for the ReadyCondition boolean, and it can be used to get the status of the instance. In general in the glance-operator we make the following assumptions:

  • A Glance/GlanceAPI is considered Ready if all the subconditions are verified
  • A Deployment (intended as the StatefulSet that represents the GlanceAPIs) is considered Ready if the number of Replicas specified in the Glance CR spec is equal to the number of available instances (ReadyCount).

Storage Requirements: PVCs usage and available models

Glance requires a staging area to manipulate data during an import operation. It is possible to copy image data into multiple stores using image import workflow, and all the import methods and plugins are based on this flow, hence some persistence is required for the resulting Glance Pod. PVCs represent the main Storage interface for the Glance service, and it's important to have a clear picture of how this interface is implemented and how it can be used in this context. In general, a Storage struct is exposed by the glance-operator API, and it helps to choose the best layout for a particular glanceAPI. If the storage layout is not defined for a single glanceAPI, it will be inherited by the top-level definition. When defining the storage requirements for a GlanceAPI, it is possible to choose between two different models:

  • External
  • PVC

External

If External is chosen, no PVCs are created, and from a Storage point of view the GlanceAPI acts like a stateless instance, where no persistence is provided. In this case, it's expected that the human operator provides some persistence via extraMounts. For this particular scenario, the most popular choice is NFS, and it can be mapped to /var/lib/glance/os_glance_staging_store

...
default:
  storage:
    external: true
...
...
extraMounts:
- extraVol:
  - extraVolType: NFS
    mounts:
    - mountPath: /var/lib/glance/os_glance_staging_store
      name: nfs
    volumes:
    - name: nfs
      nfs:
        path: <NFS PATH>
        server: <NFS IP ADDRESS>

It's important to note that the configuration described in the sample above conflicts with the distributed image import feature that uses glance-direct import method. distributed image import requires a RWO storage plugged into a particular instance: it owns the data and receives requests in case its staged data are required for an upload operation. When the External model is adopted, if Ceph is used as a backend and an image conversion operation is run in one of the existing replicas, the glance-operator does not have to make any assumption about the underlying storage that is tied to the staging area, and the conversion operation that uses the os_glance_staging_store directory (within the Pod) interacts with the RWX NFS backend provided via extraMounts. With this scenario, an image-cache PVC can still be requested and mounted to a subPath, unless the human administrator would like to manage it externally via ExtraMounts (where mounts are realized using SubPath to avoid directory overlapping).

PVC

External is not the default option though, and when a GlanceAPI instance is deployed, a PVC is created and bound to /var/lib/glance according to the StorageClass and StorageRequest passed as input.

...
default:
  replicas: 3
  storage:
    storageClass: local-storage
    storageRequest: 10G
...

In this case, if Ceph is set as a backend, no dedicated image conversion PVC is created, and the human operator must think about the PVC sizing in advance: the size of the PVC should be at least up to the largest converted image size. Concurrent conversions within the same Pod might be problematic in terms of PVC size: a conversion will fail or can't take place if the PVC is full and there's no enough space, and the upload should be retried after the previous conversion is over and the staging area space is released. However, concurrent conversion operations might happen in different Pods: it's recommended to deploy at least 3 replicas for a particular glanceAPI, and this helps to handle heavy operations like image conversion. While multiple replicas help to handle this problem better, if a PVC layout is chosen it's important to highlight that the Storage requirements in terms of sizing linearly increase with the number of glanceAPI instances and replicas.

num pvc = ∑([(local + image_cache) * num_replicas]) * layout
storage size = ∑([(storageRequest + imageCacheSize) * num_replicas]) * layout

where:

  • local: Local PVC bound to /var/lib/glance
  • image_cache: Cache PVC bound to /var/lib/glance/image-cache
  • ∑ - API: we sum the resulting value for each API
  • num_replicas: number of replicas for each glanceAPI (default is 3)
  • layout: split=2 Pods , single=1 Pod, edge=1 Pod

For a PVC based layout, the scale out of a glanceAPI in terms of replicas is limited by the available Storage provided by the storageClass, and depends on the StorageRequest. The StorageRequest is a critical parameter, it can be globally defined for all the glanceAPI, or defined with a different value for each API, and it will influence scale out operations for each of them. Other than a local PVC required to for the staging area, it is possible to enable image cache, which is translated into an additional PVC bound to each glanceAPI instance. A glance-cache PVC is bound to /var/lib/glance/image-cache, and the operator is responsible to configure the instance accordingly, setting both image_cache_max_size and the image_cache_dir parameters. The number of image cache PVCs follows the same rules described for the local PVC, and the number of requested PVCs is proportional to the number of replicas. In addition, it contributes to set the scale out limits discussed above.

Plan for a GlanceAPI deployment

As per the assumptions described above, here's a few examples of GlanceAPIs requirements in a common scenario based on the PVC model.

Example 1: PVC model without Image Cache

- GlanceAPI = 1
- num_replicas = 3
- no image cache
- storageRequest = 30G
- layout: single=1


num_pvc = (1 + 0) * 3 = 3 * 1 = 3
storage_size = 1 * [(30 + 0) * 3] = 90G

We can conclude that 90G is required for a single glanceAPI with 3 replicas.

Example 2: PVC model with Image Cache

- GlanceAPI: 3
- num_replicas = 3
- image_cache_size = 20G
- storageRequest = 30G
- layout: edge (edge = 1)

api0
 -> replica1 (20 + 30 = 50G)
 -> replica2 (20 + 30 = 50G)  => 150G (6 PVCs)
 -> replica3 (20 + 30 = 50G)
api1
 -> replica1 (20 + 30 = 50G)
 -> replica2 (20 + 30 = 50G)  => 150G (6 PVCs)
 -> replica3 (20 + 30 = 50G)
api2
 -> replica1 (20 + 30 = 50G)
 -> replica2 (20 + 30 = 50G)  => 150G (6 PVCs)
 -> replica3 (20 + 30 = 50G)

num_pvc = [(3 + 3) * 3] * 1 = 18
storage_size = 3 * [(30 + 20) * 3] = 450G

We can conclude that 150G is required for each API: given we have 3 APIs the total number is 450G.

Example 3: Mixed deployment with multiple GlanceAPI

Assume we have the following Glance CR

...
  glanceAPIs:
    api0:
      type: single
      replicas: 3
    api1:
      type: single
      replicas: 3
      storage:
        external: true
    api2:
      type: single
      replicas: 3
      imageCache:
        size: 5G
      storage:
        storageRequest: 30G
        storageClass: local-storage
    api3:
      type: single
      replicas: 3
      storage:
        external: true
  secret: osp-secret
  storage:
    storageClass: local-storage
    storageRequest: 10G
...

As per the CR above, we can gather the following information.


api0
---
- num_replicas = 3
- no image Cache
- storageRequest = 10G (inherited by the top-level definition)

num_pvc = 3
storage_size = 10 *3 = 30G

api1
---
- num_replicas = 3
- no image cache (external = true)
- no storageRequest (external = true)

num_pvc = 0
storage_size = 0

The storage should be externally provisioned

api2
---
- num_replicas = 3
- image_cache_size = 5G
- storageRequest = 30G
- layout = single

num_pvc = 2 * 3 = 6
storage_size = (30 + 5) * 3 = 105G

api3
---
- num_replicas = 3
- image_cache_size = 2G
- no storageRequest (external =  true)

The storage should be externally provisioned

num pvc = ∑([(local + image_cache) * num_replicas]) * layout
num_pvc = Sum(3 + 0 + 6 + 0) =  9
storage_size = Sum(30 + 0 + 105 + 0) = 135G

Note: we assume a local-storage storageClass exists and it's the only one used to keep the example simple, and only single layout is used. In case of split layout, the same API will double the required size.