Skip to content

openshift-deployment

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

openshift-deployment

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
README.rst
README.rst
 
 
openshift-dev-export.yaml
openshift-dev-export.yaml
 
 

README.rst

Deploying to an OpenShift cluster

As an example, these are the instructions for restoring the deployment on https://okd-dev.hbp.eu/.

  1. You can use the deployment configuration saved in openshift-dev-export.yaml provided in the repository as a starting point. Edit the route contained in this file to use the correct URL.
  2. Create the project named cortical-voluba on https://okd-dev.hbp.eu/
  3. Log in using the command-line oc tool (https://okd-dev.hbp.eu/console/command-line), switch to the cortical-voluba project with oc project cortical-voluba
  4. Import the objects from your edited YAML file using oc create -f openshift-dev-export.yaml
  5. Re-create the Persistent Volume Claims and upload the data (see below).
  6. Edit the Config Maps if needed, re-create the needed Secrets (namely redis/database-password and github-webhook-secret).
  7. Start the build. The deployment should follow automatically.
  8. Go to Builds -> Builds -> cortical-voluba -> Configuration, copy the GitHub Webhook URL and configure it into the GitHub repository (https://github.com/HumanBrainProject/cortical-voluba/settings/hooks). Make sure to set the Content Type to application/json.

Deployment on okd-dev.hbp.eu

The deployment configuration is saved to openshift-dev-export.yaml by running oc get -o yaml --export is,bc,dc,svc,route,pvc,cm,horizontalpodautoscaler > openshift-dev-export.yaml (status, resourceVersion, generation, @sha256, PersistentVolumeClaim metadata (volumeName, finalizers, annotations) and secret information is stripped manually, see https://collab.humanbrainproject.eu/#/collab/38996/nav/270508 for other edits that may be necessary).

For the record, here are the steps that were used to create this OpenShift project on https://okd-dev.hbp.eu/:

  1. Create the project / navigate to the project

  2. Configure the Flask instance

    1. Add to Project -> Browse Catalog

    2. Choose Python (does not matter, configuration will be changed later). Hit Next

    3. In Step 2 (Configuration), hit advanced options and enter these values:

      • cortical-voluba-flask as Name
      • https://github.com/HumanBrainProject/cortical-voluba.git as Git Repository
      • dev as Git Reference
      • Under Routing, enter cortical-voluba.apps-dev.hbp.eu as Hostname
      • Under Routing, check Secure route
      • Under Routing, set Insecure Traffic to Redirect
      • Under Build Configuration, uncheck Launch the first build when the build configuration is created

    4. Hit Create at the bottom of the page

    5. Follow the instructions to configure the GitHub webhook

      1. (optional) If you are going to publish the OpenShift deployment configuration, make sure that the webhook secrets refer to a real Secret resource (e.g. github-webhook-secret) instead of being stored in clear-text in the BuildConfig object.

    6. Change the build configuration to use the Docker build strategy:

      1. Go to Builds -> Builds -> cortical-voluba-flask -> Actions -> Edit YAML

      2. Replace the contents of the strategy key by:

        dockerStrategy:
  dockerfilePath: Dockerfile.flask
type: Docker

      3. Hit Save

    7. Add post-build tests and tweak build configuration

      1. Go to Builds -> Builds -> cortical-voluba-flask -> Actions -> Edit. Click on advanced options.

      2. Under Image Configuration, check Always pull the builder image from the docker registry, even if it is present locally

      3. Under Post-Commit Hooks, check Run build hooks after image is built. Choose Hook Type = Shell Script and enter the following Script:

        set -e
# Without PIP_IGNORE_INSTALLED=0 the Debian version of pip would
# re-install all dependencies in the user's home directory
# (https://github.com/pypa/pip/issues/4222#issuecomment-417672236)
PIP_IGNORE_INSTALLED=0 python3 -m pip install --user /source[tests]
cd /source
python3 -m pytest tests/

      4. Hit Save

    8. Trigger the build by hitting Start Build

    9. Configure the Flask instance

      1. Go to Applications -> Deployments -> cortical-voluba-flask -> Configuration

      2. Under Volumes, hit Add Config Files

      3. Click Create Config Map

        • Name = instance-dir

        • Key = config.py

        • Value:

          import os
CELERY_BROKER_URL = 'redis://:' + os.environ['REDIS_PASSWORD'] + '@redis:6379/0'
CELERY_RESULT_BACKEND = CELERY_BROKER_URL
CORS_ORIGINS = r'https://.*\.(hbp\.eu|humanbrainproject\.eu|humanbrainproject\.org)'
PROXY_FIX = {
    'x_for': 1,
    'x_host': 1,
    'x_port': 1,
    'x_proto': 1,
}
TEMPLATE_EQUIVOLUMETRIC_DEPTH = '/static-data/BigBrain_equivolumetric_depth.nii.gz'

      4. Hit Create

      5. Back in the Add Config Files page, choose the newly created instance-dir as Source

      6. Set Mount Path = /instance

      7. Hit Add

      8. Go to the Environment tab and add these variables:

        • INSTANCE_PATH = /instance
        • REDIS_PASSWORD from Secret redis/database-password

    10. Add Health Checks #. Go to Applications -> Deployments -> cortical-voluba-flask -> Actions -> Edit Health Checks #. Add a Readiness Probe of type HTTP GET, using Path = /health, setting some Initial Delay (e.g. 5 seconds) and Timeout (e.g. 10 seconds) #. Add a Liveness Probe of type HTTP GET, using Path = /health, setting a long Timeout (e.g. 60 seconds) #. Hit Save

  3. Configure the Celery instance

    1. Add to Project -> Browse Catalog

    2. Choose Python (does not matter, configuration will be changed later). Hit Next

    3. In Step 2 (Configuration), hit advanced options and enter these values:

    4. Hit Create at the bottom of the page

    5. Follow the instructions to configure the GitHub webhook

      1. (optional) If you are going to publish the OpenShift deployment configuration, make sure that the webhook secrets refer to a real Secret resource (e.g. github-webhook-secret) instead of being stored in clear-text in the BuildConfig object.

    6. Change the build configuration to use the Docker build strategy:

      1. Go to Builds -> Builds -> cortical-voluba-celery -> Actions -> Edit YAML

      2. Replace the contents of the strategy key by:

        dockerStrategy:
  dockerfilePath: Dockerfile.celery
type: Docker

      3. Hit Save

    7. Add post-build tests and tweak build configuration

      1. Go to Builds -> Builds -> cortical-voluba-celery -> Actions -> Edit. Click on advanced options.

      2. Under Image Configuration, check Always pull the builder image from the docker registry, even if it is present locally

      3. Under Post-Commit Hooks, check Run build hooks after image is built. Choose Hook Type = Shell Script and enter the following Script:

        set -e
# Without PIP_IGNORE_INSTALLED=0 the Debian version of pip would
# re-install all dependencies in the user's home directory
# (https://github.com/pypa/pip/issues/4222#issuecomment-417672236)
PIP_IGNORE_INSTALLED=0 python3 -m pip install --user /source[tests]
cd /source
python3 -m pytest tests/

      4. Hit Save

    8. Trigger the build by hitting Start Build

    9. Configure the Celery instance

      1. Go to Applications -> Deployments -> cortical-voluba-celery -> Configuration
      2. Under Volumes, hit Add Config Files
      3. Set Source = instance-dir, Mount Path = /instance
      4. Hit Add
      5. Go to the Environment tab and add these variables:
        • INSTANCE_PATH = /instance
        • REDIS_PASSWORD from Secret redis/database-password
      6. (recommended) Add resource requests to ensure that Celery workers will have sufficient memory and CPU to perform the computations. 4 GiB of memory are needed to run the example, and a CPU request is necessary to activate CPU autoscaling. Ideally we would need 1 full CPU, but on okd-dev this fails because there are no suitable nodes.
      7. (optional) Add an Autoscaler so that OpenShift can automatically adapt the number of Celery workers to the number of ongoing computations: go to Actions -> Add Autoscaler, set Max pods to 5, and CPU Request Target to 50%. Validate by clicking Save.

    10. Create a volume to hold the static data (equivolumetric depth for BigBrain)

      1. Go to Applications -> Deployments -> cortical-voluba-celery -> Configuration
      2. Under Volumes, hit Add Storage
      3. Hit Create Storage
      4. Set Name = static-data, Size = 1 GiB
      5. Hit Create
      6. Set Mount Path = /static-data
      7. Set the mount to Read only
      8. Hit Add

    11. Upload the static data (equivolumetric depth for BigBrain). We follow the method described on https://blog.openshift.com/transferring-files-in-and-out-of-containers-in-openshift-part-3/

      1. Install the OpenShift Command-Line Tools by following the instructions on https://okd-dev.hbp.eu/console/command-line
      2. Log in using the CLI (Under your name on the top right corner, hit Copy Login Command and paste it into a terminal)
      3. Switch to the project (oc project cortical-voluba)
      4. Run a dummy pod for rsync transfer with oc run dummy --image ylep/oc-rsync-transfer
      5. Mount the volume against the dummy pod oc set volume dc/dummy --add --name=tmp-mount --claim-name=static-data --mount-path /static-data
      6. Wait for the deployment to be complete with oc rollout status dc/dummy
      7. Get the name of the dummy pod with oc get pods --selector run=dummy
      8. Copy the data using oc rsync --compress=true --progress=true static-data/ dummy-2-7tdml:/static-data/ (replace dummy-2-7tdml with the pod name from the previous step).
      9. Verify the contents of the directory with oc rsh dummy-2-7tdml ls -l /static-data
      10. Delete everything related to the temporary pod with oc delete all --selector run=dummy

    12. Add Health Checks (TODO: figure out how to check for celery worker, I could not figure out how to use celery inspect ping).

  4. Configure the Redis instance

    1. Add to project -> Browse Catalog
    2. Choose Redis (Ephemeral) (FIXME: production should probably use persistent storage)
    3. Under Configuration, leave default values
    4. Under Binding, choose Create a secret...
    5. Hit Create