|
| 1 | +# Installing DiracX in a container |
| 2 | + |
| 3 | +!!! warning "This is for container deployment only. For kubernetes deployment please refer to [Installing DiracX](../install/installing.md)." |
| 4 | + |
| 5 | +## Notes |
| 6 | + |
| 7 | +- We are running a 'developer' setup where we can edit the diracx code directly and run the local code |
| 8 | + inside the container. This really helps with debugging ;-) |
| 9 | +- Similar to DIRAC this is installed as the (unprivileged) dirac user (and not root). There might be limitations to the approach that we have not found yet. |
| 10 | +- We give the version numbers of the code we tested, you should always check for the latest version and install this. If in doubt, please as on the DiracX [mattermost channel](https://mattermost.web.cern.ch/diracx/channels/town-square). In this example we used DIRAC v9.0.2, diracx 0.0.12 and diracx-web v0.1.0-a10. |
| 11 | +- If the same version number has to be specified in two different places, we indicate this explicitly. |
| 12 | +- We keep all of our configuration files in a folder called dirac-container. This name is chosen arbitrarily, and can be replaced with a name of your choice. However, the configuration examples assume that all files needed are kept in the same directory. |
| 13 | + |
| 14 | +## Configuration Files |
| 15 | + |
| 16 | +There are four configuration files that need to be provided to install DiracX in a container: |
| 17 | +`diracx.env`, `diracx-web.env`, `podman-compose.yaml` and `jwks.json`. |
| 18 | +In addition we also overwrite the entry point to include the database initialisation/schema management. |
| 19 | +You also need your OpenSearch server's `/etc/opensearch/root-ca.crt` (here called opensearch-ca.pem) to be able to connect to your OpenSearch server. |
| 20 | +Both `entrypoint.sh` and `opensearch-ca.pem` should be placed in the same folder as the configuration files. |
| 21 | + |
| 22 | +`diracx.env` is based on [reference/env-variables](../../reference/env-variables.md) <br> |
| 23 | +`jwks.json` is used to to generate the diracx authentication tokens. Please also see [how to rotate a secret](../../how-to/rotate-a-secret.md) <br> |
| 24 | +`podman-compose.yaml` is the container steering/manifest file. <br> |
| 25 | + |
| 26 | +### jwks.json |
| 27 | + |
| 28 | +To generate the jwks.json use a diracx container and start it with a shell (code adapted from `diracx/run_local.sh`): <br> |
| 29 | +`podman run --rm -ti ghcr.io/diracgrid/diracx/services:v0.0.12 /bin/bash` |
| 30 | +The --rm removes the container automatically after exiting. |
| 31 | +`python -m diracx.logic rotate-jwk --jwks-path "/tmp/jwks.json"` |
| 32 | +Then copy content of /tmp/jwks.json to file outside of the container to be mapped in a volume to the real thing. |
| 33 | + |
| 34 | +### diracx.env |
| 35 | + |
| 36 | +At this point you need to have created the `DiracXAuthDB` and the `jwks.json` (see above). |
| 37 | +Now you need two further keys: |
| 38 | + |
| 39 | +`DIRACX_SERVICE_AUTH_STATE_KEY`: Please see also [env-variables/diracx_service_auth_state_key](../../reference/env-variables.md#diracx_service_auth_state_key) and [dynamic-secrets-diracx-dynamic-secrets](../../explanations/chart-structure.md#dynamic-secrets-diracx-dynamic-secrets) |
| 40 | +To generate the key we follow diracx/run_local.sh: `state_key="$(head -c 32 /dev/urandom | base64)"` |
| 41 | + |
| 42 | +`DIRACX_LEGACY_EXCHANGE_HASHED_API_KEY`: |
| 43 | +This is adapted from ["generating a legacy exchange api key"] (connect.md) ("generating a legacy exchange api key") goes into dirac.cfg (as diracx:legacy...) and into diracx.env as a hash, but the gist of it is, the shared secret allows dirac and diracx to communicate. |
| 44 | +To generate the legacy key, you can use the following python snippet: |
| 45 | + |
| 46 | +```python |
| 47 | +import secrets |
| 48 | +import base64 |
| 49 | +import hashlib |
| 50 | + |
| 51 | +token = secrets.token_bytes() |
| 52 | +# This is the secret to include in the request by setting the |
| 53 | +# /DiracX/LegacyExchangeApiKey CS option in your legacy DIRAC installation (in the local -- secluded -- dirac.cfg file) |
| 54 | +print(f"API key is diracx:legacy:{base64.urlsafe_b64encode(token).decode()}") |
| 55 | + |
| 56 | +# This is the environment variable to set on the DiracX server |
| 57 | +print(f"DIRACX_LEGACY_EXCHANGE_HASHED_API_KEY={hashlib.sha256(token).hexdigest()}") |
| 58 | +``` |
| 59 | + |
| 60 | +Below is an example of a dirac.env file. The server is called 'diractest.grid.hep.ph.ic.ac.uk'. Replace as necessary. |
| 61 | + |
| 62 | +``` |
| 63 | +DIRACX_SERVICE_AUTH_STATE_KEY=[state_key from above] |
| 64 | +DIRACX_SERVICE_AUTH_TOKEN_ISSUER=["Your hostname here", e.g. "https://diractest.grid.hep.ph.ic.ac.uk"] |
| 65 | +DIRACX_SERVICE_AUTH_TOKEN_KEYSTORE=file:///keystore/jwks.json [The one you made earlier] |
| 66 | +# replace diractest.grid.hep.ph.ic.ac.uk with your hostname |
| 67 | +DIRACX_SERVICE_AUTH_ALLOWED_REDIRECTS='["https://diractest.grid.hep.ph.ic.ac.uk/#authentication-callback"]' |
| 68 | +# we are currently not using S3, nevertheless these variables need to be set |
| 69 | +DIRACX_SANDBOX_STORE_S3_CLIENT_KWARGS="{}" |
| 70 | +DIRACX_SANDBOX_STORE_BUCKET_NAME="notsetyet" |
| 71 | +# disable the jobs router (and hopefully S3) |
| 72 | +DIRACX_SERVICE_JOBS_ENABLED=false |
| 73 | +# we are using a local git repo on the node, this is not fully supported yet. |
| 74 | +# Please see: https://diracx.io/en/latest/RUN_PROD/#cs for the approved way. |
| 75 | +DIRACX_CONFIG_BACKEND_URL="git+file:///cs_store?revision=main" |
| 76 | +DIRACX_DB_URL_AUTHDB=mysql+aiomysql://YourUser:YourPassword@host.containers.internal:3306/DiracXAuthDB |
| 77 | +# these are the users and passwords you have chosen for the databases in DIRAC; they can be found in |
| 78 | +# dirac.cfg |
| 79 | +# host.containers.internal: For the development node our databases are on the same machine |
| 80 | +DIRACX_DB_URL_JOBDB=mysql+aiomysql://YourUser:YourPassword@host.containers.internal:3306/JobDB |
| 81 | +DIRACX_DB_URL_JOBLOGGINGDB=mysql+aiomysql://YourUser:YourPassword@host.containers.internal:3306/JobLoggingDB |
| 82 | +DIRACX_DB_URL_PILOTAGENTSDB=mysql+aiomysql://YourUser:YourPassword@host.containers.internal:3306/PilotAgentsDB |
| 83 | +DIRACX_DB_URL_SANDBOXMETADATADB=mysql+aiomysql://YourUser:YourPassword@host.containers.internal:3306/SandboxMetadataDB |
| 84 | +DIRACX_DB_URL_TASKQUEUEDB=mysql+aiomysql://YourUser:YourPassword@host.containers.internal:3306/TaskQueueDB |
| 85 | +# opensearch related; the user and password are specific to opensearch, |
| 86 | +DIRACX_OS_DB_JOBPARAMETERSDB={"hosts":"YourOpensearchUsername:YourOpensearchPwd@YourOpensearchServer:9200", "use_ssl":true, "ca_certs":"/etc/opensearch-ca.pem"} |
| 87 | +DIRACX_OS_DB_PILOTLOGSDB={"hosts":"YourOpensearchUsername:YourOpensearchPwd@YourOpenSearchServer:9200", "use_ssl":true, "ca_certs":"/etc/opensearch-ca.pem"} |
| 88 | +DIRACX_LEGACY_EXCHANGE_HASHED_API_KEY=(from the output from generating the legacy key) |
| 89 | +``` |
| 90 | + |
| 91 | +### diracx-web.env |
| 92 | + |
| 93 | +This file contains one line: |
| 94 | +`DIRACX_URL=https://diractest.grid.hep.ph.ic.ac.uk/api` |
| 95 | + |
| 96 | +### podman-compose.yaml |
| 97 | + |
| 98 | +Note: Please check that your ports specified in the yaml file match the ports you set up during the preparing a node for a container install procedure. The DiracX version specified must match the ones you use when you install the code. The `uvicorn` command is what actually starts diracx services. <br> |
| 99 | +Sources: <br> |
| 100 | +`ghcr.io`: from https://diracx.diracgrid.org/en/latest/RUN_PROD/#diracx-service-configuration <br> |
| 101 | +`command`: from diracx-charts/diracx/templates/diracx/deployment.yaml <br> |
| 102 | +`/opt/dirac/diracx-config:/cs_store:z,rw`: only needed because we use a local repo for the CS, a remote repo would be read directly over https. <br> |
| 103 | + |
| 104 | +``` |
| 105 | +--- |
| 106 | +services: |
| 107 | + diracx-services: |
| 108 | + ports: |
| 109 | + - 127.0.0.1:8000:8000 |
| 110 | + image: ghcr.io/diracgrid/diracx/services:v0.0.12 |
| 111 | + env_file: "diracx.env" |
| 112 | + command: "uvicorn --factory diracx.routers:create_app --host=0.0.0.0 --port=8000 --proxy-headers --forwarded-allow-ips=*" |
| 113 | + volumes: |
| 114 | + - ./entrypoint.sh:/entrypoint.sh:z,ro |
| 115 | + - ./jwks.json:/keystore/jwks.json:z,ro |
| 116 | + - /opt/dirac/diracx-config:/cs_store:z,rw |
| 117 | + - /opt/dirac/diracx:/diracx_sources:z,ro |
| 118 | + - ./opensearch-ca.pem:/etc/opensearch-ca.pem:z,ro |
| 119 | + diracx-web: |
| 120 | + image: ghcr.io/diracgrid/diracx-web/static:v0.1.0-a10 |
| 121 | + ports: |
| 122 | + - 127.0.0.1:8001:8080 |
| 123 | + env_file: "diracx-web.env" |
| 124 | +``` |
| 125 | + |
| 126 | +### entrypoint.sh |
| 127 | + |
| 128 | +```bash |
| 129 | +#!/bin/bash |
| 130 | + |
| 131 | +set -e |
| 132 | +echo "Welcome to the dark side" |
| 133 | + |
| 134 | +if [ -f /activate.sh ]; then |
| 135 | + source /activate.sh |
| 136 | +else |
| 137 | + eval "$(micromamba shell hook --shell=posix)" && micromamba activate base |
| 138 | +fi |
| 139 | + |
| 140 | +# this sets it up so that we can make changes to the code |
| 141 | +pip install -e /diracx_sources/diracx-core \ |
| 142 | + -e /diracx_sources/diracx-db \ |
| 143 | + -e /diracx_sources/diracx-logic \ |
| 144 | + -e /diracx_sources/diracx-routers \ |
| 145 | + -e /diracx_sources/diracx-client |
| 146 | + |
| 147 | +# initialise database and make updates if necessary |
| 148 | +python -m diracx.db init-sql |
| 149 | + |
| 150 | +exec "$@" |
| 151 | +``` |
| 152 | + |
| 153 | +## Install the code |
| 154 | + |
| 155 | +```bash |
| 156 | +git clone https://github.com/DIRACGrid/diracx.git |
| 157 | +git checkout v0.0.12 |
| 158 | +``` |
| 159 | + |
| 160 | +## Tests |
| 161 | + |
| 162 | +Once the containers are installed, the following tests should be working (note: when testing, we needed to run the |
| 163 | +first command twice as there seemed to be a bug in DiracX. This might be fixed by the time you are trying this.) |
| 164 | +We ran these as root, as we were setting up our apache config as root and debugging this config as part of the process, but you can run this as any user. |
| 165 | + |
| 166 | +``` |
| 167 | +[root@diractest ~]# curl -k https://diractest.grid.hep.ph.ic.ac.uk/.well-known/openid-configuration` |
| 168 | +{"issuer":"https://diractest.grid.hep.ph.ic.ac.uk","token_endpoint":"https://diractest.grid.hep.ph.ic.ac.uk/api/auth/token","userinfo_endpoint":"https://diractest.grid.hep.ph.ic.ac.uk/api/auth/userinfo","authorization_endpoint": |
| 169 | +[etc] |
| 170 | +``` |
| 171 | + |
| 172 | +``` |
| 173 | +[root@diractest ~]# curl -k https://diractest.grid.hep.ph.ic.ac.uk/.well-known/jwks.json` |
| 174 | +{"keys":[{"crv":"Ed25519","x":"3_SrPXQyalji7nL4fNFh7JGBTwQBztmjnW7ogFusiPs","key_ops":["sign","verify"],"alg":"EdDSA","kid":"019a3a1b95b77b11a18bde7813b78fe2","kty":"OKP"}]} |
| 175 | +``` |
| 176 | + |
| 177 | +To test the web container: |
| 178 | + |
| 179 | +```bash |
| 180 | +$ curl -k https://diractest.grid.hep.ph.ic.ac.uk/ |
| 181 | +<!DOCTYPE html><html lang="en"><head>[etc] (i.e. returns a web site) |
| 182 | +``` |
| 183 | + |
| 184 | +## A selection of helpful podman commands |
| 185 | + |
| 186 | +``` |
| 187 | +podman ps -a # see all containers, dead or alive |
| 188 | +podman rm diracx-container_diracx-services_1 # remove dead container by name |
| 189 | +podman-compose -f podman-compose.yaml up -d --force-recreate # restart all containers (e.g. after update) |
| 190 | +podman exec -ti diracx-container_diracx-services_1 /bin/bash # to look into a running container |
| 191 | +podman logs -f diracx-container_diracx-services_1 # to look at the logs of a running container |
| 192 | +# clean up |
| 193 | +podman images # list all images |
| 194 | +podman image prune # should delete all unused images |
| 195 | +podman image rm [image id] # in case the automated delete doesn't work |
| 196 | +``` |
0 commit comments