IA-4812 Optimize CI: Deploy for IASO by oschvr · Pull Request #3013 · BLSQ/iaso

oschvr · 2026-05-12T08:22:24Z

What problem is this PR solving?

Create an optimised CI setup for IASO that auto deploys to ElasticBeanstalk, using UV (~60% faster) and Docker HUB cache

Related JIRA tickets

IA-4812

Changes

Explain the changes that were made.

Moved the dependency builder to uv for faster build
Added caching (shared as of now) that users the registry to cache docker build layers
Using EB cli to deploy (through the eb_deploy.py python script) to autodeploy to ElasticBeanstalk
Bumped django-celery-beat to 2.6.0 for compatibility
Documented how to build locally

How to test

Push any change to this PR, this action will trigger

Notes

The action deploy_docker_manual.yml in this PR is a WIP, and is intended to replace deploy.yml as in when it is ready.

Doc

README.md

tdethier

Still need to test, so I'm leaving comments only

tdethier · 2026-05-13T16:17:12Z

 celery==5.3.6  # https://github.com/celery/celery/tags
 django-allauth==0.54.0  # https://github.com/pennersr/django-allauth/tags
-django-celery-beat==2.5.0  # https://github.com/celery/django-celery-beat/tags
+django-celery-beat==2.6.0  # https://github.com/celery/django-celery-beat/tags


What was the issue that caused you to upgrade?

tdethier · 2026-05-13T16:17:35Z

+  platform_version: null
+  profile: null
+  sc: git
+  workspace_type: Application


You still don't have empty lines at the end of files 😅

tdethier · 2026-05-13T16:19:29Z

@@ -0,0 +1,15 @@
+branch-defaults:


Should this file be part of the repo? Because I can use eb commands on the upgrade server without needing to include this file, I'm wondering why we have it

It's needed for testing the CI of this branch, but we need to remove once it's ready tomerge

tdethier · 2026-05-13T16:20:41Z

+      - name: Build and push
+        uses: docker/build-push-action@v7
+        with:
+          file: ./docker/prod/Dockerfile


If you move the dockerfile to root, shouldn't you use the one there?

very nice catch

…k strings

…m 11.13.0

tdethier

Alright I think we've more or less covered the deployment CI, let's prepare the one that runs in PRs

tdethier · 2026-05-20T10:33:13Z

+To tag a specific version instead of `latest`:
+
+```bash
+docker build -f ./docker/prod/Dockerfile -t blsq/iaso:your-tag-here --push .


We should add IMAGE_TAG in .env.example

And maybe rename it to PROD_DOCKER_IMAGE_TAG?

tdethier · 2026-05-20T11:01:26Z

-        WFP_EMAIL_RECIPIENTS_NEW_ACCOUNT: ${WFP_EMAIL_RECIPIENTS_NEW_ACCOUNT}
-        WORKER: ${WORKER}
-
+        - APP_TITLE


Note for later: white listing env vars is very annoying for plugins because their env vars need to be white listed too. I guess each plugin will need another docker compose file that extends this one in order to do that?

yes that is a great idea, the base (vanilla IASO) docker compose can extend the plugins. I guess that's the next step

tdethier · 2026-05-20T11:09:08Z

 celery==5.3.6  # https://github.com/celery/celery/tags
 django-allauth==0.54.0  # https://github.com/pennersr/django-allauth/tags
-django-celery-beat==2.5.0  # https://github.com/celery/django-celery-beat/tags
+django-celery-beat==2.6.0  # https://github.com/celery/django-celery-beat/tags


+    runs-on: ubuntu-24.04
+    outputs:
+      branch: ${{ steps.vars.outputs.branch }}
+      image_tag: ${{ steps.vars.outputs.image_tag }}
+      environment: ${{ steps.vars.outputs.environment }}
+      slack_channel: ${{ steps.vars.outputs.slack_channel }}
+    steps:
+      - name: Set configs
+        id: vars
+        env:
+          DEFAULT_GIT_REFERENCE: main
+          DEFAULT_ENVIRONMENT: upgrade
+          DEFAULT_SLACK_CHANNEL: cloud-alerts
+        run: |
+          echo "branch=${{ github.event.inputs.branch || github.ref_name || env.DEFAULT_GIT_REFERENCE }}" >> $GITHUB_OUTPUT
+          echo "image_tag=${{ github.event.inputs.tag || 'latest' }}" >> $GITHUB_OUTPUT
+          echo "environment=${{ github.event.inputs.environment || env.DEFAULT_ENVIRONMENT }}" >> $GITHUB_OUTPUT
+          echo "slack_channel=${{ github.event.inputs.slack_channel || env.DEFAULT_SLACK_CHANNEL }}" >> $GITHUB_OUTPUT
+
+  build:


+    needs: config
+    uses: ./.github/workflows/docker_build.yml
+    secrets: inherit
+    with:
+      branch: ${{ needs.config.outputs.branch }}
+      image_tag: ${{ needs.config.outputs.image_tag }}
+
+  deploy:


+    runs-on: ubuntu-24.04
+    steps:
+      - name: Check out repository code
+        uses: actions/checkout@v6
+        with:
+          ref: ${{ inputs.branch }}
+          fetch-depth: 0
+
+      - name: Setup
+        uses: ./.github/actions/setup
+        with:
+          docker_username: ${{ secrets.DOCKER_USERNAME }}
+          docker_password: ${{ secrets.DOCKER_PASSWORD }}
+
+      - name: Building IASO image based on branch ${{ inputs.branch }}
+        run: |
+          echo "
+           ___    _    ____   ___
+          |_ _|  / \  / ___| / _ \\
+           | |  / _ \ \___ \| | | |
+           | | / ___ \ ___) | |_| |
+          |___/_/   \_\____/ \___/
+          ------------------------
+          Building IASO image based on branch ${{ inputs.branch }}
+          $(date)
+          "
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v4
+
+      - name: Checking for existing image with tag ${{ inputs.image_tag }}
+        run: |
+          echo "Checking for existing image with tag ${{ inputs.image_tag }}"
+          if docker pull blsq/iaso:${{ inputs.image_tag }}; then
+            echo "Image blsq/iaso:${{ inputs.image_tag }} already exists. Skipping build and push."
+            echo "IMAGE_EXISTS=true" >> $GITHUB_ENV
+          else
+            echo "Image blsq/iaso:${{ inputs.image_tag }} does not exist. Proceeding with build and push."
+            echo "IMAGE_EXISTS=false" >> $GITHUB_ENV
+          fi
+
+      - name: Build and push image
+        if: env.IMAGE_EXISTS != 'true'
+        uses: docker/build-push-action@v7
+        with:
+          file: docker/prod/Dockerfile
+          push: true
+          tags: blsq/iaso:${{ inputs.image_tag }}
+          # https://docs.docker.com/build/cache/backends/
+          cache-from: type=registry,ref=blsq/iaso:buildcache-${{ inputs.image_tag }}
+          cache-to: type=registry,ref=blsq/iaso:buildcache-${{ inputs.image_tag }},mode=max


+    needs: [config, build]
+    runs-on: ubuntu-24.04
+    steps:
+      - name: Check out repository code
+        uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      - name: Setup
+        uses: ./.github/actions/setup
+        with:
+          docker_username: ${{ secrets.DOCKER_USERNAME }}
+          docker_password: ${{ secrets.DOCKER_PASSWORD }}
+
+      - name: Get version name
+        id: version
+        run: |
+          git describe --tags
+          echo "VERSION_NAME=$(git describe --tags --match "v[[:digit:]]*")" >> $GITHUB_OUTPUT
+
+      - name: Slack deployment start
+        uses: rtCamp/action-slack-notify@v2
+        env:
+          SLACK_WEBHOOK: ${{ secrets.TEST_SLACK_WEBHOOK }}
+          SLACK_CHANNEL: ${{ needs.config.outputs.slack_channel }}
+          SLACK_COLOR: ""
+          SLACK_USERNAME: ${{ needs.config.outputs.environment }}
+          SLACK_ICON_EMOJI: ":robot_face:"
+          SLACK_TITLE: Started deployment of ${{ steps.version.outputs.VERSION_NAME }} on ${{ needs.config.outputs.environment }} ...
+          SLACK_FOOTER: ${{  format('<https://github.com/{0}/commit/{1}/checks|🔗{2} logs> | <https://github.com/{0}/commit/{1}/checks|🔗 Commit {1}>', github.repository, github.sha, github.job) }}
+          MSG_MINIMAL: true
+
+      - name: Display banner
+        run: |
+          echo "
+           ___    _    ____   ___
+          |_ _|  / \  / ___| / _ \\
+           | |  / _ \ \___ \| | | |
+           | | / ___ \ ___) | |_| |
+          |___/_/   \_\____/ \___/
+          ------------------------
+          Deploying ${{ steps.version.outputs.VERSION_NAME }} to ${{ needs.config.outputs.environment }}
+          $(date)
+          "
+
+      - name: Tag image ${{ needs.config.outputs.image_tag }}
+        run: |
+          docker pull blsq/iaso:${{ needs.config.outputs.image_tag }}
+          docker tag blsq/iaso:${{ needs.config.outputs.image_tag }} blsq/iaso:${{ steps.version.outputs.VERSION_NAME }}
+          docker push blsq/iaso:${{ steps.version.outputs.VERSION_NAME }}
+
+      - name: Swap to prod docker config
+        run: |
+          echo "Replace docker-compose with prod version"
+          git add -f docker-compose.yml
+
+          echo "Move Dockerfile to root"
+          cp docker/prod/Dockerfile .
+
+          echo "Replace .platform with prod version"
+          rm -rf .platform
+          cp -r docker/prod/.platform .platform
+          git add -f .platform
+
+          echo "Remove dev eb extensions from git stage"
+          git rm -r --cached --quiet .ebextensions
+
+      - name: Set up Python 3.14
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.14
+
+      - name: Setup beanstalk
+        run: |
+          cp .elasticbeanstalk/config.upgrade.yml .elasticbeanstalk/config.yml
+          pip install awsebcli==3.27.1 botocore==1.42.92 boto3==1.42.92
+
+      - name: Deploy to beanstalk (Docker) using scripts/eb_deploy.py
+        run: python scripts/eb_deploy.py ${{ needs.config.outputs.environment }}
+        env:
+          IMAGE_TAG: ${{ steps.version.outputs.VERSION_NAME }}
+          VERSION_NAME: ${{ steps.version.outputs.VERSION_NAME }}
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          AWS_DEFAULT_REGION: "eu-central-1"
+
+      - name: Slack deployment success notification
+        uses: rtCamp/action-slack-notify@v2
+        env:
+          SLACK_WEBHOOK: ${{ secrets.TEST_SLACK_WEBHOOK }}
+          SLACK_CHANNEL: ${{ needs.config.outputs.slack_channel }}
+          SLACK_COLOR: ${{ job.status }}
+          SLACK_USERNAME: ${{ needs.config.outputs.environment }}
+          SLACK_ICON_EMOJI: ":robot_face:"
+          SLACK_TITLE: Iaso ${{ steps.version.outputs.VERSION_NAME }} successfully deployed on ${{ needs.config.outputs.environment }}
+          SLACK_MESSAGE: ${{  format('<https://github.com/{0}/commit/{1}/checks|🔗{2} logs> | <https://github.com/{0}/commit/{1}/checks|🔗 Commit {1}>', github.repository, github.sha, github.job) }}
+          SLACK_FOOTER: "-"
+          MSG_MINIMAL: true
+        if: success()
+
+      - name: Slack deployment failure notification
+        uses: rtCamp/action-slack-notify@v2
+        env:
+          SLACK_WEBHOOK: ${{ secrets.TEST_SLACK_WEBHOOK }}
+          SLACK_CHANNEL: ${{ needs.config.outputs.slack_channel }}
+          SLACK_USERNAME: ${{ needs.config.outputs.environment }}
+          SLACK_COLOR: ${{ job.status }}
+          SLACK_ICON_EMOJI: ":robot_face:"
+          SLACK_TITLE: Deployment failure Iaso ${{ steps.version.outputs.VERSION_NAME }} - ${{ needs.config.outputs.environment }}
+          SLACK_MESSAGE: ${{  format('<https://github.com/{0}/commit/{1}/checks|🔗{2} logs> | <https://github.com/{0}/commit/{1}/checks|🔗 Commit {1}>', github.repository, github.sha, github.job) }}
+          SLACK_FOOTER: "-"
+          MSG_MINIMAL: true
+        if: failure()


+    runs-on: ubuntu-24.04
+    steps:
+      - name: Check out repository code
+        uses: actions/checkout@v6
+
+      - name: Use node.js 22.18.0
+        uses: actions/setup-node@v4
+        with:
+          node-version: '22.18.0'
+          cache: npm
+
+      - name: upgrade npm
+        run: |
+          npm install --global npm@11.13.0
+
+      - name: Environment info
+        run: |
+          node --version
+          npm --version
+
+      - name: npm install and test
+        run: |
+          npm ci
+          npm test
+
+      - name: Check Frontend (JS/React) translations
+        run: |
+          python scripts/update_trads.py
+
+      - name: Slack notification
+        uses: rtCamp/action-slack-notify@v2
+        env:
+          SLACK_USERNAME: ${{ github.workflow }}
+          SLACK_ICON_EMOJI: ':robot_face:'
+          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
+          SLACK_CHANNEL: iaso-dev
+          SLACK_COLOR: ${{ job.status }}
+          SLACK_TITLE: ${{ format('Job {0} failed on {1}', github.job, github.ref) }}
+          SLACK_FOOTER: ${{  format('<https://github.com/{0}/commit/{1}/checks|🔗{2} logs> | <https://github.com/{0}/commit/{1}/checks|🔗 Commit {1}>', github.repository, github.sha, github.job) }}
+          MSG_MINIMAL: true
+        if: failure()
+
+  build-and-push-image:


+    runs-on: ubuntu-24.04
+    services:
+      postgres:
+        image: postgis/postgis:16-3.5
+        env:
+          POSTGRES_USER: postgres
+          POSTGRES_PASSWORD: postgres
+          POSTGRES_DB: github_actions
+        ports:
+          - 5432:5432
+        # needed because the postgres container does not provide a health check
+        options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
+    env:
+      DEBUG: true
+      DJANGO_SETTINGS_MODULE: hat.settings
+      PLUGINS: polio,wfp,wfp_auth
+      RDS_DB_NAME: github_actions
+      SECRET_KEY: secret
+      RDS_HOSTNAME: localhost
+      RDS_PASSWORD: postgres
+      RDS_PORT: 5432
+      RDS_USERNAME: postgres
+    steps:
+      - name: Check out repository code
+        uses: actions/checkout@v6
+
+      - name: Install Python 3.9
+        uses: actions/setup-python@v6
+        with:
+          python-version-file: "pyproject.toml"
+
+      - name: Install GDAL and gettext
+        run: sudo apt-get update && sudo apt install gdal-bin gettext
+
+      - name: Install uv and setup cache
+        uses: astral-sh/setup-uv@v8.1.0
+        with:
+          enable-cache: true
+          cache-dependency-glob: "uv.lock"
+
+      - name: Install backend dependencies
+        run: uv sync --locked
+
+      - name: Environment info
+        run: uv tree --all-groups
+
+      - name: Check Backend (Django) translations
+        run: |
+          uv run python manage.py make_translations
+          uv run python manage.py compile_translations
+
+      - name: Check for missing migrations
+        run: uv run python manage.py makemigrations --check
+
+      - name: Django tests
+        run: |
+          uv run python manage.py migrate
+          uv run python manage.py createcachetable
+          uv run python -W "ignore" manage.py test
+
+      - name: Slack notification
+        uses: rtCamp/action-slack-notify@v2
+        env:
+          SLACK_USERNAME: ${{ github.workflow }}
+          SLACK_ICON_EMOJI: ':robot_face:'
+          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
+          SLACK_CHANNEL: iaso-dev
+          SLACK_COLOR: ${{ job.status }}
+          SLACK_TITLE: ${{ format('Job {0} failed on {1}', github.job, github.ref) }}
+          SLACK_FOOTER: ${{  format('<https://github.com/{0}/commit/{1}/checks|🔗{2} logs> | <https://github.com/{0}/commit/{1}/checks|🔗 Commit {1}>', github.repository, github.sha, github.job) }}
+          MSG_MINIMAL: true
+        if: failure()
+
+  backend_lint_and_format:


+    runs-on: ubuntu-24.04
+    steps:
+      - name: Check out repository code
+        uses: actions/checkout@v6
+
+      - name: Install ruff
+        uses: astral-sh/ruff-action@v4.0.0
+        with:
+          args: "--version"
+          version: "0.9.6"
+
+      - name: Linting with ruff
+        run: ruff check --output-format=github .
+
+      - name: Formatting with ruff
+        run: ruff format --check .


github-advanced-security AI found potential problems May 12, 2026

View reviewed changes

Comment thread .github/workflows/deploy_docker_manual.yml Fixed

tdethier force-pushed the IA-4812-clean-and-optimize-c-is-to-build-images-for-deployment-rebase branch from dd70aed to 5a0bf7d Compare May 13, 2026 16:15

tdethier reviewed May 13, 2026

View reviewed changes

oschvr and others added 6 commits May 19, 2026 11:53

IA-4812 Rebase develop into IA-4812

b8b9e48

IA-4812 Add missing dependency (environs) to pyproject.toml

c14faf1

IA-4812 Add missing dependencies to pyproject.toml

22f9f25

IA-4812 Document uv. Test CI with rebase

68104fb

IA-4812 Use env vars as they are withouth interpolation to avoid blan…

6d774b7

…k strings

update pyproject.toml

4dbb0de

tdethier force-pushed the IA-4812-clean-and-optimize-c-is-to-build-images-for-deployment-rebase branch from 5a0bf7d to 4dbb0de Compare May 19, 2026 09:55

tdethier and others added 7 commits May 19, 2026 12:13

add blank lines

a247a57

IA-4812 Ammend comments. Add input to override latest tag. Restore np…

88d48a9

…m 11.13.0

IA-4812 Fix action tag input as empty

985aa13

IA-4812 Fix Dockerfile path

c432ec6

IA-4812 Restore Dockerfile path

7715da8

IA-4812 Remove cp of config.upgrade.yml

a7d385b

IA-4812 Restore config.upgrade.yml

55db510

tdethier reviewed May 20, 2026

View reviewed changes

oschvr and others added 2 commits May 21, 2026 10:08

IA-4812 Rename IMAGE_TAG to PROD_DOCKER_IMAGE_TAG. Update .env.example

67d3e3c

build test CI + refactor

d9a3754