ci: simplify deploy workflows — Terraform owns Container App config, CI owns image version (#1081)

## Summary

Now that Terraform manages the Container App configuration, the deploy
jobs no longer need to re-configure identity, registry, secrets, or env
vars on every run. This PR strips those steps and adds production
hardening.

## What changed

### Removed (Terraform owns these now)
- `az containerapp identity assign` — UAMI is set in Terraform HCL
- `az containerapp registry set` — ACR pull via managed identity is in
Terraform HCL
- `az containerapp secret set` — all Key Vault secret refs are in
Terraform HCL
- `--replace-env-vars` on `az containerapp update` — env vars are in
Terraform HCL

### Changed
- `deploy-development`: now just loads artifact → pushes to dev ACR →
`az containerapp update --image :sha`
- `deploy-production`: replaced artifact download + push with **`az acr
import`** (server-side copy from dev ACR to prod ACR — faster, no large
artifact download)
- Build job: removed prod registry tags (image only goes to dev ACR at
build time; prod gets it via import)

### Added (production hardening)
- Deploy by **image digest** (`@sha256:...`) instead of mutable tag —
immutable reference
- **Post-deploy image verification** — reads deployed image from
Container App, asserts it matches expected digest; catches silent
rollback if Terraform recreates the resource
- **Smoke test** — `curl --fail /health` on the Container App FQDN
- **Git deploy tag** — `deployed/prod/<sha>` pushed to repo as a durable
audit record

## Prerequisites (confirm before merging)

Verify Terraform HCL for the web Container App has all of:
- [ ] `identity { user_assigned_identity_ids = [...] }` — UAMI attached
- [ ] `registry { ... identity = uami_id }` — ACR pull via managed
identity
- [ ] All 14 secrets as Key Vault refs under `secret {
key_vault_secret_uri }`
- [ ] All env vars under `template.container.env`
- [ ] OIDC identity has `AcrPush` (or `AcrImporter`) on **prod ACR** for
`az acr import`

## RBAC note

The OIDC identity for this repo needs `AcrPush` on the prod ACR in
addition to the dev ACR — `az acr import` writes to prod. Terraform
should own this role assignment.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: BenjaminMichaelis

.github/workflows/Build-Test-And-Deploy.yml

@@ -63,7 +63,7 @@ jobs:
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v4
 
-# Build but no push with a PR
+      # Build but no push with a PR
       - name: Docker build (no push)
         if: github.event_name == 'pull_request' || github.event_name == 'merge_group'
         uses: docker/build-push-action@v7
@@ -74,18 +74,20 @@ jobs:
           context: .
           build-args: ACCESS_TO_NUGET_FEED=false
 
+      # Only build for dev registry — prod gets the image via az acr import in deploy-production
       - name: Build Container Image
         if: github.event_name != 'pull_request_target' && github.event_name != 'pull_request'
         uses: docker/build-push-action@v7
         with:
-          tags: ${{ vars.DEVCONTAINER_REGISTRY }}/essentialcsharpweb:${{ github.sha }},${{ vars.DEVCONTAINER_REGISTRY }}/essentialcsharpweb:latest,${{ vars.PRODCONTAINER_REGISTRY }}/essentialcsharpweb:${{ github.sha }},${{ vars.PRODCONTAINER_REGISTRY }}/essentialcsharpweb:latest
+          tags: ${{ vars.DEVCONTAINER_REGISTRY }}/essentialcsharpweb:${{ github.sha }},${{ vars.DEVCONTAINER_REGISTRY }}/essentialcsharpweb:latest
           file: ./EssentialCSharp.Web/Dockerfile
           context: .
           secrets: |
             "nuget_pat=${{ secrets.AZURE_DEVOPS_PAT }}"
           outputs: type=docker,dest=${{ github.workspace }}/essentialcsharpwebimage.tar
           cache-from: type=gha
           cache-to: type=gha,mode=max
+
       - name: Upload artifact
         uses: actions/upload-artifact@v7
         with:
@@ -101,6 +103,9 @@ jobs:
       cancel-in-progress: false
     environment:
       name: "Development"
+    permissions:
+      id-token: write
+      contents: read
 
     steps:
       - name: Azure Login
@@ -126,55 +131,20 @@ jobs:
           REGISTRY="${{ vars.DEVCONTAINER_REGISTRY }}"
           az acr login --name "${REGISTRY%.azurecr.io}"
 
-      - name: Push Image to Container Registry
+      - name: Push Image to Dev Container Registry
         run: docker push --all-tags ${{ vars.DEVCONTAINER_REGISTRY }}/essentialcsharpweb
 
-      - name: Configure Container App Identity and Registry
-        uses: azure/CLI@v3
+      - name: Deploy to Container App
         env:
           CONTAINER_APP_NAME: ${{ vars.CONTAINER_APP_NAME }}
           RESOURCEGROUP: ${{ vars.RESOURCEGROUP }}
-          CONTAINER_REGISTRY: ${{ vars.DEVCONTAINER_REGISTRY }}
-        with:
-          inlineScript: |
-            # Container app must already exist; use az containerapp up manually to bootstrap if needed
-            az extension add --name containerapp --upgrade
-            az containerapp identity assign --name $CONTAINER_APP_NAME --resource-group $RESOURCEGROUP --user-assigned ${{ secrets.WEB_UAMI_RESOURCE_ID }}
-            az containerapp registry set --name $CONTAINER_APP_NAME --resource-group $RESOURCEGROUP --server $CONTAINER_REGISTRY --identity ${{ secrets.WEB_UAMI_RESOURCE_ID }}
+        run: |
+          az extension add --name containerapp --upgrade --only-show-errors
+          az containerapp update \
+            --name "$CONTAINER_APP_NAME" \
+            --resource-group "$RESOURCEGROUP" \
+            --image "${{ vars.DEVCONTAINER_REGISTRY }}/essentialcsharpweb:${{ github.sha }}"
 
-      - name: Assign Managed Identity to Container App and Set Secrets and Environment Variables
-        uses: azure/CLI@v3
-        env:
-          CONTAINER_APP_NAME: ${{ vars.CONTAINER_APP_NAME }}
-          RESOURCEGROUP: ${{ vars.RESOURCEGROUP }}
-          CONTAINER_REGISTRY: ${{ vars.DEVCONTAINER_REGISTRY }}
-          KEYVAULTURI: ${{ secrets.ESSENTIALCSHARP_KEYVAULT_URI }}
-          MANAGEDIDENTITYID: ${{ secrets.WEB_UAMI_RESOURCE_ID }}
-          AZURECLIENTID: ${{ secrets.WEB_UAMI_CLIENT_ID }}
-          TRYDOTNET_ORIGIN: ${{ vars.TRYDOTNET_ORIGIN }}
-        with:
-          inlineScript: |
-            az containerapp secret set -n $CONTAINER_APP_NAME -g $RESOURCEGROUP --secrets github-clientid=keyvaultref:$KEYVAULTURI/secrets/authentication-github-clientid,identityref:$MANAGEDIDENTITYID  \
-              github-clientsecret=keyvaultref:$KEYVAULTURI/secrets/authentication-github-clientsecret,identityref:$MANAGEDIDENTITYID msft-clientid=keyvaultref:$KEYVAULTURI/secrets/authentication-microsoft-clientid,identityref:$MANAGEDIDENTITYID \
-              msft-clientsecret=keyvaultref:$KEYVAULTURI/secrets/authentication-microsoft-clientsecret,identityref:$MANAGEDIDENTITYID emailsender-apikey=keyvaultref:$KEYVAULTURI/secrets/authmessagesender-apikey,identityref:$MANAGEDIDENTITYID \
-              emailsender-secret=keyvaultref:$KEYVAULTURI/secrets/authmessagesender-secretkey,identityref:$MANAGEDIDENTITYID emailsender-name=keyvaultref:$KEYVAULTURI/secrets/authmessagesender-sendfromname,identityref:$MANAGEDIDENTITYID \
-              emailsender-email=keyvaultref:$KEYVAULTURI/secrets/authmessagesender-sendfromemail,identityref:$MANAGEDIDENTITYID connectionstring=keyvaultref:$KEYVAULTURI/secrets/connectionstrings-essentialcsharpwebcontextconnection,identityref:$MANAGEDIDENTITYID \
-              captcha-sitekey=keyvaultref:$KEYVAULTURI/secrets/captcha-sitekey,identityref:$MANAGEDIDENTITYID captcha-secretkey=keyvaultref:$KEYVAULTURI/secrets/captcha-secretkey,identityref:$MANAGEDIDENTITYID \
-              appinsights-connectionstring=keyvaultref:$KEYVAULTURI/secrets/applicationinsights-connectionstring,identityref:$MANAGEDIDENTITYID \
-              ai-endpoint=keyvaultref:$KEYVAULTURI/secrets/AIOptions--Endpoint,identityref:$MANAGEDIDENTITYID \
-              ai-vectordeployment=keyvaultref:$KEYVAULTURI/secrets/AIOptions--VectorGenerationDeploymentName,identityref:$MANAGEDIDENTITYID ai-chatdeployment=keyvaultref:$KEYVAULTURI/secrets/AIOptions--ChatDeploymentName,identityref:$MANAGEDIDENTITYID \
-              ai-systemprompt=keyvaultref:$KEYVAULTURI/secrets/AIOptions--SystemPrompt,identityref:$MANAGEDIDENTITYID \
-              postgres-vectorstore-connectionstring=keyvaultref:$KEYVAULTURI/secrets/ConnectionStrings--PostgresVectorStore,identityref:$MANAGEDIDENTITYID
-            az containerapp update --name $CONTAINER_APP_NAME --resource-group $RESOURCEGROUP \
-              --image $CONTAINER_REGISTRY/essentialcsharpweb:${{ github.sha }} \
-              --replace-env-vars Authentication__github__clientId=secretref:github-clientid Authentication__github__clientSecret=secretref:github-clientsecret \
-              Authentication__microsoft__clientId=secretref:msft-clientid Authentication__microsoft__clientSecret=secretref:msft-clientsecret AuthMessageSender__ApiKey=secretref:emailsender-apikey AuthMessageSender__SecretKey=secretref:emailsender-secret \
-              AuthMessageSender__SendFromName=secretref:emailsender-name AuthMessageSender__SendFromEmail=secretref:emailsender-email ConnectionStrings__EssentialCSharpWebContextConnection=secretref:connectionstring ASPNETCORE_ENVIRONMENT=Staging \
-              AZURE_CLIENT_ID=$AZURECLIENTID HCaptcha__SiteKey=secretref:captcha-sitekey HCaptcha__SecretKey=secretref:captcha-secretkey APPLICATIONINSIGHTS_CONNECTION_STRING=secretref:appinsights-connectionstring \
-              AIOptions__Endpoint=secretref:ai-endpoint AIOptions__VectorGenerationDeploymentName=secretref:ai-vectordeployment AIOptions__ChatDeploymentName=secretref:ai-chatdeployment \
-              AIOptions__SystemPrompt=secretref:ai-systemprompt ConnectionStrings__PostgresVectorStore=secretref:postgres-vectorstore-connectionstring \
-              TryDotNet__Origin=$TRYDOTNET_ORIGIN DataProtection__AzureKeyVaultKeyUri=$KEYVAULTURI/keys/dataprotection \
-              HCaptcha__ExpectedHostname=essentialcsharp.com
       - name: Logout of Azure CLI
         if: always()
         uses: azure/CLI@v3
@@ -193,80 +163,92 @@ jobs:
       cancel-in-progress: false
     environment:
       name: "Production"
+    permissions:
+      id-token: write
+      contents: write  # needed for git deploy tag
 
     steps:
+      - uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+
       - name: Azure Login
         uses: azure/login@v3
         with:
           client-id: ${{ secrets.AZURE_CLIENT_ID }}
           tenant-id: ${{ secrets.AZURE_TENANT_ID }}
           subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
 
-      - name: Download artifact
-        uses: actions/download-artifact@v8
-        with:
-          name: essentialcsharpwebimage
-          path: ${{ github.workspace }}
-
-      - name: Load image
+      # Server-side copy from dev ACR to prod ACR — no artifact download needed.
+      # PREREQUISITE: prod OIDC identity must have AcrPull on the dev ACR (Terraform RBAC).
+      - name: Import image from dev ACR to prod ACR
+        id: import
         run: |
-          docker load --input ${{ github.workspace }}/essentialcsharpwebimage.tar
-          docker image ls -a
+          DEV_ACR="${{ vars.DEVCONTAINER_REGISTRY }}"
+          PROD_ACR="${{ vars.PRODCONTAINER_REGISTRY }}"
+          az acr import \
+            --name "${PROD_ACR%.azurecr.io}" \
+            --source "${DEV_ACR}/essentialcsharpweb:${{ github.sha }}" \
+            --registry "${DEV_ACR%.azurecr.io}" \
+            --image "essentialcsharpweb:${{ github.sha }}" \
+            --image "essentialcsharpweb:latest" \
+            --force
+          DIGEST=$(az acr repository show \
+            --name "${PROD_ACR%.azurecr.io}" \
+            --image "essentialcsharpweb:${{ github.sha }}" \
+            --query "digest" -o tsv)
+          if [ -z "$DIGEST" ]; then
+            echo "::error::Failed to capture image digest from prod ACR after import"
+            exit 1
+          fi
+          echo "digest=$DIGEST" >> $GITHUB_OUTPUT
 
-      - name: Log in to container registry
+      - name: Deploy to Container App
+        env:
+          CONTAINER_APP_NAME: ${{ vars.CONTAINER_APP_NAME }}
+          RESOURCEGROUP: ${{ vars.RESOURCEGROUP }}
         run: |
-          REGISTRY="${{ vars.PRODCONTAINER_REGISTRY }}"
-          az acr login --name "${REGISTRY%.azurecr.io}"
+          az extension add --name containerapp --upgrade --only-show-errors
+          az containerapp update \
+            --name "$CONTAINER_APP_NAME" \
+            --resource-group "$RESOURCEGROUP" \
+            --image "${{ vars.PRODCONTAINER_REGISTRY }}/essentialcsharpweb@${{ steps.import.outputs.digest }}"
 
-      - name: Push Image to Container Registry
-        run: docker push --all-tags ${{ vars.PRODCONTAINER_REGISTRY }}/essentialcsharpweb
-
-      - name: Configure Container App Identity and Registry
-        uses: azure/CLI@v3
+      - name: Verify deployed image
         env:
           CONTAINER_APP_NAME: ${{ vars.CONTAINER_APP_NAME }}
           RESOURCEGROUP: ${{ vars.RESOURCEGROUP }}
-          CONTAINER_REGISTRY: ${{ vars.PRODCONTAINER_REGISTRY }}
-        with:
-          inlineScript: |
-            # Container app must already exist; use az containerapp up manually to bootstrap if needed
-            az extension add --name containerapp --upgrade
-            az containerapp identity assign --name $CONTAINER_APP_NAME --resource-group $RESOURCEGROUP --user-assigned ${{ secrets.WEB_UAMI_RESOURCE_ID }}
-            az containerapp registry set --name $CONTAINER_APP_NAME --resource-group $RESOURCEGROUP --server $CONTAINER_REGISTRY --identity ${{ secrets.WEB_UAMI_RESOURCE_ID }}
+        run: |
+          DEPLOYED=$(az containerapp show \
+            --name "$CONTAINER_APP_NAME" \
+            --resource-group "$RESOURCEGROUP" \
+            --query "properties.template.containers[0].image" -o tsv)
+          EXPECTED="${{ vars.PRODCONTAINER_REGISTRY }}/essentialcsharpweb@${{ steps.import.outputs.digest }}"
+          if [ "$DEPLOYED" != "$EXPECTED" ]; then
+            echo "::error::Image mismatch! Expected $EXPECTED but found $DEPLOYED"
+            exit 1
+          fi
+          echo "Deployed image verified: $DEPLOYED"
 
-      - name: Assign Managed Identity to Container App and Set Secrets and Environment Variables
-        uses: azure/CLI@v3
+      - name: Smoke test
         env:
           CONTAINER_APP_NAME: ${{ vars.CONTAINER_APP_NAME }}
           RESOURCEGROUP: ${{ vars.RESOURCEGROUP }}
-          CONTAINER_REGISTRY: ${{ vars.PRODCONTAINER_REGISTRY }}
-          KEYVAULTURI: ${{ secrets.ESSENTIALCSHARP_KEYVAULT_URI }}
-          MANAGEDIDENTITYID: ${{ secrets.WEB_UAMI_RESOURCE_ID }}
-          AZURECLIENTID: ${{ secrets.WEB_UAMI_CLIENT_ID }}
-          TRYDOTNET_ORIGIN: ${{ vars.PROD_TRYDOTNET_ORIGIN }}
-        with:
-          inlineScript: |
-            az containerapp secret set -n $CONTAINER_APP_NAME -g $RESOURCEGROUP --secrets github-clientid=keyvaultref:$KEYVAULTURI/secrets/authentication-github-clientid,identityref:$MANAGEDIDENTITYID  \
-              github-clientsecret=keyvaultref:$KEYVAULTURI/secrets/authentication-github-clientsecret,identityref:$MANAGEDIDENTITYID msft-clientid=keyvaultref:$KEYVAULTURI/secrets/authentication-microsoft-clientid,identityref:$MANAGEDIDENTITYID \
-              msft-clientsecret=keyvaultref:$KEYVAULTURI/secrets/authentication-microsoft-clientsecret,identityref:$MANAGEDIDENTITYID emailsender-apikey=keyvaultref:$KEYVAULTURI/secrets/authmessagesender-apikey,identityref:$MANAGEDIDENTITYID \
-              emailsender-secret=keyvaultref:$KEYVAULTURI/secrets/authmessagesender-secretkey,identityref:$MANAGEDIDENTITYID emailsender-name=keyvaultref:$KEYVAULTURI/secrets/authmessagesender-sendfromname,identityref:$MANAGEDIDENTITYID \
-              emailsender-email=keyvaultref:$KEYVAULTURI/secrets/authmessagesender-sendfromemail,identityref:$MANAGEDIDENTITYID connectionstring=keyvaultref:$KEYVAULTURI/secrets/connectionstrings-essentialcsharpwebcontextconnection,identityref:$MANAGEDIDENTITYID \
-              captcha-sitekey=keyvaultref:$KEYVAULTURI/secrets/captcha-sitekey,identityref:$MANAGEDIDENTITYID captcha-secretkey=keyvaultref:$KEYVAULTURI/secrets/captcha-secretkey,identityref:$MANAGEDIDENTITYID \
-              appinsights-connectionstring=keyvaultref:$KEYVAULTURI/secrets/applicationinsights-connectionstring,identityref:$MANAGEDIDENTITYID \
-              ai-endpoint=keyvaultref:$KEYVAULTURI/secrets/AIOptions--Endpoint,identityref:$MANAGEDIDENTITYID \
-              ai-vectordeployment=keyvaultref:$KEYVAULTURI/secrets/AIOptions--VectorGenerationDeploymentName,identityref:$MANAGEDIDENTITYID ai-chatdeployment=keyvaultref:$KEYVAULTURI/secrets/AIOptions--ChatDeploymentName,identityref:$MANAGEDIDENTITYID \
-              ai-systemprompt=keyvaultref:$KEYVAULTURI/secrets/AIOptions--SystemPrompt,identityref:$MANAGEDIDENTITYID \
-              postgres-vectorstore-connectionstring=keyvaultref:$KEYVAULTURI/secrets/ConnectionStrings--PostgresVectorStore,identityref:$MANAGEDIDENTITYID
-            az containerapp update --name $CONTAINER_APP_NAME --resource-group $RESOURCEGROUP \
-              --image $CONTAINER_REGISTRY/essentialcsharpweb:${{ github.sha }} \
-              --replace-env-vars Authentication__github__clientId=secretref:github-clientid Authentication__github__clientSecret=secretref:github-clientsecret \
-              Authentication__microsoft__clientId=secretref:msft-clientid Authentication__microsoft__clientSecret=secretref:msft-clientsecret AuthMessageSender__ApiKey=secretref:emailsender-apikey AuthMessageSender__SecretKey=secretref:emailsender-secret \
-              AuthMessageSender__SendFromName=secretref:emailsender-name AuthMessageSender__SendFromEmail=secretref:emailsender-email ConnectionStrings__EssentialCSharpWebContextConnection=secretref:connectionstring ASPNETCORE_ENVIRONMENT=Production \
-              AZURE_CLIENT_ID=$AZURECLIENTID HCaptcha__SiteKey=secretref:captcha-sitekey HCaptcha__SecretKey=secretref:captcha-secretkey APPLICATIONINSIGHTS_CONNECTION_STRING=secretref:appinsights-connectionstring \
-              AIOptions__Endpoint=secretref:ai-endpoint AIOptions__VectorGenerationDeploymentName=secretref:ai-vectordeployment AIOptions__ChatDeploymentName=secretref:ai-chatdeployment \
-              AIOptions__SystemPrompt=secretref:ai-systemprompt ConnectionStrings__PostgresVectorStore=secretref:postgres-vectorstore-connectionstring \
-              TryDotNet__Origin=$TRYDOTNET_ORIGIN DataProtection__AzureKeyVaultKeyUri=$KEYVAULTURI/keys/dataprotection \
-              HCaptcha__ExpectedHostname=essentialcsharp.com
+        run: |
+          FQDN=$(az containerapp show \
+            --name "$CONTAINER_APP_NAME" \
+            --resource-group "$RESOURCEGROUP" \
+            --query "properties.configuration.ingress.fqdn" -o tsv)
+          # --retry-all-errors ensures HTTP 5xx (cold-start 503s) also trigger retries
+          curl --fail --retry 10 --retry-delay 15 --retry-all-errors --max-time 30 "https://$FQDN/health"
+
+      - name: Tag commit as deployed
+        run: |
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+          git config user.name "github-actions[bot]"
+          # -f allows re-tagging the same SHA on workflow re-runs
+          git tag -f "deployed/prod/${{ github.sha }}"
+          git push origin "deployed/prod/${{ github.sha }}" --force
 
       - name: Logout of Azure CLI
         if: always()