feat(deployment): use sync-waves for triggering ingest cronjobs (#6375)

issue described in
https://loculus.slack.com/archives/C05G172HL6L/p1778229491333099

PostSync was getting previews stuck inn sync status for multiple minutes
and leading to issues removing previews, switching to sync with a later
syncWave partially solves this (sync is still waiting for hooks and
taking slightly longer than before) but does not lead to too many errors
in ingest and can be deleted immedietly.

🚀 Preview: Add `preview` label to enable

Author: anna-parker

ingest/README.md

@@ -98,8 +98,8 @@ Indirect requirements:
 
 The pipeline runs in different kubernetes resources:
 
-- An "approve" deployment (uses the `autoapprove.py` script) that continuously polls the Loculus backend for sequences in status `WAITING_FOR_APPROVAL` and approves them, one replica with kubernetes resource type apps/v1/Deployment.
-- An "ingest" cronjob batch/v1/CronJob that runs the ingest pipeline regularly. Cronjobs are scheduled to start ingest for a different organism every 30min to reduce load. To improve development experience ArgoCD spawns ingest pods using a PostSync-hook trigger (this waits for all other pods to be healthy after an ArgoCD sync - reducing ingest failures due to the backend being unreachable)
+- An "approve" deployment (uses the `autoapprove.py` script) that continuously polls the Loculus backend for sequences in status `WAITING_FOR_APPROVAL` and approves them. This has one replica with kubernetes resource type apps/v1/Deployment.
+- An "ingest" cronjob batch/v1/CronJob that runs the ingest pipeline regularly. Cronjobs are scheduled to start ingest for a different organism every 30min to reduce load. To improve development experience ArgoCD spawns ingest pods using sync waves (10th wave).
 
 In production the ingest pipeline runs in a docker container that takes a config file as input.
 

kubernetes/loculus/templates/ingest.yaml

@@ -115,22 +115,15 @@ spec:
             "args" (list "snakemake" "results/submitted" "results/revised" "--all-temp")
           ) | nindent 8 }}
 {{/*
-PostSync-hook trigger: kicks off a one-off ingest Job from the CronJob's
-template after argocd reports the sync as healthy. PostSync waits for
-all resources (e.g. backend, ena-submission) to be Healthy before firing,
-so the spawned ingest pod doesn't start hammering services that aren't
-ready yet. The trigger itself exits in seconds; the spawned ingest Job
-runs untracked by argocd. Concurrency with a scheduled CronJob run is
-handled by the shared lock init container in the spawned pod.
+sync-wave: 10 for the trigger Job, so it runs at the end of an argocd sync.
 */}}
 ---
 apiVersion: batch/v1
 kind: Job
 metadata:
   name: loculus-ingest-trigger-{{ $key }}
   annotations:
-    argocd.argoproj.io/hook: PostSync
-    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded
+    argocd.argoproj.io/sync-wave: "10"
 spec:
   activeDeadlineSeconds: 120
   backoffLimit: 0