Revert "feat(deployment): use sync-waves for triggering ingest cronjobs" (#6572)

Reverts #6375

https://loculus.slack.com/archives/C05G172HL6L/p1780303552350439

This causes various issues:

silo takes too long to start on PPX so the sync constantly fails
with ttl the ingest cronjobs die after 10min (if sync completes and
triggers their creation) and are constantly recreated in a loop


🚀 Preview: Add `preview` label to enable

Co-authored-by: Theo Sanderson <theo@sndrsn.co.uk>

Author: anna-parker

ingest/README.md

@@ -98,8 +98,8 @@ Indirect requirements:
 
 The pipeline runs in different kubernetes resources:
 
-- An "approve" deployment (uses the `autoapprove.py` script) that continuously polls the Loculus backend for sequences in status `WAITING_FOR_APPROVAL` and approves them. This has one replica with kubernetes resource type apps/v1/Deployment.
-- An "ingest" cronjob batch/v1/CronJob that runs the ingest pipeline regularly. Cronjobs are scheduled to start ingest for a different organism every 30min to reduce load. To improve development experience ArgoCD spawns ingest pods using sync waves (10th wave).
+- An "approve" deployment (uses the `autoapprove.py` script) that continuously polls the Loculus backend for sequences in status `WAITING_FOR_APPROVAL` and approves them, one replica with kubernetes resource type apps/v1/Deployment.
+- An "ingest" cronjob batch/v1/CronJob that runs the ingest pipeline regularly. Cronjobs are scheduled to start ingest for a different organism every 30min to reduce load. To improve development experience ArgoCD spawns ingest pods using a PostSync-hook trigger (this waits for all other pods to be healthy after an ArgoCD sync - reducing ingest failures due to the backend being unreachable)
 
 In production the ingest pipeline runs in a docker container that takes a config file as input.
 

kubernetes/loculus/templates/ingest.yaml

@@ -115,15 +115,22 @@ spec:
             "args" (list "snakemake" "results/submitted" "results/revised" "--all-temp")
           ) | nindent 8 }}
 {{/*
-sync-wave: 10 for the trigger Job, so it runs at the end of an argocd sync.
+PostSync-hook trigger: kicks off a one-off ingest Job from the CronJob's
+template after argocd reports the sync as healthy. PostSync waits for
+all resources (e.g. backend, ena-submission) to be Healthy before firing,
+so the spawned ingest pod doesn't start hammering services that aren't
+ready yet. The trigger itself exits in seconds; the spawned ingest Job
+runs untracked by argocd. Concurrency with a scheduled CronJob run is
+handled by the shared lock init container in the spawned pod.
 */}}
 ---
 apiVersion: batch/v1
 kind: Job
 metadata:
   name: loculus-ingest-trigger-{{ $key }}
   annotations:
-    argocd.argoproj.io/sync-wave: "10"
+    argocd.argoproj.io/hook: PostSync
+    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded
 spec:
   activeDeadlineSeconds: 120
   backoffLimit: 0