Skip to content

Harden CI database migration deploy retries #356

@kevinhermawan

Description

@kevinhermawan

Summary

CI deploys are intermittently failing during prisma migrate deploy against orchestration/mediapulse databases, then succeeding on retry. We should harden the deployment workflow so transient DB/network issues do not require manual reruns while keeping migration execution safe and deterministic.

Scope

  • In scope:
    • Add a dedicated migration stage in CI that runs before deploy jobs.
    • Add bounded retry with backoff for transient migration failures.
    • Ensure migrations are not executed concurrently from multiple deploy jobs.
    • Document expected behavior and failure conditions in workflow comments.
  • Out of scope:
    • Changing Prisma schema or migration SQL content.
    • Reworking application runtime DB connection logic.

Acceptance criteria

  • CI has a single migration job that runs before app/agent deploy jobs.
  • Migration step retries transient failures (e.g. network hiccups) with bounded attempts and delay.
  • Deploy workflows no longer trigger duplicate/concurrent migration execution paths.
  • Workflow fails clearly after retry budget is exhausted.
  • At least one successful pipeline run demonstrates migration + deploy flow without manual rerun.

Dependencies

  • Depends on: none
  • Blocks: none

Notes

Observed symptom: logs show prisma migrate deploy failing intermittently in CI and succeeding when the workflow is retried manually.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions