Document Deployment-compatible migrations (#2987)

Author: r4victor

contributing/MIGRATIONS.md

@@ -11,32 +11,39 @@ alembic revision -m "<some message>" --autogenerate
 
 Then adjust the generated migration if needed.
 
-## PostgreSQL enums
+## Deployment-compatible migrations
 
-If you modify any enums used in SQLAlchemy models, you will need to set up PostgreSQL
-in order to generate a PostgreSQL-specific enum migration.
+The `dstack` server claims to support multi-replica setups with zero-downtime deployments.
+This means DB migrations should not make changes that break old replicas.
+Incompatible changes should be introduced in multiple stages (releases), following
+the [expand and contract pattern](https://www.prisma.io/dataguide/types/relational/expand-and-contract-pattern).
 
-1. Run PostgreSQL.
+**Note**: If it's impossible to make the migration compatible with older versions, the PR should say so explicitly, so that the change is planned and released with the migration notice.
 
-   ```shell
-   docker run --rm -p 5432:5432 -e POSTGRES_PASSWORD=password postgres
-   ```
+Below are some common changes and how to make them.
 
-1. Create a database for `dstack`.
+### Removing a column
 
-   ```shell
-   psql -h localhost -U postgres --command "CREATE DATABASE dstack"
-   ```
+1. First release:
+   * Stop reading the column. In SQLAlchemy this can be done by setting `deferred=True` on a model field.
+2. Second release:
+   * Drop the column.
 
-1. Run `dstack server` once to create the previous database schema.
+### Changing a column
 
-   ```shell
-   DSTACK_DATABASE_URL=postgresql+asyncpg://postgres:password@localhost:5432/dstack dstack server
-   ```
+These steps apply to **renaming a column** or **changing the type of a column**
 
-1. Generate the migration.
+1. First release:
+   * Introduce a new column with the new type.
+   * Write to both the new and the old column.
+2. Second release:
+   * Migrate data from the old column to the new column.
+   * Start reading the new column.
+   * Stop reading the old column.
+   * Stop writing to the old column.
+3. Third release:
+   * Drop the old column.
 
-   ```shell
-   cd src/dstack/_internal/server/
-   DSTACK_DATABASE_URL=postgresql+asyncpg://postgres:password@localhost:5432/dstack alembic revision -m "<some message>" --autogenerate
-   ```
+### Altering multiple tables
+
+Altering a table requires Postgres to [take an ACCESS EXCLUSIVE lock](https://www.postgresql.org/docs/current/sql-altertable.html). (This applies not only to statements that rewrite the tables but also to statements that modify tables metadata.) Altering multiple tables can cause deadlocks due to conflict with read operations since the `dstack` server does not define an order for read operations. Altering multiple tables should be done in separate transactions/migrations.