Remove the -parallelism=1 requirement (pre-allocate KMS nonces)

[h4x3rotab]

Problem

All deploys are stuck on terraform apply -parallelism=1, one cluster at a time. Anything concurrent — -parallelism > 1 within one apply, or two applies at once — fails during app creation with:

[ERR-02-009] Failed to obtain app ID from centralized KMS:
duplicate key value violates unique constraint "ix_dstack_app_nonces_address"

Root cause

The KMS allocates the per-team-wallet nonce (obtain_app_id) with an unlocked read-modify-write, so concurrent creates derive the same app address and collide. The proper server-side fix is tracked upstream in Phala-Network/phala-cloud-monorepo#1544 (route obtain_app_id through the existing .with_for_update() lock).

Fix we can ship here (no wait on the backend)

The provider already exposes custom_app_id + nonce on phala_app (verified on 0.3.0-beta.4), and the KMS has a predict endpoint GET /kms/phala/next_app_id?counts=N. So we can pre-assign distinct nonces and deploy in parallel:

1. Pre-flight: one serialized call to next_app_id for the team to get N distinct {app_id, nonce} pairs.
2. Feed one pair into each phala_app (coordinator + each workload group) via custom_app_id + nonce.
3. Drop the -parallelism=1 requirement from the docs / README.

Each create then takes the validate_manual_nonce path with a pre-assigned distinct address, so the unique constraint is never contended.

Constraints to handle

• next_app_id caps at 20 per batch; batch for larger fleets.
• Nonce must stay within ~20 of the team's current max; each nonce is single-use.
• Wiring the predict call into Terraform needs an http/external data source or a small pre-step script.

Acceptance criteria

• clusters/patroni-demo deploys cleanly with -parallelism > 1, and two clusters apply concurrently without the nonce collision.
• README / docs no longer instruct -parallelism=1.

Notes

• This is a client-side mitigation; once #1544 lands, the manual pre-allocation becomes optional (the auto path is safe again).
• Verified current behavior on provider 0.3.0-beta.4, real cloud (h4xusers workspace).