Skip to content

README.md

Latest commit

 

History

History
74 lines (48 loc) · 2.98 KB

README.md

File metadata and controls

74 lines (48 loc) · 2.98 KB
Raw

AI / ML Toolkit Blueprints

The directories below contain solutions for provisioning Slurm clusters using the latest VM families from Google Cloud

Further documentation for A3 Ultra, A4 and A4X solutions are available at Create an AI-optimized Slurm cluster.

Selective Deployment and Destruction using --only and --skip flags

You can control which groups in a blueprint are deployed or destroyed using the --only and --skip flags with the gcluster deploy and gcluster destroy commands. This is useful for saving time by not acting on components unnecessarily or for more granular control over resources.

A blueprint is divided into logical groups (the exact group names depend on the blueprint YAML). Check the blueprint file (e.g., ...-blueprint.yaml) for the exact group names for that blueprint.

--only <group1>,<group2>,...

Use the --only flag to have the command act on only the specified, comma-separated groups. Other groups will be untouched.

Examples:

  • Deploy only the base group:
./gcluster deploy -d a3high-slurm-deployment.yaml examples/machine-learning/a3-highgpu-8g/a3high-slurm-blueprint.yaml --only base
  • Destroy only the image group:
./gcluster destroy deployment-name --only image
  • Deploy only the base and cluster groups:
./gcluster deploy -d a3high-slurm-deployment.yaml examples/machine-learning/a3-highgpu-8g/a3high-slurm-blueprint.yaml --only base,cluster

--skip <group1>,<group2>,...

Use the --skip flag to have the command act on all groups except those specified in the comma-separated list.

Examples:

  • Deploy everything except the image group:
./gcluster deploy -d a4high-slurm-deployment.yaml examples/machine-learning/a4-highgpu-8g/a4high-slurm-blueprint.yaml --skip image
  • Destroy everything except the base group:
./gcluster destroy deployment-name --skip base

Use cases:

  • Faster iteration: When developing, only deploy the group you are modifying (e.g., --only base).
  • Partial teardown: Selectively destroy parts of a deployment without affecting others (e.g., --only image to remove image but keep networking and other things).
  • Avoiding unchanged parts: Use --skip to not redeploy parts you know are stable or should be preserved (e.g., --skip cluster,image).
  • Retry failed operations: If a deploy or destroy fails on a specific group, rerun the command targeting just that group using --only.

Notes:

  • The exact group names vary by blueprint. Always consult the blueprint YAML used in the command for the correct names.
  • The initial deployment may have additional constraints (for example, images may always need to be built the first time). Check the blueprint-specific README for blueprint-specific restrictions.