Skip to content

Refresh GB200 and GB300 GMC deployment guidance #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Prgolan28 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Refresh GB200 and GB300 GMC deployment guidance #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 13 additions & 13 deletions nvidia/GB200/README-GB200.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -994,21 +994,21 @@ configuration via OCI web console than use JSON input file:

oci compute-management instance-configuration create --compartment-id ocid1.compartment... --instance-details XXX_JSON_XXX 

**Note:** It is recommended that you create the GMC **with all of the**
**available hosts** in the fabric. This is necessary to properly size the
NVLink partition (support for multi-cast) since once the partition is
created it will not be updated when adding new nodes. The only way to
update it once it has be created is to delete the GMC and start over or
idle the workload on the rack and then have OCI update the partition in
the background.

Create the GMC.  This will bring up \<size\> instances.  You can use the
"available-host-count" from the "compute-gpu-memory-fabric list"
step.  Use the target compartment OCID: 
When creating a GMC, set `targetSize` at creation time to the maximum
number of instances you expect to use in that GMC. This sizes the NVLink
multicast limits for the partition and allows the GMC to grow toward
`targetSize` as hosts become available or return from repair. For GB200,
GB200-v2, and GB200-v3, use `targetSize` 18 for each GMC unless your
deployment plan says otherwise. You do not need an explicit `--size` when
`targetSize` is set.

Create the GMC. Use the target compartment OCID:

oci compute compute-gpu-memory-cluster create --availability-domain XXX \
--compartment-id ocid1.compartment... --compute-cluster-id ocid1.computecluster... \
--instance-configuration-id XXX --gpu-memory-fabric-id ocid1.computegpumemoryfabric... --size XX    
--instance-configuration-id ocid1.instanceconfiguration... \
--gpu-memory-fabric-id ocid1.computegpumemoryfabric... \
--gpu-memory-cluster-scale-config '{"targetSize":18,"isUpsizeEnabled":true,"isDownsizeEnabled":false}'

To increase the number of instances in an existing GMC set the size
parameter to the total size you want (it is not an increment to the
Expand All @@ -1021,7 +1021,7 @@ termination is used.

To terminate all instances on a GMC and delete the GMC itself:

oci compute compute-gpu-memory-cluster delete --compute-gpu-memory-cluster-id ocid1.computecluster.oc1....
oci compute compute-gpu-memory-cluster delete --compute-gpu-memory-cluster-id ocid1.computegpumemorycluster.oc1....

#### IMEX

Expand Down
32 changes: 19 additions & 13 deletions nvidia/GB300/README-GB300.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1130,21 +1130,20 @@ configuration via OCI web console than use JSON input file:

oci compute-management instance-configuration create --compartment-id ocid1.compartment... --instance-details XXX_JSON_XXX 

**Note:** It is recommended that you create the GMC **with all of the**
**available hosts** in the fabric. This is necessary to properly size the
NVLink partition (support for multi-cast) since once the partition is
created it will not be updated when adding new nodes. The only way to
update it once it has be created is to delete the GMC and start over or
idle the workload on the rack and then have OCI update the partition in
the background.

Create the GMC.  This will bring up \<size\> instances.  You can use the
"available-host-count" from the "compute-gpu-memory-fabric list"
step.  Use the target compartment OCID: 
When creating a GMC, set `targetSize` at creation time to the maximum
number of instances you expect to use in that GMC. This sizes the NVLink
multicast limits for the partition and allows the GMC to grow toward
`targetSize` as hosts become available or return from repair. For GB300,
use `targetSize` 18 for each GMC unless your deployment plan says
otherwise. You do not need an explicit `--size` when `targetSize` is set.

Create the GMC. Use the target compartment OCID:

oci compute compute-gpu-memory-cluster create --availability-domain XXX \
--compartment-id ocid1.compartment... --compute-cluster-id ocid1.computecluster... \
--instance-configuration-id XXX --gpu-memory-fabric-id ocid1.computegpumemoryfabric... --size XX    
--instance-configuration-id ocid1.instanceconfiguration... \
--gpu-memory-fabric-id ocid1.computegpumemoryfabric... \
--gpu-memory-cluster-scale-config '{"targetSize":18,"isUpsizeEnabled":true,"isDownsizeEnabled":false}'

To increase the number of instances in an existing GMC set the size
parameter to the total size you want (it is not an increment to the
Expand All @@ -1157,7 +1156,14 @@ termination is used.

To terminate all instances on a GMC and delete the GMC itself:

oci compute compute-gpu-memory-cluster delete --compute-gpu-memory-cluster-id ocid1.computecluster.oc1....
oci compute compute-gpu-memory-cluster delete --compute-gpu-memory-cluster-id ocid1.computegpumemorycluster.oc1....

After launch, confirm the host topology:

curl -sH "Authorization: Bearer Oracle" -L http://169.254.169.254/opc/v2/host \
| jq '.rdmaFabricData.planes, .rdmaTopologyData.customerGpuMemoryFabric'

Single-plane hosts should report `1`; Multiplanar hosts should report `4`.

#### IMEX

Expand Down