Add Crusoe Cloud backend (#3602)

* Add Crusoe Cloud backend

Add a VM-based Crusoe Cloud backend supporting single-node and
multi-node (cluster) provisioning with InfiniBand.

Key features:
- gpuhunt online provider for offers with project quota filtering
- HMAC-SHA256 authenticated REST API client
- Image selection based on GPU type (SXM/PCIe/ROCm/CPU)
- Storage: persistent data disk for types without ephemeral NVMe;
  auto-detects and RAID-0s NVMe for types with ephemeral storage;
  moves containerd storage so containers get the full disk space
- Cluster support via IB partitions
- Two-phase termination with data disk cleanup

Tested end-to-end:
- L40S: fleet, dev env, GPU, configurable disk (200GB), clean termination
- A100-PCIe: fleet, dev env, GPU, NVMe auto-mount (880GB), clean termination
- A100-SXM-IB cluster: IB partition created, 1 node provisioned with IB
  and 8x NVMe RAID-0 (7TB); 2nd node failed on capacity (out_of_stock)
- Offers: quota enforcement, disk sizes correct per instance type

Not tested (no capacity/quota):
- H100-SXM-IB, MI300X-IB, MI355X-RoCE (no hardware available)
- CPU-only instances c1a/s1a (no quota)
- Spot provisioning (disabled in gpuhunt, see TODO)
- Full 2-node cluster with IB connectivity test

TODOs:
- Spot: disabled until Crusoe confirms how to request spot billing
  via the VM create API endpoint
- gpuhunt dependency: currently installed from PR branch; switch to
  pinned version after gpuhunt PR #211 is merged and released

AI Assistance: This implementation was developed with AI assistance.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fetch Crusoe locations dynamically instead of hardcoding

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix VM image selection for SXM instance types

The _get_image function checked gpu_type (e.g. 'A100') for 'SXM', but
gpuhunt normalizes GPU names and strips the SXM qualifier. Check the
instance type name instead (e.g. 'a100-80gb-sxm-ib.8x') which
preserves the '-sxm' indicator.

Without this fix, SXM-IB instances used the PCIe docker image which
lacks IB drivers, HPC-X, and NCCL topology files. Verified with a
2-node A100-SXM-IB NCCL all_reduce test: 193 GB/s bus bandwidth.

Made-with: Cursor

* Switch gpuhunt dependency from PR branch to main

Made-with: Cursor

* Add TODOs to pin gpuhunt and remove allow-direct-references before merging

Made-with: Cursor

* Pin gpuhunt==0.1.17 (matches master)

Made-with: Cursor

---------

Co-authored-by: Cursor <cursoragent@cursor.com>

Author: peterschmidt85

docs/docs/concepts/backends.md

@@ -929,6 +929,34 @@ projects:
     * `sizes` - read
     * `ssh_key` - create, read, update,delete
 
+### Crusoe Cloud
+
+Log into your [Crusoe Cloud](https://console.crusoecloud.com/) console and create an API key
+under your account settings. Note your project ID from the project settings page.
+
+Then, go ahead and configure the backend:
+
+<div editor-title="~/.dstack/server/config.yml">
+
+```yaml
+projects:
+- name: main
+  backends:
+    - type: crusoe
+      project_id: your-project-id
+      creds:
+        type: access_key
+        access_key: your-access-key
+        secret_key: your-secret-key
+      regions:
+        - us-east1-a
+        - us-southcentral1-a
+```
+
+</div>
+
+`regions` is optional. If not specified, all available Crusoe regions are used.
+
 ### Hot Aisle
 
 Log in to the SSH TUI as described in the [Hot Aisle Quick Start](https://hotaisle.xyz/quick-start/).

docs/docs/reference/server/config.yml.md

@@ -335,6 +335,23 @@ to configure [backends](../../concepts/backends.md) and other [server-level sett
         type:
             required: true
 
+##### `projects[n].backends[type=crusoe]` { #crusoe data-toc-label="crusoe" }
+
+#SCHEMA# dstack._internal.core.backends.crusoe.models.CrusoeBackendConfigWithCreds
+    overrides:
+        show_root_heading: false
+        type:
+            required: true
+        item_id_prefix: crusoe-
+
+###### `projects[n].backends[type=crusoe].creds` { #crusoe-creds data-toc-label="creds" }
+
+#SCHEMA# dstack._internal.core.backends.crusoe.models.CrusoeAccessKeyCreds
+    overrides:
+        show_root_heading: false
+        type:
+            required: true
+
 ##### `projects[n].backends[type=hotaisle]` { #hotaisle data-toc-label="hotaisle" }
 
 #SCHEMA# dstack._internal.core.backends.hotaisle.models.HotAisleBackendConfigWithCreds

frontend/src/types/backend.d.ts

@@ -1,6 +1,7 @@
 declare type TBackendType =
     | 'aws'
     | 'azure'
+    | 'crusoe'
     | 'cudo'
     | 'datacrunch'
     | 'dstack'

pyproject.toml

@@ -256,6 +256,9 @@ fluentbit = [
     "elasticsearch>=8.0.0",
     "dstack[server]",
 ]
+crusoe = [
+    "dstack[server]",
+]
 all = [
-    "dstack[gateway,server,aws,azure,gcp,verda,kubernetes,lambda,nebius,oci,fluentbit]",
+    "dstack[gateway,server,aws,azure,gcp,verda,kubernetes,lambda,nebius,oci,crusoe,fluentbit]",
 ]

src/dstack/_internal/core/backends/configurators.py

@@ -35,6 +35,15 @@
 except ImportError:
     pass
 
+try:
+    from dstack._internal.core.backends.crusoe.configurator import (
+        CrusoeConfigurator,
+    )
+
+    _CONFIGURATOR_CLASSES.append(CrusoeConfigurator)
+except ImportError:
+    pass
+
 try:
     from dstack._internal.core.backends.cudo.configurator import (
         CudoConfigurator,

src/dstack/_internal/core/backends/crusoe/backend.py

@@ -0,0 +1,16 @@
+from dstack._internal.core.backends.base.backend import Backend
+from dstack._internal.core.backends.crusoe.compute import CrusoeCompute
+from dstack._internal.core.backends.crusoe.models import CrusoeConfig
+from dstack._internal.core.models.backends.base import BackendType
+
+
+class CrusoeBackend(Backend):
+    TYPE = BackendType.CRUSOE
+    COMPUTE_CLASS = CrusoeCompute
+
+    def __init__(self, config: CrusoeConfig):
+        self.config = config
+        self._compute = CrusoeCompute(self.config)
+
+    def compute(self) -> CrusoeCompute:
+        return self._compute