Merge branch 'master' into issue_2650_multinode_on_shared_instances

Author: r4victor

docs/assets/stylesheets/extra.css

@@ -29,6 +29,10 @@
     margin-left: 32px;
 }
 
+[dir=ltr] .md-typeset :is(.md-typeset__scrollwrap,p,h4,h3,.tabbed-set,ul):not(.admonition-title) :is(.admonition,details, .termy) {
+    margin-left: 0;
+}
+
 @media screen and (max-width: 76.1875em) {
     .md-header {
         background-color: rgb(255, 255, 255);

docs/docs/concepts/fleets.md

@@ -263,26 +263,30 @@ Define a fleet configuration as a YAML file in your project directory. The file
 </div>
 
 ??? info "Requirements" 
-    1.&nbsp;Hosts should be pre-installed with Docker.
+    1.&nbsp;Hosts must be pre-installed with Docker.
 
     === "NVIDIA"
-        2.&nbsp;Hosts with NVIDIA GPUs should also be pre-installed with CUDA 12.1 and
+        2.&nbsp;Hosts with NVIDIA GPUs must also be pre-installed with CUDA 12.1 and
         [NVIDIA Container Toolkit :material-arrow-top-right-thin:{ .external }](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).
 
     === "AMD"
-        2.&nbsp;Hosts with AMD GPUs should also be pre-installed with AMDGPU-DKMS kernel driver (e.g. via
+        2.&nbsp;Hosts with AMD GPUs must also be pre-installed with AMDGPU-DKMS kernel driver (e.g. via
         [native package manager :material-arrow-top-right-thin:{ .external }](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/native-install/index.html)
         or [AMDGPU installer :material-arrow-top-right-thin:{ .external }](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/amdgpu-install.html).)
 
     === "Intel Gaudi"
-        2.&nbsp;Hosts with Intel Gaudi accelerators should be pre-installed with [Gaudi software and drivers](https://docs.habana.ai/en/latest/Installation_Guide/Driver_Installation.html#driver-installation).
-        This should include the drivers, `hl-smi`, and Habana Container Runtime.
+        2.&nbsp;Hosts with Intel Gaudi accelerators must be pre-installed with [Gaudi software and drivers](https://docs.habana.ai/en/latest/Installation_Guide/Driver_Installation.html#driver-installation).
+        This must include the drivers, `hl-smi`, and Habana Container Runtime.
 
     === "Tenstorrent"
-        2.&nbsp;Hosts with Tenstorrent accelerators should be pre-installed with [Tenstorrent software](https://docs.tenstorrent.com/getting-started/README.html#software-installation).
-        This should include the drivers, `tt-smi`, and HugePages.
+        2.&nbsp;Hosts with Tenstorrent accelerators must be pre-installed with [Tenstorrent software](https://docs.tenstorrent.com/getting-started/README.html#software-installation).
+        This must include the drivers, `tt-smi`, and HugePages.
 
-    3.&nbsp;The user specified should have passwordless `sudo` access.
+    3.&nbsp;The user specified must have passwordless `sudo` access.
+
+    4.&nbsp;The SSH server must be running and configured with `AllowTcpForwarding yes` in `/etc/ssh/sshd_config`.
+
+    5.&nbsp;The firewall must allow SSH and should forbid any other connections from external networks. For `placement: cluster` fleets, it should also allow any communication between fleet nodes.
 
 To create or update the fleet, pass the fleet configuration to [`dstack apply`](../reference/cli/dstack/apply.md):
 
@@ -341,10 +345,10 @@ divided into, allowing multiple jobs to use these blocks concurrently.
       hosts:
         - hostname: 3.255.177.51
           blocks: 4
-        - hostaname: 3.255.177.52
+        - hostname: 3.255.177.52
           # As many as possible, according to numbers of GPUs and CPUs
           blocks: auto
-        - hostaname: 3.255.177.53
+        - hostname: 3.255.177.53
           # Do not sclice. This is the default value, may be omitted
           blocks: 1
     ```

docs/docs/concepts/tasks.md

@@ -426,6 +426,34 @@ retry:
 If one job of a multi-node task fails with retry enabled,
 `dstack` will stop all the jobs and resubmit the run.
 
+### Priority
+
+Be default, submitted runs are scheduled in the order they were submitted.
+When compute resources are limited, you may want to prioritize some runs over others.
+This can be done by specifying the [`priority`](../reference/dstack.yml/task.md) property in the run configuration:
+
+<div editor-title=".dstack.yml">
+
+```yaml
+type: task
+name: train
+
+python: "3.10"
+
+# Commands of the task
+commands:
+  - pip install -r fine-tuning/qlora/requirements.txt
+  - python fine-tuning/qlora/train.py
+
+priority: 50
+```
+
+</div>
+
+`dstack` tries to provision runs with higher priority first.
+Note that if a high priority run cannot be scheduled,
+it does not block other runs with lower priority from scheduling.
+
 --8<-- "docs/concepts/snippets/manage-fleets.ext"
 
 --8<-- "docs/concepts/snippets/manage-runs.ext"

docs/docs/installation/index.md

@@ -139,71 +139,71 @@ Configuration is updated at ~/.dstack/config.yml
 
 This configuration is stored in `~/.dstack/config.yml`.
 
-### (Optional) Enable shell autocompletion
+??? info "Shell autocompletion"
 
-`dstack` supports shell autocompletion for `bash` and `zsh`.
+    `dstack` supports shell autocompletion for `bash` and `zsh`.
 
-=== "bash"
+    === "bash"
 
-    First, validate if completion scripts load correctly in your current shell session:
-    
-    <div class="termy">
-    
-    ```shell
-    $ eval "$(dstack completion bash)"
-    ```
+        First, validate if completion scripts load correctly in your current shell session:
+        
+        <div class="termy">
+        
+        ```shell
+        $ eval "$(dstack completion bash)"
+        ```
 
-    </div>
-    
-    If completions work as expected and you would like them to persist across shell sessions, add the completion script to your shell profile using these commands:
-    
-    <div class="termy">
-    
-    ```shell
-    $ mkdir -p ~/.dstack
-    $ dstack completion bash > ~/.dstack/completion.sh
-    $ echo 'source ~/.dstack/completion.sh' >> ~/.bashrc
-    ```
-    
-    </div>
+        </div>
+        
+        If completions work as expected and you would like them to persist across shell sessions, add the completion script to your shell profile using these commands:
+        
+        <div class="termy">
+        
+        ```shell
+        $ mkdir -p ~/.dstack
+        $ dstack completion bash > ~/.dstack/completion.sh
+        $ echo 'source ~/.dstack/completion.sh' >> ~/.bashrc
+        ```
+        
+        </div>
 
-=== "zsh"
-    
-    First, validate if completion scripts load correctly in your current shell session:
-    
-    <div class="termy">
-    
-    ```shell
-    $ eval "$(dstack completion zsh)"
-    ```
+    === "zsh"
+        
+        First, validate if completion scripts load correctly in your current shell session:
+        
+        <div class="termy">
+        
+        ```shell
+        $ eval "$(dstack completion zsh)"
+        ```
 
-    </div>
-    
-    If completions work as expected and you would like them to persist across shell sessions, you can install them via Oh My Zsh using these commands:
-    
-    <div class="termy">
-    
-    ```shell
-    $ mkdir -p ~/.oh-my-zsh/completions
-    $ dstack completion zsh > ~/.oh-my-zsh/completions/_dstack
-    ```
+        </div>
         
-    </div>
+        If completions work as expected and you would like them to persist across shell sessions, you can install them via Oh My Zsh using these commands:
+        
+        <div class="termy">
+        
+        ```shell
+        $ mkdir -p ~/.oh-my-zsh/completions
+        $ dstack completion zsh > ~/.oh-my-zsh/completions/_dstack
+        ```
+            
+        </div>
 
-    And if you don't use Oh My Zsh:
+        And if you don't use Oh My Zsh:
 
-    <div class="termy">
-    
-    ```shell
-    $ mkdir -p ~/.dstack
-    $ dstack completion zsh > ~/.dstack/completion.sh
-    $ echo 'source ~/.dstack/completion.sh' >> ~/.zshrc
-    ```
-    
-    </div>
+        <div class="termy">
+        
+        ```shell
+        $ mkdir -p ~/.dstack
+        $ dstack completion zsh > ~/.dstack/completion.sh
+        $ echo 'source ~/.dstack/completion.sh' >> ~/.zshrc
+        ```
+        
+        </div>
 
-    > If you get an error similar to `2: command not found: compdef`, then add the following line to the beginning of your `~/.zshrc` file:
-    > `autoload -Uz compinit && compinit`.
+        > If you get an error similar to `2: command not found: compdef`, then add the following line to the beginning of your `~/.zshrc` file:
+        > `autoload -Uz compinit && compinit`.
 
 
 !!! info "What's next?"

docs/overrides/main.html

@@ -122,7 +122,7 @@
                     <a href="/examples#inference" class="tx-footer__section-link">Inference</a>
                     <a href="/examples#accelerators" class="tx-footer__section-link">Accelerators</a>
                     <a href="/examples#llms" class="tx-footer__section-link">LLMs</a>
-                    <a href="/examples#misc" class="tx-footer__section-link">Misc</a>
+                    <!-- <a href="/examples#misc" class="tx-footer__section-link">Misc</a> -->
                 </div>
 
                 <div class="tx-footer__section">
@@ -136,7 +136,7 @@
                 <div class="tx-footer__section">
                     <div class="tx-footer__section-title">Company</div>
                     <a href="https://calendly.com/dstackai/discovery-call" target="_blank" class="tx-footer__section-link external">Talk to an expert</a>
-                    <a href="/partners" class="tx-footer__section-link">Partners</a>
+                    <!-- <a href="/partners" class="tx-footer__section-link">Partners</a> -->
                     <a href="/terms" class="tx-footer__section-link">Terms</a>
                     <a href="/privacy" class="tx-footer__section-link">Privacy</a>
                     <!--<a href="https://dstackai.notion.site/Jobs-at-dstack-ff01a450c3174904b70c36124f2d8f4b" target="_blank" class="tx-footer__section-link external">Careers</a>-->

pyproject.toml

@@ -166,7 +166,11 @@ lambda = [
     "dstack[server]",
 ]
 oci = [
-    "oci",
+    "oci>=2.150.0",
+    "cryptography>=44.0.3",
+    # pyopenssl is indirect to avoid uv falling back to the old version
+    # due to an upper limit from oci
+    "pyopenssl>=23.2.0",
     "dstack[server]",
 ]
 nebius = [

src/dstack/_internal/core/models/configurations.py

@@ -23,6 +23,9 @@
 MAX_INT64 = 2**63 - 1
 SERVICE_HTTPS_DEFAULT = True
 STRIP_PREFIX_DEFAULT = True
+RUN_PRIOTIRY_MIN = 0
+RUN_PRIOTIRY_MAX = 100
+RUN_PRIORITY_DEFAULT = 0
 
 
 class RunConfigurationType(str, Enum):
@@ -221,14 +224,26 @@ class BaseRunConfiguration(CoreModel):
             )
         ),
     ] = None
-    # deprecated since 0.18.31; task, service -- no effect; dev-environment -- executed right before `init`
-    setup: CommandsList = []
     resources: Annotated[
         ResourcesSpec, Field(description="The resources requirements to run the configuration")
     ] = ResourcesSpec()
+    priority: Annotated[
+        Optional[int],
+        Field(
+            ge=RUN_PRIOTIRY_MIN,
+            le=RUN_PRIOTIRY_MAX,
+            description=(
+                f"The priority of the run, an integer between `{RUN_PRIOTIRY_MIN}` and `{RUN_PRIOTIRY_MAX}`."
+                " `dstack` tries to provision runs with higher priority first."
+                f" Defaults to `{RUN_PRIORITY_DEFAULT}`"
+            ),
+        ),
+    ] = None
     volumes: Annotated[
         List[Union[MountPoint, str]], Field(description="The volumes mount points")
     ] = []
+    # deprecated since 0.18.31; task, service -- no effect; dev-environment -- executed right before `init`
+    setup: CommandsList = []
 
     @validator("python", pre=True, always=True)
     def convert_python(cls, v, values) -> Optional[PythonVersion]:

src/dstack/_internal/server/background/tasks/process_submitted_jobs.py

@@ -93,11 +93,20 @@ async def _process_next_submitted_job():
         async with lock:
             res = await session.execute(
                 select(JobModel)
+                .join(JobModel.run)
                 .where(
                     JobModel.status == JobStatus.SUBMITTED,
                     JobModel.id.not_in(lockset),
                 )
-                .order_by(JobModel.last_processed_at.asc())
+                # Jobs are process in FIFO sorted by priority globally,
+                # thus runs from different project can "overtake" each other by using higher priorities.
+                # That's not a big problem as long as projects do not compete for the same compute resources.
+                # Jobs with lower priorities from other projects will be processed without major lag
+                # as long as new higher priority runs are not constantly submitted.
+                # TODO: Consider processing jobs from different projects fairly/round-robin
+                # Fully fair processing can be tricky to implement via the current DB queue as
+                # there can be many projects and we are limited by the max DB connections.
+                .order_by(RunModel.priority.desc(), JobModel.last_processed_at.asc())
                 .limit(1)
                 .with_for_update(skip_locked=True)
             )

src/dstack/_internal/server/migrations/versions/bca2fdf130bf_add_runmodel_priority.py

@@ -0,0 +1,34 @@
+"""Add RunModel.priority
+
+Revision ID: bca2fdf130bf
+Revises: 20166748b60c
+Create Date: 2025-05-14 15:24:21.269775
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "bca2fdf130bf"
+down_revision = "20166748b60c"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table("runs", schema=None) as batch_op:
+        batch_op.add_column(sa.Column("priority", sa.Integer(), nullable=True))
+    batch_op.execute("UPDATE runs SET priority = 0")
+    with op.batch_alter_table("runs", schema=None) as batch_op:
+        batch_op.alter_column("priority", nullable=False)
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table("runs", schema=None) as batch_op:
+        batch_op.drop_column("priority")
+
+    # ### end Alembic commands ###

src/dstack/_internal/server/models.py

@@ -348,6 +348,7 @@ class RunModel(BaseModel):
     resubmission_attempt: Mapped[int] = mapped_column(Integer, default=0)
     run_spec: Mapped[str] = mapped_column(Text)
     service_spec: Mapped[Optional[str]] = mapped_column(Text)
+    priority: Mapped[int] = mapped_column(Integer, default=0)
 
     jobs: Mapped[List["JobModel"]] = relationship(
         back_populates="run", lazy="selectin", order_by="[JobModel.replica_num, JobModel.job_num]"