how-to setup on dfki cluster?

[ArneBinder]

Not clear how to get into a state where one can do the installation.

[ArneBinder]

After trying it out, I recommend to recommend uv.

Installation instructions: https://docs.astral.sh/uv/getting-started/installation/

How-to start a model in online mode:

1. Create uv cache directory once: mkdir -p /netscratch/$USER/cache/uv
2. Run gpt-oss-2b:

srun --partition=RTXA6000-SLT \
     --job-name=vllm_serve \
     --nodes=1 \
     --ntasks=1 \
     --cpus-per-task=6 \
     --gpus-per-task=1 \
     --mem-per-cpu=4G \
     --time=1-00:00:00 \
     uv run --with vllm --cache-dir /netscratch/$USER/cache/uv \
            vllm serve "openai/gpt-oss-20b" \
                       --download-dir=/ds/models/llms/cache \
                       --port=8000

Note: This may take some time, wait for Application startup complete.

[Erechtheus]

Good catch! I’m just too lazy to switch from conda to uv ;-)