Eval bug: Model cannot load into my second GPU

[CCJetWing]

Name and Version

version: 9683 (8086439)
built with GNU 15.2.0 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA

Hardware

Ryzen 5900X, RTX 3090 *2

Models

Qwen3.6-27B-Q5_K_M, Qwen3.6-27B-Q4_K_M, Qwen3.6-27B-Q3_K_M

Problem description & steps to reproduce

even though I use -ts 1,1 the model is not separated into 2 GPUs and with -mg 1 the model only loads to CUDA0

First Bad Commit

No response

Relevant log output

~/llama.cpp/build/bin/llama-server -m ~/LLM/Qwen3.6-27B-Q3_K_M.gguf --mmproj ~/LLM/mmproj-BF16.gguf --port 8000 -mg 1 -ts 1,1
0.00.073.902 I log_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.073.917 I device_info:
0.00.245.532 I - CUDA0 : NVIDIA GeForce RTX 3090 (24575 MiB, 31032 MiB free)
0.00.389.626 I - CUDA1 : NVIDIA GeForce RTX 3090 (24575 MiB, 30950 MiB free)
0.00.389.654 I - CPU : AMD Ryzen 9 5900X 12-Core Processor (32045 MiB, 32045 MiB free)
0.00.389.752 I system_info: n_threads = 12 (n_threads_batch = 12) / 24 | CUDA : ARCHS = 860 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | FA_ALL_QUANTS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.389.771 I srv llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.389.861 I srv init: using 23 threads for HTTP server
0.00.390.265 I srv start: binding port with default address family
0.00.391.598 I srv llama_server: loading model
0.00.391.619 I srv load_model: loading model '/home/ubuntu/LLM/Qwen3.6-27B-Q3_K_M.gguf'
0.00.676.052 I srv load_model: [mtmd] estimated worst-case memory usage of mmproj is 1134.00 MiB
0.00.676.100 I common_init_result: fitting params to device memory ...
0.00.676.101 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.03.745.123 W llama_context: n_ctx_seq (244736) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.04.518.644 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 15296.00 MiB on device 0: cudaMalloc failed: out of memory
0.04.518.665 E alloc_tensor_range: failed to allocate CUDA0 buffer of size 16039018496
0.04.520.244 E llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache
0.04.520.269 E common_init_result: failed to create context with model '/home/ubuntu/LLM/Qwen3.6-27B-Q3_K_M.gguf'
0.04.520.275 E common_init_from_params: failed to create context with model '/home/ubuntu/LLM/Qwen3.6-27B-Q3_K_M.gguf'
Segmentation fault ~/llama.cpp/build/bin/llama-server -m ~/LLM/Qwen3.6-27B-Q3_K_M.gguf --mmproj ~/LLM/mmproj-BF16.gguf --port 8000 -mg 1 -ts 1,1

nvidia-smi
Fri Jun 19 12:14:04 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 610.47 KMD Version: 610.47 CUDA UMD Version: 13.3 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Driver-Model | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3090 WDDM | 00000000:0A:00.0 On | N/A |
| 53% 56C P2 156W / 390W | 21956MiB / 24576MiB | 97% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 3090 WDDM | 00000000:0B:00.0 On | N/A |
| 0% 53C P2 140W / 390W | 332MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

[vbooka1]

| 0 NVIDIA GeForce RTX 3090 WDDM | 00000000:0A:00.0 On | N/A |
| 53% 56C P2 156W / 390W | 21956MiB / 24576MiB | 97% Default |

"21956MiB / 24576MiB" = your GPU 0 has something loaded already when you are trying to load the new model.

If you want to use GPU 1 only then you can use the following environment variables:

CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1 ~/llama.cpp/build/bin/llama-server ...