Add minimal H100/H200/B200 inference Dockerfiles (no CUDA toolkit at runtime, non-root; Wolfi + Chainguard-FIPS variants)

[erikrozi]

Motivation

The main docker/Dockerfile (829 lines) is a multi-CUDA, multi-arch build matrix with a devel base in the final image (nvcc kept for runtime JIT). For single-node inference on our own H100/H200/B200 cluster we want a Chainguard-style minimal image: small CVE surface, minimal toolchain, non-root, suitable for hardened (FedRAMP/FedStart-style) clusters — with a FIPS-base option.

Modifications

Two new Dockerfiles. Both share an identical builder stage and differ only in the runtime base.

docker/min-inference.Dockerfile — two stages:

builder: nvidia/cuda:13.0.1-cudnn-devel-ubuntu24.04
  - pip install the pinned released wheel  sglang==0.5.12.post1  (prebuilt;
    includes the grpc rust extension, so no Rust/proto build) plus torch cu13,
    sgl-kernel, flash-attn, flashinfer wheels — nothing compiled vs local toolkit
  - kernels==0.14.1 pin (kernels>=0.15 needs LayerRepository(revision=...) that
    transformers 5.8.1 doesn't pass yet)
  - flashinfer-jit-cache cu130 (prebaked cubins)
  - kernels lock/download to prefetch sgl-kernel cubins
  - apply patches/glm4_moe_*.patch against the installed package (patch -p2)
  - assemble /opt/cuda-min: nvcc/ptxas/nvvm + CUDA headers + libcudadevrt/libcudart
    (~685MB vs ~5GB full toolkit; skipped when INCLUDE_NVCC=0)
        |
        |  COPY site-packages + caches + cuda-min + compat libs
        v
runtime: cgr.dev/chainguard/wolfi-base:latest
  - apk: python-3.12 + gcc-14-default + glibc-dev + binutils (Triton launcher
    stubs and nvcc's host compiler) + numactl/libgomp/libstdc++/zlib/openssl/curl
  - minimal nvcc subset at /usr/local/cuda so runtime kernel JIT
    (sgl-kernel jit e.g. fused_rope, DeepGEMM) compiles on first use
  - no git/ssh/apt; CUDA runtime libs ride inside the pip wheels;
    libcuda.so.1 injected by the NVIDIA container toolkit at run time
  - non-root numeric user 10001, caches under /home/sglang/.cache

docker/min-inference-fips.Dockerfile — FIPS variant for FedStart/FedRAMP. Identical builder; runtime base is parametrized via RUNTIME_BASE / RUNTIME_APK:

production (licensed): FROM cgr.dev/<ORG>/python-fips:3.12   (or chainguard-base-fips)
  - validated OpenSSL FIPS provider + STIG-hardened + non-root, maintained by CG
  - RUNTIME_APK empty: python 3.12 is in the base; the gcc-14/glibc-dev/binutils
    host-compile toolchain is delivered via Chainguard Custom Assembly, since the
    per-image license has no open APK repository
public prototype (no license): FROM wolfi-base + apk python-3.12 + toolchain
  - same substrate python-fips is assembled from (FIPS provider/STIG aside),
    so it builds and runs today for validation

Rationale (python-fips over pytorch-fips): this image brings its own pinned wheel stack (torch 2.11 cu13 + sgl-kernel + flashinfer, all cp312); a thin python base keeps that stack under our control, whereas pytorch-fips ships its own torch/CUDA on CG's cadence and would collide with our pinned ABI.

Key properties (both files):

• INCLUDE_NVCC build arg (default 1). Set 0 for a compiler-free image; then prebake or volume-mount the JIT caches under /home/sglang/.cache after a warmup run on a GPU node.
• Wolfi/Chainguard glibc declares rsqrt/rsqrtf noexcept under g++'s implicit _GNU_SOURCE, colliding with CUDA's crt/math_functions.h. Fixed by a builder-stage sed that adds noexcept (true) to those two declarations in the bundled cuda-min headers (replaces an earlier -U_GNU_SOURCE workaround that broke _GNU_SOURCE-gated POSIX symbols during host compile). gcc 14 is used because CUDA 13's nvcc rejects gcc >15.
• TORCH_CUDA_ARCH_LIST build arg, default "9.0;10.0" (H100/H200 + B200); override for single-arch builds.
• CUDA forward-compat libs at /usr/local/cuda/compat (libcuda 580.82.07) for older host drivers (e.g. R550): opt in with -e LD_LIBRARY_PATH="/usr/local/cuda/compat:/usr/local/nvidia/lib:/usr/local/nvidia/lib64".
• Explicit NVIDIA_VISIBLE_DEVICES=all / NVIDIA_DRIVER_CAPABILITIES=compute,utility so the container toolkit mounts compute libs under Kubernetes.
• Trivy --scanners vuln: upstream lmsysorg/sglang:latest = 2295 findings (2183 OS-level, 6 critical) vs this image = 2 findings, 0 OS-level (diskcache 5.6.3 MEDIUM no-fix and torch 2.11.0 CVE-2025-3000 LOW, both shared with upstream). Pinning the released wheel eliminated the prior 5 sglang 0.0.0.dev0 version-string artifacts seen on a source build.
• FedStart-style requirements: non-root numeric user, native TLS via --ssl-certfile/--ssl-keyfile/--ssl-ca-certs/--enable-ssl-refresh, cert-mount and air-gapped weights (HF_HUB_OFFLINE=1) notes in the header.
• GLM-4 MoE shared-expert TP patches under patches/; the shared_output / tp_size division is guarded with not should_use_flashinfer_cutlass_moe_fp4_allgather() since the FP4-allgather path skips the post-experts all-reduce (per Devin Review finding).
• Excluded on purpose (documented in header): RDMA/multi-node stack, sm_103/arm64.

Accuracy Tests

N/A — packaging only; no upstream model/kernel code changes beyond the GLM-4 patches. Verified on a CPU-only host for both images as uid 10001: image builds, import sglang reports 0.5.12.post1, python is 3.12.13, GLM-4 patch applied, and sglang's real runtime JIT path (_jit_fused_rope_module(is_neox=True, rope_dim=64, bf16) — the exact module that crashed an earlier GPU deploy) compiles+links+loads through tvm-ffi/ninja/nvcc/gcc-14 inside the Wolfi/Chainguard runtime (JIT_LOAD_OK). The only missing symbol at import is libcuda.so.1, the file the NVIDIA container toolkit injects on a GPU host.

Speed Tests and Profiling

N/A — packaging only.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

Link to Devin session: https://app.devin.ai/sessions/63e154c0f009403bb1b6252261c72502
Requested by: @erikrozi

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

/rerun-failed-ci

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 7 additional findings in Devin Review.

[devin-ai-integration]

🚩 Copying entire /usr/local/bin from builder includes unnecessary files

Line 112 copies all of /usr/local/bin from the builder into the runtime image. This includes pip, wheel, setuptools entry points, Rust/cargo binaries, and every other CLI tool installed during the build. The main Dockerfile (docker/Dockerfile) is more selective about what it copies. This increases image size unnecessarily and widens the attack surface for a hardened deployment image. Consider copying only the specific entry points needed at runtime (e.g., kernels, sglang entry points).

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 9 additional findings in Devin Review.

[devin-ai-integration]

🔴 Non-root user cannot create Triton cache dir due to root-owned intermediate .cache/ directory

The COPY --from=builder --chown=10001:10001 /root/.cache/huggingface /home/sglang/.cache/huggingface on line 113 implicitly creates the intermediate directory /home/sglang/.cache/ with root:root ownership (Docker/BuildKit applies --chown only to the destination and its contents, not to auto-created parent directories). Since the container runs as non-root user 10001 (line 128), and TRITON_CACHE_DIR is set to /home/sglang/.cache/triton (line 126), Triton will fail with a permission error when it tries to os.makedirs the cache directory at runtime, because user 10001 cannot create new subdirectories inside the root:root /home/sglang/.cache/ parent. This breaks the Triton runtime launcher compilation that the Dockerfile explicitly provisions for (lines 100-101 install gcc, glibc-dev, python-3.12-dev specifically for Triton JIT).

The fix is to ensure /home/sglang/.cache/ is owned by the runtime user. For example, add a RUN after the COPY lines:

RUN chown 10001:10001 /home/sglang/.cache

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Checked this against the actual built image — the premise doesn't hold here. With BuildKit, COPY --chown applies the ownership to auto-created parent directories as well, so /home/sglang/.cache ends up owned by the runtime user:

$ docker run --rm sglang:min-inference sh -c 'ls -ld /home/sglang/.cache && mkdir -p /home/sglang/.cache/triton && echo MKDIR_OK'
drwxr-sr-x    1 sglang   sglang        4096 Jun 10 16:19 /home/sglang/.cache
MKDIR_OK

The Triton cache dir is creatable by uid 10001 at runtime, so no change needed.

[devin-ai-integration]

Devin Review found 2 new potential issues.

[devin-ai-integration]

🚩 Dockerfile gcc comment may overstate nvcc compatibility range

Line 129 states "nvcc accepts (it rejects gcc > 15; Wolfi default is 16)" but NVIDIA's CUDA 13.0 documentation lists gcc 14 as the maximum supported host compiler. If the actual cutoff is gcc > 14 (not > 15), the comment is slightly misleading, though the code is correct since it installs gcc-14-default. Minor documentation accuracy issue.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

The comment matches this toolkit's actual behavior: nvcc 13.0.1's host-compiler check in this image errors with "unsupported GNU version! gcc versions later than 15 are not supported" (observed empirically when Wolfi's default gcc 16 was installed). The documented support matrix (gcc ≤14) is the officially tested range, but the hard cutoff enforced by crt/host_config.h is >15, which is what the comment describes. Keeping gcc-14-default keeps us within the documented matrix either way, so no code change needed.

[devin-ai-integration]

Devin Review found 3 new potential issues.

[devin-ai-integration]

🚩 Shared expert weight replication increases per-rank GPU memory

Patch 1 unconditionally sets tp_rank=0, tp_size=1 for shared experts, meaning every TP rank now loads the FULL shared expert weights rather than a 1/tp_size shard. For example, with 8-way TP, each rank holds the entire shared expert weight matrix (vs 1/8th before). This is a deliberate tradeoff — simplifying the TP logic at the cost of higher per-rank memory for the shared expert parameters. For production deployments on memory-constrained GPUs, this increased footprint should be validated against the target model's shared expert size.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Correct, and it's a deliberate tradeoff carried over from the deployment-side patches: GLM-4 MoE's shared expert is small relative to the routed experts (1 shared vs 128+ routed), so full replication costs roughly (tp_size-1)/tp_size of one expert's weights per rank — negligible against the routed expert memory on the target 8×H100 deployment, where it has been validated. Leaving as-is.

[devin-ai-integration]

🚩 Dual-stream overlap may degrade with non-TP shared experts

In forward_normal_dual_stream, shared experts run on the current stream while routed experts run on alt_stream for parallel execution. After Patch 1, shared experts compute the FULL result (not a TP-sharded partial), meaning they do tp_size× more FLOPs on the current stream. If the shared expert computation now takes longer than the routed experts, the current stream becomes the bottleneck and the dual-stream overlap benefit is reduced. This doesn't affect correctness but could impact latency in multi-GPU TP configurations.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Acknowledged — the shared expert does tp_size× more FLOPs on the current stream, but it's a single small expert vs the routed experts' much larger workload on alt_stream, so the routed path remains the critical path at realistic batch sizes. This mirrors what upstream already does unconditionally for all the a2a backends (deepep/mooncake/flashinfer/etc.); the patch just extends that to the plain TP path. Real-perf validation is happening on the 8×H100 load test.

[devin-ai-integration]

🚩 Wolfi runtime base uses :latest tag — builds are not reproducible

Line 125 uses cgr.dev/chainguard/wolfi-base:latest which means the runtime base image can change between builds. For a Dockerfile aimed at hardened/FedRAMP-style deployments (as stated in the header comments), pinning to a specific digest or date tag would improve build reproducibility and auditability. This is a best-practice concern rather than a correctness bug.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Fair best-practice point. Wolfi only publishes :latest (it's a rolling distro by design — that's how it keeps zero known CVEs), so the right pinning mechanism is a digest, which goes stale on every Wolfi rebuild. For now :latest is intentional so rebuilds pick up patched packages; for an audited release pipeline the build should resolve and record the digest (e.g. docker buildx imagetools inspect) at release time rather than hardcoding one in the Dockerfile. Happy to add a WOLFI_BASE build arg if a pinned digest workflow is wanted.