Add GPU/Device Support and Fix Symlink Deduplication Issues (#176)

* Implement docker device request in CLI

Now one can pass in the cro runtime and GPU like
  --cro-device-request '{"Count":-1, \
    "Capabilities":[["gpu"]]}' \
  --cro-runtime nvidia \
  ...

* Fix issues with duplicate symlink processing

Correct issues with duplicate paths from symlink
processing, when they both point to the same
inode. This fixes a behavior where slimtookit
would randomly generate o-byte files for the
actual file referenced by the symlink.

* fix: resolve ptrace monitor bugs causing hangs with multiprocess apps

Infinite loop in getStringParam: when a syscall had a NULL path
argument, PtracePeekData returned EIO with count=0. The function
silenced EIO, never advanced the pointer, and had no exit condition,
burning 100% CPU until exit. Fix: return early
on NULL pointer and on any PtracePeekData error.

* fix null ref crash when ptrace disabled

* fix: prevent namespace package directories from being excluded by IsSubdir false positive

OKReturnStatus for checkFile syscalls intentionally accepts ENOENT (-2)
to track Python import search paths. When Python imports a namespace
package (a directory without __init__.py), it probes for several
__init__.* variants -- all returning ENOENT -- before stat'ing the
directory itself (success). The radix tree walk in FileActivity() then
sees the directory as a prefix of these ghost child paths and marks it
IsSubdir=true, excluding it from the slim image. Since the ghost
children don't exist on disk, neither the directory nor any files are
preserved.

Add HasSuccessfulAccess to FSActivityInfo, set it only when retVal==0,
and require it in the IsSubdir determination so that ENOENT-only child
paths cannot cause a parent directory to be dropped.

* example use case with nvidia runtime

* Fix HasSuccessfulAccess for open-type syscalls where success is fd >= 0, not retVal == 0

* Key deduplicateFileMap by (dev, inode) instead of inode alone to avoid false dedup across mounts

* Fail fast on invalid --cro-device-request JSON instead of silently dropping the device config

* Remove unused SIGNAL_PIPE named pipe from test_vllm.sh

* Update pkg/monitor/ptrace/ptrace.go

Co-authored-by: kilo-code-bot[bot] <240665456+kilo-code-bot[bot]@users.noreply.github.com>

* Update pkg/monitor/ptrace/ptrace.go

Co-authored-by: kilo-code-bot[bot] <240665456+kilo-code-bot[bot]@users.noreply.github.com>

* patch up kilo-code-bot fix

* correct indentation

* Revert HasSuccessfulAccess to use retVal==0 check instead of OKReturnStatus

The kilo-code-bot suggested changing the HasSuccessfulAccess condition from
`e.retVal == 0 || p.SyscallType() == OpenFileType` to `p.OKReturnStatus(e.retVal)`,
arguing that the former incorrectly treated all OpenFileType syscalls as successful.

However, the original expression was correct:

- For OpenFileType (open, openat, readlink): the outer condition on line 276-278
  already filters to successful calls via `p.OKReturnStatus(e.retVal)`, which
  requires fd >= 0. The `|| p.SyscallType() == OpenFileType` clause is redundant,
  not wrong -- it simply ensures HasSuccessfulAccess=true for events that already
  passed the success filter.

- For CheckFileType (stat, access, lstat): `e.retVal == 0` correctly marks only
  files that actually exist on disk. The replacement `p.OKReturnStatus(e.retVal)`
  also accepts ENOENT (-2) and ENOTDIR (-20), causing non-existent "ghost" paths
  to be marked as HasSuccessfulAccess=true.

This broke the namespace package fix (da8eb2c): when Python probes for
__init__.cpython-311.so, __init__.so, __init__.py in a directory that has none
of them (namespace package), those ENOENT stat results created ghost children
with HasSuccessfulAccess=true. The FileActivity() radix tree walk then marked
the parent directory as IsSubdir and excluded it from the slim image. Since the
ghost children don't exist on disk, neither the directory nor its contents were
preserved, causing libraries to go missing.

* chore: moved to vLLM v0.17.1-cu130 for testing

* lint: fixed indenting in ptrace.go with gofmt

* add unit tests for bugs related to ghost paths

* Revert OKReturnStatus to success-only and remove HasSuccessfulAccess

OKReturnStatus for checkFileSyscallProcessor was broadened in 9aca3e9 to
accept ENOENT (-2) and ENOTDIR (-20) for Python import tracking. However,
ghost paths (non-existent files) that entered fsActivity via this gate
were never used by the artifact pipeline — prepareArtifact() discards them
at os.Lstat(). Meanwhile they caused three side effects:

1. Ghost leaf paths leaked into Report.FSActivity and triggered
   prepareArtifact() calls for non-existent files
2. Ghost paths were published as MDETypeArtifact events
3. The permissive OKReturnStatus semantics caused a naming trap that led
   to a regression (3f8dcb4) where HasSuccessfulAccess was set using
   OKReturnStatus, breaking namespace package directories

This reverts OKReturnStatus to the upstream original (retVal == 0),
removes the HasSuccessfulAccess field and its plumbing (no longer needed
since only real paths enter fsActivity), and simplifies the FileActivity()
radix tree walk.

Tests updated to verify: ENOENT paths are rejected, no ghost paths in
reports, namespace package directories are preserved, IsSubdir works
correctly for real children.

---------

Co-authored-by: kilo-code-bot[bot] <240665456+kilo-code-bot[bot]@users.noreply.github.com>

Author: akshaver

examples/nvidia_runtime/.gitignore

@@ -0,0 +1,5 @@
+host-config.json
+vllm_test_results.json
+vllm_test_results_slim.json
+original_log.txt
+slim_log.txt

examples/nvidia_runtime/Dockerfile

@@ -0,0 +1,6 @@
+FROM nvcr.io/nvidia/pytorch:25.04-py3
+
+# Add the tests to the entrypoint set. Docker Slim only traces/monitors the processes started by the entrypoint.
+RUN echo "pytest /opt/pytorch/pytorch/test/test_cuda.py::TestCuda::test_graph_cudnn_dropout" > /opt/nvidia/entrypoint.d/99-trace.sh
+RUN chmod +x /opt/nvidia/entrypoint.d/99-trace.sh
+

examples/nvidia_runtime/README.md

@@ -0,0 +1,19 @@
+
+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+As a pre-requisite, install nvidia-container toolkit, including adding the nvidia runtime. Then you should be able to translate runtime and capabilities from a OCI/Docker string like `--runtime=nvidia --gpus all` to `--cro-device-request '{"Count":-1, "Capabilities":[["gpu"]]}' --cro-runtime nvidia`
+
+See the example `test_nvidia_smi.sh`, which slims ubuntu to just the files necessary to run the runtime mounted nvidia-smi. Similarly, see `test_nvidia_pytorch.sh` which minimizes nvidia-pytorch to run a subset of the CUDA tests.
+

examples/nvidia_runtime/test_nvidia_pytorch.sh

@@ -0,0 +1,97 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Create host config file with ulimit settings and capabilities
+cat > host-config.json <<'EOF'
+{
+  "IpcMode": "host",
+  "CapAdd": ["SYS_ADMIN"],
+  "Ulimits": [
+    {
+      "Name": "memlock",
+      "Soft": -1,
+      "Hard": -1
+    },
+    {
+      "Name": "stack",
+      "Soft": 67108864,
+      "Hard": 67108864
+    },
+    {
+      "Name": "nofile",
+      "Soft": 1048576,
+      "Hard": 1048576
+    }
+  ]
+}
+EOF
+
+# Build the slim image
+# CAP_SYS_ADMIN is added via host-config.json for fanotify support (required for filesystem monitoring)
+# Build custom image with test in entrypoint first
+echo "Building custom test image with pytest in entrypoint..."
+docker build -t nvcr.io/nvidia/pytorch:25.04-py3-test -f Dockerfile .
+
+echo "Running mint on the test image..."
+mint build \
+  --target nvcr.io/nvidia/pytorch:25.04-py3-test \
+  --tag nvcr.io/nvidia/pytorch:25.04-py3-slim \
+  --cro-host-config-file host-config.json \
+  --cro-shm-size 1200 \
+  --cro-device-request '{"Count":-1, "Capabilities":[["gpu"]]}' \
+  --cro-runtime nvidia \
+  --http-probe=false \
+  --continue-after 10 \
+  --preserve-path /etc/ld.so.conf \
+  --preserve-path /etc/ld.so.conf.d \
+  .
+
+# Get output of original and slim images stored in a log file
+echo "Running original image..."
+docker run --rm --runtime nvidia --gpus all nvcr.io/nvidia/pytorch:25.04-py3-test > original_log.txt 2>&1
+echo "Running slim image..."
+docker run --rm --runtime nvidia --gpus all nvcr.io/nvidia/pytorch:25.04-py3-slim > slim_log.txt 2>&1
+
+# Verify that both logs contain the pytest success message (ignoring timing)
+echo "Checking test results..."
+
+# Look for "X passed" pattern in both logs
+original_passed=$(grep -oE "[0-9]+ passed" original_log.txt | head -1)
+slim_passed=$(grep -oE "[0-9]+ passed" slim_log.txt | head -1)
+
+if [ -z "$original_passed" ]; then
+    echo "Error: Original image test did not pass"
+    echo "Original log tail:"
+    tail -20 original_log.txt
+    exit 1
+fi
+
+if [ -z "$slim_passed" ]; then
+    echo "Error: Slim image test did not pass"
+    echo "Slim log tail:"
+    tail -20 slim_log.txt
+    exit 1
+fi
+
+echo "Original image: $original_passed"
+echo "Slim image: $slim_passed"
+
+if [ "$original_passed" = "$slim_passed" ]; then
+    echo "SUCCESS: Both images passed the same number of tests!"
+else
+    echo "Warning: Different number of tests passed (original: $original_passed, slim: $slim_passed)"
+fi
+
+echo "Successfully minimized nvidia-pytorch to run a subset of the CUDA tests"

examples/nvidia_runtime/test_nvidia_smi.sh

@@ -0,0 +1,34 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Build the slim image
+mint build --target ubuntu:24.04 --tag ubuntu:24.04-slim   --cro-shm-size 1200 --cro-device-request '{"Count":-1, "Capabilities":[["gpu"]]}' --cro-runtime nvidia --http-probe=false --exec "/usr/bin/nvidia-smi" .
+
+# Get output of original and slim images stored in a log file
+docker run --rm --runtime nvidia --gpus all ubuntu:24.04 nvidia-smi > original_log.txt
+docker run --rm --runtime nvidia --gpus all ubuntu:24.04-slim nvidia-smi > slim_log.txt
+
+# verify that both logs include the nvidia-smi output with an assert
+assert_contains() {
+    if ! grep -q "$1" "$2"; then
+        echo "Error: '$1' not found in $2"
+        exit 1
+    fi
+}
+
+# verify that both logs include the nvidia-smi output with an assert
+assert_contains "NVIDIA-SMI" original_log.txt
+assert_contains "NVIDIA-SMI" slim_log.txt
+