Merge branch 'main' into spolisetty/tri-1194-integrate-triton-logging-into-dynamo

Author: pskiran1

.pre-commit-config.yaml

@@ -1,28 +1,5 @@
-# Copyright 2023-2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions
-# are met:
-#  * Redistributions of source code must retain the above copyright
-#    notice, this list of conditions and the following disclaimer.
-#  * Redistributions in binary form must reproduce the above copyright
-#    notice, this list of conditions and the following disclaimer in the
-#    documentation and/or other materials provided with the distribution.
-#  * Neither the name of NVIDIA CORPORATION nor the names of its
-#    contributors may be used to endorse or promote products derived
-#    from this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
-# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
-# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
-# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
-# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
-# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# SPDX-FileCopyrightText: Copyright (c) 2023-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: BSD-3-Clause
 
 repos:
 - repo: https://github.com/PyCQA/isort
@@ -73,12 +50,24 @@ repos:
   - id: requirements-txt-fixer
   - id: trailing-whitespace
 
-- repo: local
+- repo: https://github.com/triton-inference-server/developer_tools
+  rev: v0.1.0
   hooks:
   - id: add-license
-    name: Add License
-    entry: python tools/add_copyright.py
+- repo: local
+  hooks:
+  # Incremental SPDX-header adoption: migrate each source file to the two-line
+  # SPDX header the first time it is touched. pre-commit runs hooks only on the
+  # files staged in a commit, so scoping by file type (below) -- not by directory
+  # -- rolls SPDX out gradually as files change. The hook maintains an existing
+  # SPDX header, replaces a legacy long-form NVIDIA BSD header in place, or
+  # inserts one. It runs after add-license above, which only maintains the
+  # copyright year on the string both headers share.
+  - id: add-spdx-license
+    name: Add SPDX License Header
+    entry: python tools/add_spdx_header.py
     language: python
+    files: \.(py|pyi|sh|bash|yaml|yml|cc|cpp|cxx|h|hpp|cu|cuh)$
     stages: [pre-commit]
     verbose: true
     require_serial: true

Dockerfile.QA

@@ -353,8 +353,8 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 RUN rm -f /usr/bin/python && \
     ln -s /usr/bin/python3 /usr/bin/python
 
-RUN pip3 install --upgrade "numpy<2" pillow attrdict future "grpcio<1.68" requests gsutil \
-                           "awscli<=1.36.40" six "grpcio-channelz<1.68" prettytable virtualenv \
+RUN pip3 install --upgrade "numpy<2" pillow attrdict future "grpcio>=1.81.1" requests gsutil \
+                           "awscli<=1.36.40" six grpcio-channelz prettytable virtualenv \
                            check-jsonschema
 
 # go needed for example go client test.

Dockerfile.sdk

@@ -83,7 +83,7 @@ RUN apt-get update && \
             software-properties-common \
             vim \
             wget && \
-    pip3 install --upgrade "grpcio-tools<1.68" cmake==4.0.3 auditwheel
+    pip3 install --upgrade grpcio-tools cmake==4.0.3 auditwheel
 
 ENV CMAKE_POLICY_MINIMUM_REQUIRED=3.5
 
@@ -182,7 +182,7 @@ RUN apt-get update && \
             python3-wheel \
             vim \
             wget && \
-    pip3 install "grpcio<1.68" "grpcio-tools<1.68" && \
+    pip3 install "grpcio>=1.81.1" grpcio-tools && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/* /var/cache/apt/archives/*;
 

build.py

@@ -1,29 +1,6 @@
 #!/usr/bin/env python3
-# Copyright 2020-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions
-# are met:
-#  * Redistributions of source code must retain the above copyright
-#    notice, this list of conditions and the following disclaimer.
-#  * Redistributions in binary form must reproduce the above copyright
-#    notice, this list of conditions and the following disclaimer in the
-#    documentation and/or other materials provided with the distribution.
-#  * Neither the name of NVIDIA CORPORATION nor the names of its
-#    contributors may be used to endorse or promote products derived
-#    from this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
-# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-# PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
-# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
-# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
-# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
-# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
-# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# SPDX-FileCopyrightText: Copyright (c) 2020-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: BSD-3-Clause
 
 import argparse
 import importlib.util
@@ -2556,6 +2533,20 @@ def enable_all():
         required=False,
         help="Override specified backend CMake argument in the build as <backend>:<name>=<value>. The argument is passed to CMake as -D<name>=<value>. This flag only impacts CMake arguments that are used by build.py. To unconditionally add a CMake argument to the backend build use --extra-backend-cmake-arg.",
     )
+    parser.add_argument(
+        "--build-presets-file",
+        type=str,
+        required=False,
+        default=None,
+        help="(EXPERIMENTAL; requires TRITON_BUILD_EXPERIMENTAL=1) Path to a build "
+        "presets JSON file that pins per-component cmake flags (tag, cmake_args, "
+        "extra_cmake_args, library_path) for core/backends/repoagents/caches. "
+        "Command-line flags take precedence over the file. With --dryrun, a "
+        "provenance-annotated snapshot of every resolved cmake flag (labeled "
+        "cli/preset/default) is written to build_presets.json in the build "
+        "directory; that file can be loaded back with this flag. See "
+        "tools/build/build_presets.py for the schema.",
+    )
     parser.add_argument(
         "--release-version",
         required=False,
@@ -2870,6 +2861,45 @@ def enable_all():
             OVERRIDE_BACKEND_CMAKE_FLAGS[be] = {}
         OVERRIDE_BACKEND_CMAKE_FLAGS[be][parts[0]] = parts[1]
 
+    # Per-backend clone-organization overrides. Empty unless the experimental
+    # --build-presets-file supplies a TRITON_REPO_ORGANIZATION for a backend, in
+    # which case the backend build loop below clones from that organization.
+    backend_org_overrides = {}
+
+    # Snapshots captured BEFORE applying a preset, so the --dryrun dump can tell
+    # which values came from the CLI vs the preset file vs build.py defaults.
+    _bp_before_backends = dict(backends)
+    _bp_before_repoagents = dict(repoagents)
+    _bp_before_caches = dict(caches)
+    _bp_cli_extra_be = {be: set(m) for be, m in EXTRA_BACKEND_CMAKE_FLAGS.items()}
+    _bp_cli_override_be = {be: set(m) for be, m in OVERRIDE_BACKEND_CMAKE_FLAGS.items()}
+    _bp_cli_core = set(EXTRA_CORE_CMAKE_FLAGS) | set(OVERRIDE_CORE_CMAKE_FLAGS)
+
+    # Experimental: apply a build preset (all load/validate/precedence logic lives
+    # in tools/build/build_presets.py). Command-line flags win over the file.
+    if FLAGS.build_presets_file is not None:
+        from tools.build import build_presets
+
+        try:
+            for msg in build_presets.apply(
+                FLAGS.build_presets_file,
+                backends=backends,
+                repoagents=repoagents,
+                caches=caches,
+                cli_backend_specs=FLAGS.backend,
+                cli_repoagent_specs=FLAGS.repoagent,
+                cli_cache_specs=FLAGS.cache,
+                library_paths=library_paths,
+                extra_backend_flags=EXTRA_BACKEND_CMAKE_FLAGS,
+                override_backend_flags=OVERRIDE_BACKEND_CMAKE_FLAGS,
+                extra_core_flags=EXTRA_CORE_CMAKE_FLAGS,
+                override_core_flags=OVERRIDE_CORE_CMAKE_FLAGS,
+                backend_org_overrides=backend_org_overrides,
+            ):
+                log(msg)
+        except build_presets.BuildPresetError as e:
+            fail(str(e))
+
     # Initialize map of common components and repo-tag for each.
     components = {
         "common": default_repo_tag,
@@ -2907,6 +2937,61 @@ def enable_all():
 
     # Write the build script that invokes cmake for the core, backends, repo-agents, and caches.
     pathlib.Path(FLAGS.build_dir).mkdir(parents=True, exist_ok=True)
+
+    # Experimental: on a --dryrun, emit a provenance-annotated snapshot of the
+    # fully-resolved cmake configuration -- every -D each component receives, as
+    # in cmake_build -- to <build-dir>/build_presets.json, labeling each value's
+    # source (cli / preset / default). The file can be loaded back with
+    # --build-presets-file to pin every flag. Gated behind
+    # TRITON_BUILD_EXPERIMENTAL=1.
+    if FLAGS.dryrun and os.getenv("TRITON_BUILD_EXPERIMENTAL") == "1":
+        from tools.build import build_presets
+
+        _bp_be = [b for b in backends if b not in CORE_BACKENDS]
+        try:
+            for msg in build_presets.write_snapshot(
+                os.path.join(FLAGS.build_dir, "build_presets.json"),
+                parser=parser,
+                argv=sys.argv,
+                core_args=core_cmake_args(
+                    components, backends, script_cmake_dir, script_install_dir
+                ),
+                backend_args={
+                    be: backend_cmake_args(
+                        images, components, be, script_install_dir, library_paths
+                    )
+                    for be in _bp_be
+                },
+                repoagent_args={
+                    ra: repoagent_cmake_args(images, components, ra, script_install_dir)
+                    for ra in repoagents
+                },
+                cache_args={
+                    ca: cache_cmake_args(images, components, ca, script_install_dir)
+                    for ca in caches
+                },
+                backends=backends,
+                repoagents=repoagents,
+                caches=caches,
+                before_backends=_bp_before_backends,
+                before_repoagents=_bp_before_repoagents,
+                before_caches=_bp_before_caches,
+                cli_backend_specs=FLAGS.backend,
+                cli_repoagent_specs=FLAGS.repoagent,
+                cli_cache_specs=FLAGS.cache,
+                cli_extra_be=_bp_cli_extra_be,
+                cli_override_be=_bp_cli_override_be,
+                cli_core=_bp_cli_core,
+                extra_backend_flags=EXTRA_BACKEND_CMAKE_FLAGS,
+                override_backend_flags=OVERRIDE_BACKEND_CMAKE_FLAGS,
+                extra_core_flags=EXTRA_CORE_CMAKE_FLAGS,
+                override_core_flags=OVERRIDE_CORE_CMAKE_FLAGS,
+                library_paths=library_paths,
+            ):
+                log(msg)
+        except build_presets.BuildPresetError as e:
+            fail(str(e))
+
     with BuildScript(
         os.path.join(FLAGS.build_dir, script_name),
         verbose=FLAGS.verbose,
@@ -2940,7 +3025,9 @@ def enable_all():
             if be == "armnn_tflite":
                 github_organization = "https://gitlab.com/arm-research/smarter/"
             else:
-                github_organization = FLAGS.github_organization
+                github_organization = backend_org_overrides.get(
+                    be, FLAGS.github_organization
+                )
 
             if be == "vllm":
                 backend_clone(

docs/customization_guide/build.md

@@ -180,6 +180,75 @@ you have a branch called "mybranch" in the
 repo that you want to use in the build, you would specify
 --backend=onnxruntime:mybranch.
 
+#### Experimental: Build Presets
+
+> **Experimental.** This feature is gated behind the
+> `TRITON_BUILD_EXPERIMENTAL=1` environment variable; without it the
+> `--build-presets-file` flag is rejected.
+
+Build presets let you (a) inspect exactly which cmake flags each component
+receives, and (b) pin those flags via a single JSON file with
+`--build-presets-file <path>`.
+
+**Snapshot (dump).** When run with `--dryrun`, build.py writes a
+provenance-annotated snapshot of the fully-resolved cmake configuration to
+`build_presets.json` in the build directory. It records, for `core`, each
+`backend`, `repoagent`, and `cache`, every `-D` flag that lands in `cmake_build`,
+each labeled with its `source`: `cli` (explicit command line), `preset` (from a
+loaded presets file), or `default` (build.py default/derived):
+
+```json
+{
+  "backends": {
+    "onnxruntime": {
+      "tag": { "value": "main", "source": "default" },
+      "cmake_args": {
+        "TRITON_ENABLE_GPU": { "value": "ON", "source": "cli" },
+        "TRITON_BUILD_ONNXRUNTIME_VERSION": { "value": "1.27.0", "source": "default" }
+      }
+    }
+  }
+}
+```
+
+**Reload.** That same file can be fed back with `--build-presets-file` to pin its
+flags (the `source` field is informational on load). A hand-written preset may
+use bare scalars instead of `{value, source}` objects:
+
+```json
+{
+  "backends": {
+    "onnxruntime": {
+      "tag": "r25.08_fix",
+      "cmake_args": { "TRITON_ENABLE_ONNXRUNTIME_OPENVINO": "OFF" }
+    },
+    "python": {
+      "extra_cmake_args": { "TRITON_BOOST_URL": "https://.../boost_1_80_0.tar.gz" }
+    }
+  }
+}
+```
+
+Notes:
+- A component named in the file must also be included in the build (via
+  `--backend`/`--repoagent`/`--cache` or `--enable-all`).
+- Command-line flags always take precedence over the file.
+- `cmake_args` are flags build.py emits natively (applied via the override
+  channel); `extra_cmake_args` are user-added flags build.py does not emit
+  (applied via the append channel). Setting `TRITON_REPO_ORGANIZATION` in a
+  backend's `cmake_args` sets that backend's clone organization *per backend*
+  (both the `git clone` URL and the `-D`), which the global
+  `--github-organization` cannot do.
+- `CMAKE_INSTALL_PREFIX` is omitted from snapshots (it is an absolute build-dir
+  path). Repoagent/cache `cmake_args` are shown for visibility but only their
+  `tag` is re-pinned on load. Reloading pins `-D` values but does not re-derive
+  conditionally-emitted flags, so reload alongside the same top-level flags (or
+  `--enable-all`) for exact reproduction.
+
+A ready-to-copy example lives at
+[`tools/build/build_presets.example.json`](../../tools/build/build_presets.example.json);
+the full schema is documented in `tools/build/build_presets.py`.
+
 #### CPU-Only Build
 
 If you want to build without GPU support you must specify individual

docs/customization_guide/deploy.md

@@ -38,7 +38,9 @@ standalone process.  This guide is intended to provide some key points
 and best practices that users deploying Triton based solutions should
 consider.
 
-| [Deploying Behind a Secure Gateway or Proxy](#deploying-behind-a-secure-proxy-or-gateway) | [Running with Least Privilege](#running-with-least-privilege) |
+| [Deploying Behind a Secure Proxy or Gateway](#deploying-behind-a-secure-proxy-or-gateway) |
+[Securing Model and Backend Code](#securing-model-and-backend-code) |
+[Running with Least Privilege](#running-with-least-privilege) |
 
 > [!IMPORTANT]
 > Ultimately the security of a solution based on Triton
@@ -84,6 +86,31 @@ as an "Application" or "Service" within the trusted internal network.
 * [https://konghq.com/blog/enterprise/envoy-service-mesh]
 * [https://www.solo.io/topics/envoy-proxy/]
 
+## Securing Model and Backend Code
+
+> [!WARNING]
+> Some Triton backends execute code loaded from the model repository.
+> Depending on the backend, this code may run in the Triton server
+> process or in a separate process managed by the backend. It can
+> exercise the operating-system privileges and access available to that
+> process, including filesystem access, available credentials, and
+> network access. Triton does not provide a security sandbox for
+> arbitrary model or backend code.
+
+Only deploy executable model and backend code from trusted sources.
+Restrict write access to model repositories and backend directories,
+restrict model control APIs to trusted operators when they are enabled,
+and review executable code before deployment.
+
+Even when a caller is authenticated and authorized, treat
+request-derived values as untrusted until they have been validated for
+their intended use. Before using these values in outbound network
+requests, filesystem operations, subprocess invocations,
+deserialization, or media decoding, validate them against an explicit
+policy and enforce limits on input size, execution time, concurrency,
+and other resource use. Use deployment-level controls, such as outbound
+network restrictions, to limit the impact of validation failures.
+
 ## Running with Least Privilege
 
   The security principle of least privilege advocates that a process be
@@ -293,7 +320,11 @@ Specifies the maximum number of states (inference request/response queues) that
 Specifies the maximum number of inference response objects that can remain allocated in each response queue at any given time. This option is particularly useful in decoupled mode, where multiple responses are generated for a single request. By default, this value is set to `INT_MAX`.
 
 > [!Warning]
-> Setting this value too low may negatively impact performance.
-
-
+> Operators exposing decoupled gRPC streaming to untrusted clients should
+> explicitly configure `--grpc-max-response-pool-size` to a bounded value
+> appropriate for their workload, rather than relying on the default
+> `INT_MAX`. Choose a value based on model behavior (decoupled vs
+> non-decoupled), expected concurrency, and available memory.
 
+> [!Warning]
+> Setting this value too low may negatively impact performance.