From e997feba0b9c8042f66709515aa9e69e5ba9f172 Mon Sep 17 00:00:00 2001 From: Misha Chornyi <99709299+mc-nv@users.noreply.github.com> Date: Wed, 11 Jun 2025 12:07:05 -0700 Subject: [PATCH 01/14] TPRD-1554: update readme and versions (#8244) --- Dockerfile.sdk | 2 +- README.md | 233 +----------------- TRITON_VERSION | 2 +- build.py | 8 +- deploy/aws/values.yaml | 2 +- deploy/fleetcommand/values.yaml | 6 +- deploy/gcp/values.yaml | 2 +- .../perf-analyzer-script/triton_client.yaml | 2 +- .../server-deployer/build_and_push.sh | 6 +- .../server-deployer/chart/triton/values.yaml | 6 +- .../server-deployer/schema.yaml | 2 +- .../gke-marketplace-app/trt-engine/README.md | 6 +- deploy/k8s-onprem/values.yaml | 2 +- deploy/oci/values.yaml | 2 +- docs/introduction/compatibility.md | 25 +- docs/introduction/release_notes.md | 4 +- python/openai/README.md | 2 +- qa/common/gen_jetson_trt_models | 2 +- qa/common/gen_qa_custom_ops | 2 +- qa/common/gen_qa_model_repository | 2 +- 20 files changed, 44 insertions(+), 274 deletions(-) diff --git a/Dockerfile.sdk b/Dockerfile.sdk index 33414c3fe4..d1c4b5a189 100644 --- a/Dockerfile.sdk +++ b/Dockerfile.sdk @@ -29,7 +29,7 @@ # # Base image on the minimum Triton container -ARG BASE_IMAGE=nvcr.io/nvidia/tritonserver:25.05-py3-min +ARG BASE_IMAGE=nvcr.io/nvidia/tritonserver:25.06-py3-min ARG TRITON_CLIENT_REPO_SUBDIR=clientrepo ARG TRITON_PA_REPO_SUBDIR=perfanalyzerrepo diff --git a/README.md b/README.md index 2de74ae96a..3acc072ce3 100644 --- a/README.md +++ b/README.md @@ -28,235 +28,4 @@ [![License](https://img.shields.io/badge/License-BSD3-lightgrey.svg)](https://opensource.org/licenses/BSD-3-Clause) >[!WARNING] ->You are currently on the `main` branch which tracks under-development progress ->towards the next release. The current release is version [2.58.0](https://github.com/triton-inference-server/server/releases/latest) ->and corresponds to the 25.05 container release on NVIDIA GPU Cloud (NGC). - -# Triton Inference Server - -Triton Inference Server is an open source inference serving software that -streamlines AI inferencing. Triton enables teams to deploy any AI model from -multiple deep learning and machine learning frameworks, including TensorRT, -TensorFlow, PyTorch, ONNX, OpenVINO, Python, RAPIDS FIL, and more. Triton -Inference Server supports inference across cloud, data center, edge and embedded -devices on NVIDIA GPUs, x86 and ARM CPU, or AWS Inferentia. Triton Inference -Server delivers optimized performance for many query types, including real time, -batched, ensembles and audio/video streaming. Triton inference Server is part of -[NVIDIA AI Enterprise](https://www.nvidia.com/en-us/data-center/products/ai-enterprise/), -a software platform that accelerates the data science pipeline and streamlines -the development and deployment of production AI. - -Major features include: - -- [Supports multiple deep learning - frameworks](https://github.com/triton-inference-server/backend#where-can-i-find-all-the-backends-that-are-available-for-triton) -- [Supports multiple machine learning - frameworks](https://github.com/triton-inference-server/fil_backend) -- [Concurrent model - execution](docs/user_guide/architecture.md#concurrent-model-execution) -- [Dynamic batching](docs/user_guide/model_configuration.md#dynamic-batcher) -- [Sequence batching](docs/user_guide/model_configuration.md#sequence-batcher) and - [implicit state management](docs/user_guide/architecture.md#implicit-state-management) - for stateful models -- Provides [Backend API](https://github.com/triton-inference-server/backend) that - allows adding custom backends and pre/post processing operations -- Supports writing custom backends in python, a.k.a. - [Python-based backends.](https://github.com/triton-inference-server/backend/blob/main/docs/python_based_backends.md#python-based-backends) -- Model pipelines using - [Ensembling](docs/user_guide/architecture.md#ensemble-models) or [Business - Logic Scripting - (BLS)](https://github.com/triton-inference-server/python_backend#business-logic-scripting) -- [HTTP/REST and GRPC inference - protocols](docs/customization_guide/inference_protocols.md) based on the community - developed [KServe - protocol](https://github.com/kserve/kserve/tree/master/docs/predict-api/v2) -- A [C API](docs/customization_guide/inference_protocols.md#in-process-triton-server-api) and - [Java API](docs/customization_guide/inference_protocols.md#java-bindings-for-in-process-triton-server-api) - allow Triton to link directly into your application for edge and other in-process use cases -- [Metrics](docs/user_guide/metrics.md) indicating GPU utilization, server - throughput, server latency, and more - -**New to Triton Inference Server?** Make use of -[these tutorials](https://github.com/triton-inference-server/tutorials) -to begin your Triton journey! - -Join the [Triton and TensorRT community](https://www.nvidia.com/en-us/deep-learning-ai/triton-tensorrt-newsletter/) and -stay current on the latest product updates, bug fixes, content, best practices, -and more. Need enterprise support? NVIDIA global support is available for Triton -Inference Server with the -[NVIDIA AI Enterprise software suite](https://www.nvidia.com/en-us/data-center/products/ai-enterprise/). - -## Serve a Model in 3 Easy Steps - -```bash -# Step 1: Create the example model repository -git clone -b r25.05 https://github.com/triton-inference-server/server.git -cd server/docs/examples -./fetch_models.sh - -# Step 2: Launch triton from the NGC Triton container -docker run --gpus=1 --rm --net=host -v ${PWD}/model_repository:/models nvcr.io/nvidia/tritonserver:25.05-py3 tritonserver --model-repository=/models --model-control-mode explicit --load-model densenet_onnx - -# Step 3: Sending an Inference Request -# In a separate console, launch the image_client example from the NGC Triton SDK container -docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:25.05-py3-sdk /workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg - -# Inference should return the following -Image '/workspace/images/mug.jpg': - 15.346230 (504) = COFFEE MUG - 13.224326 (968) = CUP - 10.422965 (505) = COFFEEPOT -``` -Please read the [QuickStart](docs/getting_started/quickstart.md) guide for additional information -regarding this example. The quickstart guide also contains an example of how to launch Triton on [CPU-only systems](docs/getting_started/quickstart.md#run-on-cpu-only-system). New to Triton and wondering where to get started? Watch the [Getting Started video](https://youtu.be/NQDtfSi5QF4). - -## Examples and Tutorials - -Check out [NVIDIA LaunchPad](https://www.nvidia.com/en-us/data-center/products/ai-enterprise-suite/trial/) -for free access to a set of hands-on labs with Triton Inference Server hosted on -NVIDIA infrastructure. - -Specific end-to-end examples for popular models, such as ResNet, BERT, and DLRM -are located in the -[NVIDIA Deep Learning Examples](https://github.com/NVIDIA/DeepLearningExamples) -page on GitHub. The -[NVIDIA Developer Zone](https://developer.nvidia.com/nvidia-triton-inference-server) -contains additional documentation, presentations, and examples. - -## Documentation - -### Build and Deploy - -The recommended way to build and use Triton Inference Server is with Docker -images. - -- [Install Triton Inference Server with Docker containers](docs/customization_guide/build.md#building-with-docker) (*Recommended*) -- [Install Triton Inference Server without Docker containers](docs/customization_guide/build.md#building-without-docker) -- [Build a custom Triton Inference Server Docker container](docs/customization_guide/compose.md) -- [Build Triton Inference Server from source](docs/customization_guide/build.md#building-on-unsupported-platforms) -- [Build Triton Inference Server for Windows 10](docs/customization_guide/build.md#building-for-windows-10) -- Examples for deploying Triton Inference Server with Kubernetes and Helm on [GCP](deploy/gcp/README.md), - [AWS](deploy/aws/README.md), and [NVIDIA FleetCommand](deploy/fleetcommand/README.md) -- [Secure Deployment Considerations](docs/customization_guide/deploy.md) - -### Using Triton - -#### Preparing Models for Triton Inference Server - -The first step in using Triton to serve your models is to place one or -more models into a [model repository](docs/user_guide/model_repository.md). Depending on -the type of the model and on what Triton capabilities you want to enable for -the model, you may need to create a [model -configuration](docs/user_guide/model_configuration.md) for the model. - -- [Add custom operations to Triton if needed by your model](docs/user_guide/custom_operations.md) -- Enable model pipelining with [Model Ensemble](docs/user_guide/architecture.md#ensemble-models) - and [Business Logic Scripting (BLS)](https://github.com/triton-inference-server/python_backend#business-logic-scripting) -- Optimize your models setting [scheduling and batching](docs/user_guide/architecture.md#models-and-schedulers) - parameters and [model instances](docs/user_guide/model_configuration.md#instance-groups). -- Use the [Model Analyzer tool](https://github.com/triton-inference-server/model_analyzer) - to help optimize your model configuration with profiling -- Learn how to [explicitly manage what models are available by loading and - unloading models](docs/user_guide/model_management.md) - -#### Configure and Use Triton Inference Server - -- Read the [Quick Start Guide](docs/getting_started/quickstart.md) to run Triton Inference - Server on both GPU and CPU -- Triton supports multiple execution engines, called - [backends](https://github.com/triton-inference-server/backend#where-can-i-find-all-the-backends-that-are-available-for-triton), including - [TensorRT](https://github.com/triton-inference-server/tensorrt_backend), - [TensorFlow](https://github.com/triton-inference-server/tensorflow_backend), - [PyTorch](https://github.com/triton-inference-server/pytorch_backend), - [ONNX](https://github.com/triton-inference-server/onnxruntime_backend), - [OpenVINO](https://github.com/triton-inference-server/openvino_backend), - [Python](https://github.com/triton-inference-server/python_backend), and more -- Not all the above backends are supported on every platform supported by Triton. - Look at the - [Backend-Platform Support Matrix](https://github.com/triton-inference-server/backend/blob/main/docs/backend_platform_support_matrix.md) - to learn which backends are supported on your target platform. -- Learn how to [optimize performance](docs/user_guide/optimization.md) using the - [Performance Analyzer](https://github.com/triton-inference-server/perf_analyzer/blob/main/README.md) - and - [Model Analyzer](https://github.com/triton-inference-server/model_analyzer) -- Learn how to [manage loading and unloading models](docs/user_guide/model_management.md) in - Triton -- Send requests directly to Triton with the [HTTP/REST JSON-based - or gRPC protocols](docs/customization_guide/inference_protocols.md#httprest-and-grpc-protocols) - -#### Client Support and Examples - -A Triton *client* application sends inference and other requests to Triton. The -[Python and C++ client libraries](https://github.com/triton-inference-server/client) -provide APIs to simplify this communication. - -- Review client examples for [C++](https://github.com/triton-inference-server/client/blob/main/src/c%2B%2B/examples), - [Python](https://github.com/triton-inference-server/client/blob/main/src/python/examples), - and [Java](https://github.com/triton-inference-server/client/blob/main/src/java/src/main/java/triton/client/examples) -- Configure [HTTP](https://github.com/triton-inference-server/client#http-options) - and [gRPC](https://github.com/triton-inference-server/client#grpc-options) - client options -- Send input data (e.g. a jpeg image) directly to Triton in the [body of an HTTP - request without any additional metadata](https://github.com/triton-inference-server/server/blob/main/docs/protocol/extension_binary_data.md#raw-binary-request) - -### Extend Triton - -[Triton Inference Server's architecture](docs/user_guide/architecture.md) is specifically -designed for modularity and flexibility - -- [Customize Triton Inference Server container](docs/customization_guide/compose.md) for your use case -- [Create custom backends](https://github.com/triton-inference-server/backend) - in either [C/C++](https://github.com/triton-inference-server/backend/blob/main/README.md#triton-backend-api) - or [Python](https://github.com/triton-inference-server/python_backend) -- Create [decoupled backends and models](docs/user_guide/decoupled_models.md) that can send - multiple responses for a request or not send any responses for a request -- Use a [Triton repository agent](docs/customization_guide/repository_agents.md) to add functionality - that operates when a model is loaded and unloaded, such as authentication, - decryption, or conversion -- Deploy Triton on [Jetson and JetPack](docs/user_guide/jetson.md) -- [Use Triton on AWS - Inferentia](https://github.com/triton-inference-server/python_backend/tree/main/inferentia) - -### Additional Documentation - -- [FAQ](docs/user_guide/faq.md) -- [User Guide](docs/README.md#user-guide) -- [Customization Guide](docs/README.md#customization-guide) -- [Release Notes](https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/index.html) -- [GPU, Driver, and CUDA Support -Matrix](https://docs.nvidia.com/deeplearning/dgx/support-matrix/index.html) - -## Contributing - -Contributions to Triton Inference Server are more than welcome. To -contribute please review the [contribution -guidelines](CONTRIBUTING.md). If you have a backend, client, -example or similar contribution that is not modifying the core of -Triton, then you should file a PR in the [contrib -repo](https://github.com/triton-inference-server/contrib). - -## Reporting problems, asking questions - -We appreciate any feedback, questions or bug reporting regarding this project. -When posting [issues in GitHub](https://github.com/triton-inference-server/server/issues), -follow the process outlined in the [Stack Overflow document](https://stackoverflow.com/help/mcve). -Ensure posted examples are: -- minimal – use as little code as possible that still produces the - same problem -- complete – provide all parts needed to reproduce the problem. Check - if you can strip external dependencies and still show the problem. The - less time we spend on reproducing problems the more time we have to - fix it -- verifiable – test the code you're about to provide to make sure it - reproduces the problem. Remove all other problems that are not - related to your request/question. - -For issues, please use the provided bug report and feature request templates. - -For questions, we recommend posting in our community -[GitHub Discussions.](https://github.com/triton-inference-server/server/discussions) - -## For more information - -Please refer to the [NVIDIA Developer Triton page](https://developer.nvidia.com/nvidia-triton-inference-server) -for more information. +> You are currently on the `r25.06` branch which tracks under-development progress towards the next release. diff --git a/TRITON_VERSION b/TRITON_VERSION index 6d43c77ac3..e7a9ca6a62 100644 --- a/TRITON_VERSION +++ b/TRITON_VERSION @@ -1 +1 @@ -2.59.0dev +2.59.0 diff --git a/build.py b/build.py index 150719ccc0..5ced5327f2 100755 --- a/build.py +++ b/build.py @@ -71,14 +71,14 @@ # DEFAULT_TRITON_VERSION_MAP = { - "release_version": "2.58.0dev", - "triton_container_version": "25.05dev", - "upstream_container_version": "25.05", + "release_version": "2.59.0", + "triton_container_version": "25.06", + "upstream_container_version": "25.06", "ort_version": "1.22.0", "ort_openvino_version": "2025.1.0", "standalone_openvino_version": "2025.1.0", "dcgm_version": "3.3.6", - "vllm_version": "0.8.4", + "vllm_version": "0.9.0.1", "rhel_py_version": "3.12.3", } diff --git a/deploy/aws/values.yaml b/deploy/aws/values.yaml index cddaaaf3a6..c24b0ed7b8 100644 --- a/deploy/aws/values.yaml +++ b/deploy/aws/values.yaml @@ -27,7 +27,7 @@ replicaCount: 1 image: - imageName: nvcr.io/nvidia/tritonserver:25.05-py3 + imageName: nvcr.io/nvidia/tritonserver:25.06-py3 pullPolicy: IfNotPresent modelRepositoryPath: s3://triton-inference-server-repository/model_repository numGpus: 1 diff --git a/deploy/fleetcommand/values.yaml b/deploy/fleetcommand/values.yaml index b4044b795a..d2300ef421 100644 --- a/deploy/fleetcommand/values.yaml +++ b/deploy/fleetcommand/values.yaml @@ -27,7 +27,7 @@ replicaCount: 1 image: - imageName: nvcr.io/nvidia/tritonserver:25.05-py3 + imageName: nvcr.io/nvidia/tritonserver:25.06-py3 pullPolicy: IfNotPresent numGpus: 1 serverCommand: tritonserver @@ -47,13 +47,13 @@ image: # # To set model control mode, uncomment and configure below # TODO: Fix the following url, it is invalid - # See https://github.com/triton-inference-server/server/blob/r25.05/docs/user_guide/model_management.md + # See https://github.com/triton-inference-server/server/blob/r25.06/docs/user_guide/model_management.md # for more details #- --model-control-mode=explicit|poll|none # # Additional server args # - # see https://github.com/triton-inference-server/server/blob/r25.05/README.md + # see https://github.com/triton-inference-server/server/blob/r25.06/README.md # for more details service: diff --git a/deploy/gcp/values.yaml b/deploy/gcp/values.yaml index 3770934cc8..73948fb39d 100644 --- a/deploy/gcp/values.yaml +++ b/deploy/gcp/values.yaml @@ -27,7 +27,7 @@ replicaCount: 1 image: - imageName: nvcr.io/nvidia/tritonserver:25.05-py3 + imageName: nvcr.io/nvidia/tritonserver:25.06-py3 pullPolicy: IfNotPresent modelRepositoryPath: gs://triton-inference-server-repository/model_repository numGpus: 1 diff --git a/deploy/gke-marketplace-app/benchmark/perf-analyzer-script/triton_client.yaml b/deploy/gke-marketplace-app/benchmark/perf-analyzer-script/triton_client.yaml index 38a65ed8e8..8be23e0f2c 100644 --- a/deploy/gke-marketplace-app/benchmark/perf-analyzer-script/triton_client.yaml +++ b/deploy/gke-marketplace-app/benchmark/perf-analyzer-script/triton_client.yaml @@ -33,7 +33,7 @@ metadata: namespace: default spec: containers: - - image: nvcr.io/nvidia/tritonserver:25.05-py3-sdk + - image: nvcr.io/nvidia/tritonserver:25.06-py3-sdk imagePullPolicy: Always name: nv-triton-client securityContext: diff --git a/deploy/gke-marketplace-app/server-deployer/build_and_push.sh b/deploy/gke-marketplace-app/server-deployer/build_and_push.sh index 6cb6f76e0a..b31804b9a2 100755 --- a/deploy/gke-marketplace-app/server-deployer/build_and_push.sh +++ b/deploy/gke-marketplace-app/server-deployer/build_and_push.sh @@ -27,9 +27,9 @@ export REGISTRY=gcr.io/$(gcloud config get-value project | tr ':' '/') export APP_NAME=tritonserver -export MAJOR_VERSION=2.58 -export MINOR_VERSION=2.58.0 -export NGC_VERSION=25.05-py3 +export MAJOR_VERSION=2.59 +export MINOR_VERSION=2.59.0 +export NGC_VERSION=25.06-py3 docker pull nvcr.io/nvidia/$APP_NAME:$NGC_VERSION diff --git a/deploy/gke-marketplace-app/server-deployer/chart/triton/values.yaml b/deploy/gke-marketplace-app/server-deployer/chart/triton/values.yaml index ed69180192..52b1ab2f21 100644 --- a/deploy/gke-marketplace-app/server-deployer/chart/triton/values.yaml +++ b/deploy/gke-marketplace-app/server-deployer/chart/triton/values.yaml @@ -31,14 +31,14 @@ maxReplicaCount: 3 tritonProtocol: HTTP # HPA GPU utilization autoscaling target HPATargetAverageValue: 85 -modelRepositoryPath: gs://triton_sample_models/25.05 -publishedVersion: '2.58.0' +modelRepositoryPath: gs://triton_sample_models/25.06 +publishedVersion: '2.59.0' gcpMarketplace: true image: registry: gcr.io repository: nvidia-ngc-public/tritonserver - tag: 25.05-py3 + tag: 25.06-py3 pullPolicy: IfNotPresent # modify the model repository here to match your GCP storage bucket numGpus: 1 diff --git a/deploy/gke-marketplace-app/server-deployer/schema.yaml b/deploy/gke-marketplace-app/server-deployer/schema.yaml index 68c171d28a..acaafab408 100644 --- a/deploy/gke-marketplace-app/server-deployer/schema.yaml +++ b/deploy/gke-marketplace-app/server-deployer/schema.yaml @@ -89,7 +89,7 @@ properties: modelRepositoryPath: type: string title: Bucket where models are stored. Please make sure the user/service account to create the GKE app has permission to this GCS bucket. Read Triton documentation on configs and formatting details, supporting TensorRT, TensorFlow, Pytorch, Onnx ... etc. - default: gs://triton_sample_models/25.05 + default: gs://triton_sample_models/25.06 image.ldPreloadPath: type: string title: Leave this empty by default. Triton allows users to create custom layers for backend such as TensorRT plugin or Tensorflow custom ops, the compiled shared library must be provided via LD_PRELOAD environment variable. diff --git a/deploy/gke-marketplace-app/trt-engine/README.md b/deploy/gke-marketplace-app/trt-engine/README.md index 01475e551c..dba63cf63e 100644 --- a/deploy/gke-marketplace-app/trt-engine/README.md +++ b/deploy/gke-marketplace-app/trt-engine/README.md @@ -33,7 +33,7 @@ ``` docker run --gpus all -it --network host \ --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 \ - -v ~:/scripts nvcr.io/nvidia/tensorrt:25.05-py3 + -v ~:/scripts nvcr.io/nvidia/tensorrt:25.06-py3 pip install onnx six torch tf2onnx tensorflow @@ -57,7 +57,7 @@ mkdir -p engines python3 builder.py -m models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/model.ckpt -o engines/bert_large_int8_bs1_s128.engine -b 1 -s 128 -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/ -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/vocab.txt --int8 --fp16 --strict --calib-num 1 -iln -imh -gsutil cp bert_large_int8_bs1_s128.engine gs://triton_sample_models/25.05/bert/1/model.plan +gsutil cp bert_large_int8_bs1_s128.engine gs://triton_sample_models/25.06/bert/1/model.plan ``` -For each Triton upgrade, container version used to generate the model, and the model path in GCS `gs://triton_sample_models/25.05/` should be updated accordingly with the correct version. +For each Triton upgrade, container version used to generate the model, and the model path in GCS `gs://triton_sample_models/25.06/` should be updated accordingly with the correct version. diff --git a/deploy/k8s-onprem/values.yaml b/deploy/k8s-onprem/values.yaml index 27cb9bf33b..ac6d7a3480 100644 --- a/deploy/k8s-onprem/values.yaml +++ b/deploy/k8s-onprem/values.yaml @@ -30,7 +30,7 @@ tags: openshift: false image: - imageName: nvcr.io/nvidia/tritonserver:25.05-py3 + imageName: nvcr.io/nvidia/tritonserver:25.06-py3 pullPolicy: IfNotPresent modelRepositoryServer: < Replace with the IP Address of your file server > modelRepositoryPath: /srv/models diff --git a/deploy/oci/values.yaml b/deploy/oci/values.yaml index b082864f3e..be6eb13a0e 100644 --- a/deploy/oci/values.yaml +++ b/deploy/oci/values.yaml @@ -27,7 +27,7 @@ replicaCount: 1 image: - imageName: nvcr.io/nvidia/tritonserver:25.05-py3 + imageName: nvcr.io/nvidia/tritonserver:25.06-py3 pullPolicy: IfNotPresent modelRepositoryPath: s3://https://.compat.objectstorage..oraclecloud.com:443/triton-inference-server-repository numGpus: 1 diff --git a/docs/introduction/compatibility.md b/docs/introduction/compatibility.md index afa4ab0ac8..4332d4c8ae 100644 --- a/docs/introduction/compatibility.md +++ b/docs/introduction/compatibility.md @@ -38,20 +38,21 @@ | Triton release version | NGC Tag | Python version | Torch version | TensorRT version | TensorRT-LLM version | CUDA version | CUDA Driver version | Size | | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 25.06 | nvcr.io/nvidia/tritonserver:25.06-trtllm-python-py3 | Python 3.12.3 | 2.7.0a0+79aa17489c.nv25.4 | 10.10.0.31 | 0.20.0 | 12.9.0.036 | 575.51.03 | 18.3G | | 25.05 | nvcr.io/nvidia/tritonserver:25.05-trtllm-python-py3 | Python 3.12.3 | 2.7.0a0+7c8ec84dab.nv25.3 | 10.9.0.34 | 0.19.0 | 12.8.1.012 | 570.124.06 | 17G | | 25.04 | nvcr.io/nvidia/tritonserver:25.04-trtllm-python-py3 | Python 3.12.3 | 2.7.0a0+7c8ec84dab.nv25.3 | 10.9.0.34 | 0.18.2 | 12.8.1.012 | 570.124.06 | 17G | -| 25.03 | nvcr.io/nvidia/tritonserver:25.03-trtllm-python-py3 | Python 3.12.3 | 2.7.0a0%2B7c8ec84dab.nv25.3 | 10.9.0.34 | 0.18.0 | 12.8.1.012 | 570.124.06 | 28G | -| 25.02 | nvcr.io/nvidia/tritonserver:25.02-trtllm-python-py3 | Python 3.12.3 | 2.6.0a0%2Becf3bae40a.nv25.1 | 10.8.0.43 | 0.17.0.post1 | 12.8.0.038 | 570.86.10 | 28G | -| 25.01 | nvcr.io/nvidia/tritonserver:25.01-trtllm-python-py3 | Python 3.12.3 | 2.6.0a0%2Becf3bae40a.nv25.1 | 10.8.0.43 | 0.17.0 | 12.8.0.038 | 570.86.10 | 30G | -| 24.12 | nvcr.io/nvidia/tritonserver:24.12-trtllm-python-py3 | Python 3.12.3 | 2.6.0a0%2Bdf5bbc09d1.nv24.11 | 10.7.0 | 0.16.0 | 12.6.3 | 560.35.05 | 22G | -| 24.11 | nvcr.io/nvidia/tritonserver:24.11-trtllm-python-py3 | Python 3.10.12 | 2.5.0a0%2Be000cf0ad9.nv24.10 | 10.6.0 | 0.15.0 | 12.6.3 | 555.42.06 | 24.8G | -| 24.10 | nvcr.io/nvidia/tritonserver:24.10-trtllm-python-py3 | Python 3.10.12 | 2.4.0a0%2B3bcc3cddb5.nv24.7 | 10.4.0 | 0.14.0 | 12.5.1.007 | 555.42.06 | 23.3G | -| 24.09 | nvcr.io/nvidia/tritonserver:24.09-trtllm-python-py3 | Python 3.10.12 | 2.4.0a0%2B3bcc3cddb5.nv24.7 | 10.4.0 | 0.13.0 | 12.5.1.007 | 555.42.06 | 21G | -| 24.08 | nvcr.io/nvidia/tritonserver:24.08-trtllm-python-py3 | Python 3.10.12 | 2.4.0a0%2B3bcc3cddb5.nv24.7 | 10.3.0 | 0.12.0 | 12.5.1.007 | 555.42.06 | 21G | -| 24.07 | nvcr.io/nvidia/tritonserver:24.07-trtllm-python-py3 | Python 3.10.12 | 2.4.0a0%2B07cecf4168.nv24.5 | 10.1.0 | 0.11.0 | 12.4.1.003 | 550.54.15 | 23G | -| 24.06 | nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3 | Python 3.10.12 | 2.3.0a0%2B40ec155e58.nv24.3 | 10.0.1 | 0.10.0 | 12.4.0.041 | 550.54.14 | 31G | -| 24.05 | nvcr.io/nvidia/tritonserver:24.05-trtllm-python-py3 | Python 3.10.12 | 2.3.0a0%2Bebedce2 | 10.0.1.6 | 0.9.0 | 12.3.2.001 | 545.23.08 | 34G | -| 24.04 | nvcr.io/nvidia/tritonserver:24.04-trtllm-python-py3 | Python 3.10.12 | 2.3.0a0%2Bebedce2 | 9.3.0.post12.dev1 | 0.9.0 | 12.3.2.001 | 545.23.08 | 34G | +| 25.03 | nvcr.io/nvidia/tritonserver:25.03-trtllm-python-py3 | Python 3.12.3 | 2.7.0a0+7c8ec84dab.nv25.3 | 10.9.0.34 | 0.18.0 | 12.8.1.012 | 570.124.06 | 28G | +| 25.02 | nvcr.io/nvidia/tritonserver:25.02-trtllm-python-py3 | Python 3.12.3 | 2.6.0a0+ecf3bae40a.nv25.1 | 10.8.0.43 | 0.17.0.post1 | 12.8.0.038 | 570.86.10 | 28G | +| 25.01 | nvcr.io/nvidia/tritonserver:25.01-trtllm-python-py3 | Python 3.12.3 | 2.6.0a0+ecf3bae40a.nv25.1 | 10.8.0.43 | 0.17.0 | 12.8.0.038 | 570.86.10 | 30G | +| 24.12 | nvcr.io/nvidia/tritonserver:24.12-trtllm-python-py3 | Python 3.12.3 | 2.6.0a0+df5bbc09d1.nv24.11 | 10.7.0 | 0.16.0 | 12.6.3 | 560.35.05 | 22G | +| 24.11 | nvcr.io/nvidia/tritonserver:24.11-trtllm-python-py3 | Python 3.10.12 | 2.5.0a0+e000cf0ad9.nv24.10 | 10.6.0 | 0.15.0 | 12.6.3 | 555.42.06 | 24.8G | +| 24.10 | nvcr.io/nvidia/tritonserver:24.10-trtllm-python-py3 | Python 3.10.12 | 2.4.0a0+3bcc3cddb5.nv24.7 | 10.4.0 | 0.14.0 | 12.5.1.007 | 555.42.06 | 23.3G | +| 24.09 | nvcr.io/nvidia/tritonserver:24.09-trtllm-python-py3 | Python 3.10.12 | 2.4.0a0+3bcc3cddb5.nv24.7 | 10.4.0 | 0.13.0 | 12.5.1.007 | 555.42.06 | 21G | +| 24.08 | nvcr.io/nvidia/tritonserver:24.08-trtllm-python-py3 | Python 3.10.12 | 2.4.0a0+3bcc3cddb5.nv24.7 | 10.3.0 | 0.12.0 | 12.5.1.007 | 555.42.06 | 21G | +| 24.07 | nvcr.io/nvidia/tritonserver:24.07-trtllm-python-py3 | Python 3.10.12 | 2.4.0a0+07cecf4168.nv24.5 | 10.1.0 | 0.11.0 | 12.4.1.003 | 550.54.15 | 23G | +| 24.06 | nvcr.io/nvidia/tritonserver:24.06-trtllm-python-py3 | Python 3.10.12 | 2.3.0a0+40ec155e58.nv24.3 | 10.0.1 | 0.10.0 | 12.4.0.041 | 550.54.14 | 31G | +| 24.05 | nvcr.io/nvidia/tritonserver:24.05-trtllm-python-py3 | Python 3.10.12 | 2.3.0a0+ebedce2 | 10.0.1.6 | 0.9.0 | 12.3.2.001 | 545.23.08 | 34G | +| 24.04 | nvcr.io/nvidia/tritonserver:24.04-trtllm-python-py3 | Python 3.10.12 | 2.3.0a0+ebedce2 | 9.3.0.post12.dev1 | 0.9.0 | 12.3.2.001 | 545.23.08 | 34G | ## Container Name: vllm-python-py3 diff --git a/docs/introduction/release_notes.md b/docs/introduction/release_notes.md index f297ec3ac2..7038abbb01 100644 --- a/docs/introduction/release_notes.md +++ b/docs/introduction/release_notes.md @@ -25,9 +25,9 @@ # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. --> -# [Triton Inference Server Release 25.05](https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/rel-25-05.html#rel-25-05) +# [Triton Inference Server Release 25.06](https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/rel-25-06.html#rel-25-06) -The Triton Inference Server container image, release 25.05, is available +The Triton Inference Server container image, release 25.06, is available on [NGC](https://ngc.nvidia.com/catalog/containers/nvidia:tritonserver) and is open source on [GitHub](https://github.com/triton-inference-server/server). Release notes can diff --git a/python/openai/README.md b/python/openai/README.md index 51d8c5db72..5851918ff2 100644 --- a/python/openai/README.md +++ b/python/openai/README.md @@ -51,7 +51,7 @@ docker run -it --net=host --gpus all --rm \ -v ${HOME}/.cache/huggingface:/root/.cache/huggingface \ -e HF_TOKEN \ - nvcr.io/nvidia/tritonserver:25.05-vllm-python-py3 + nvcr.io/nvidia/tritonserver:25.06-vllm-python-py3 ``` 2. Launch the OpenAI-compatible Triton Inference Server: diff --git a/qa/common/gen_jetson_trt_models b/qa/common/gen_jetson_trt_models index 8726e7c271..b081fe9c4e 100755 --- a/qa/common/gen_jetson_trt_models +++ b/qa/common/gen_jetson_trt_models @@ -34,7 +34,7 @@ # Make all generated files accessible outside of container umask 0000 # Set the version of the models -TRITON_VERSION=${TRITON_VERSION:=25.05} +TRITON_VERSION=${TRITON_VERSION:=25.06} # Set the CUDA device to use CUDA_DEVICE=${RUNNER_ID:=0} # Set TensorRT image diff --git a/qa/common/gen_qa_custom_ops b/qa/common/gen_qa_custom_ops index 96801e4470..108b1daa53 100755 --- a/qa/common/gen_qa_custom_ops +++ b/qa/common/gen_qa_custom_ops @@ -37,7 +37,7 @@ ## ############################################################################ -TRITON_VERSION=${TRITON_VERSION:=25.05} +TRITON_VERSION=${TRITON_VERSION:=25.06} NVIDIA_UPSTREAM_VERSION=${NVIDIA_UPSTREAM_VERSION:=$TRITON_VERSION} PYTORCH_IMAGE=${PYTORCH_IMAGE:=nvcr.io/nvidia/pytorch:$NVIDIA_UPSTREAM_VERSION-py3} UBUNTU_IMAGE=${UBUNTU_IMAGE:=ubuntu:24.04} diff --git a/qa/common/gen_qa_model_repository b/qa/common/gen_qa_model_repository index 7f4741d1c7..2deeb97da5 100755 --- a/qa/common/gen_qa_model_repository +++ b/qa/common/gen_qa_model_repository @@ -48,7 +48,7 @@ ## ############################################################################ -TRITON_VERSION=${TRITON_VERSION:=25.05} +TRITON_VERSION=${TRITON_VERSION:=25.06} # ONNX. Use ONNX_OPSET 0 to use the default for ONNX version ONNX_VERSION=1.16.1 From 1d430f6e999e64c733508a7c78589b3244c921b1 Mon Sep 17 00:00:00 2001 From: Yingge He <157551214+yinggeh@users.noreply.github.com> Date: Wed, 18 Jun 2025 13:35:42 -0700 Subject: [PATCH 02/14] ci: Fix onnx_model_store resnet_v1 name issues (#8255) Co-authored-by: Misha Chornyi <99709299+mc-nv@users.noreply.github.com> --- qa/L0_long_running_stress/scenarios.py | 10 +++++----- qa/L0_long_running_stress/test.sh | 5 ++--- 2 files changed, 7 insertions(+), 8 deletions(-) diff --git a/qa/L0_long_running_stress/scenarios.py b/qa/L0_long_running_stress/scenarios.py index 650f228ab5..5b952f5b97 100755 --- a/qa/L0_long_running_stress/scenarios.py +++ b/qa/L0_long_running_stress/scenarios.py @@ -203,7 +203,7 @@ def __init__( # Add no validation models self.options_.append( PerfAnalyzerScenario.ModelOption( - "resnet_v1_50_def", 32, (1, 4, 1), queue_latency_range_us + "resnet_v1_50", 32, (1, 4, 1), queue_latency_range_us ) ) for trial in sequence_trials: @@ -334,7 +334,7 @@ def run(self, client_metadata): class ResNetScenario(Scenario): def __init__(self, name, batch_size=32, verbose=False, out_stream=sys.stdout): super().__init__(name, [], verbose, out_stream) - self.model_name_ = "resnet_v1_50_def" + self.model_name_ = "resnet_v1_50" self.batch_size_ = batch_size img = self.preprocess("../images/vulture.jpeg") @@ -353,7 +353,7 @@ def preprocess(self, filename): return scaled def postprocess(self, results): - output_array = results.as_numpy("resnet_v1_50/predictions/Softmax") + output_array = results.as_numpy("resnet_v1_50/predictions/Softmax:0") if len(output_array) != self.batch_size_: raise Exception( "expected {} results, got {}".format( @@ -377,12 +377,12 @@ def postprocess(self, results): def run(self, client_metadata): triton_client = client_metadata[0] - inputs = [grpcclient.InferInput("input", self.image_data_.shape, "FP32")] + inputs = [grpcclient.InferInput("input:0", self.image_data_.shape, "FP32")] inputs[0].set_data_from_numpy(self.image_data_) outputs = [ grpcclient.InferRequestedOutput( - "resnet_v1_50/predictions/Softmax", class_count=1 + "resnet_v1_50/predictions/Softmax:0", class_count=1 ) ] res = triton_client.infer(self.model_name_, inputs, outputs=outputs) diff --git a/qa/L0_long_running_stress/test.sh b/qa/L0_long_running_stress/test.sh index 1973028bcf..92d8c09b0d 100755 --- a/qa/L0_long_running_stress/test.sh +++ b/qa/L0_long_running_stress/test.sh @@ -130,9 +130,8 @@ cp -r ../custom_models/custom_zero_1_float32 $MODEL_DIR/custom_zero_1_float32 && echo "{ key: \"execute_delay_ms\"; value: { string_value: \"10000\" }}" >> config.pbtxt && \ echo "]" >> config.pbtxt) -cp -r $DATADIR/onnx_model_store/resnet_v1_50 $MODEL_DIR/resnet_v1_50_def && \ - (cd $MODEL_DIR/resnet_v1_50_def && \ - sed -i 's/^name: "resnet_v1_50"/name: "resnet_v1_50_def"/' config.pbtxt && \ +cp -r $DATADIR/onnx_model_store/resnet_v1_50 $MODEL_DIR/. && \ + (cd $MODEL_DIR/resnet_v1_50 && \ echo "optimization { }" >> config.pbtxt) SERVER_ARGS="--model-repository=`pwd`/$MODEL_DIR" From 64c3cd2461d1730be3b13f72f8f117420deb1504 Mon Sep 17 00:00:00 2001 From: Misha Chornyi <99709299+mc-nv@users.noreply.github.com> Date: Wed, 18 Jun 2025 13:38:04 -0700 Subject: [PATCH 03/14] Update versions (#8260) --- deploy/fleetcommand/Chart.yaml | 2 +- .../server-deployer/chart/triton/Chart.yaml | 4 ++-- .../gke-marketplace-app/server-deployer/data-test/schema.yaml | 2 +- deploy/gke-marketplace-app/server-deployer/schema.yaml | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/deploy/fleetcommand/Chart.yaml b/deploy/fleetcommand/Chart.yaml index 0664a5d959..b819cbb4a5 100644 --- a/deploy/fleetcommand/Chart.yaml +++ b/deploy/fleetcommand/Chart.yaml @@ -26,7 +26,7 @@ apiVersion: v1 # appVersion is the Triton version; update when changing release -appVersion: "2.58.0" +appVersion: "2.59.0" description: Triton Inference Server (Fleet Command) name: triton-inference-server # version is the Chart version; update when changing anything in the chart diff --git a/deploy/gke-marketplace-app/server-deployer/chart/triton/Chart.yaml b/deploy/gke-marketplace-app/server-deployer/chart/triton/Chart.yaml index 5011733bbc..3206248935 100644 --- a/deploy/gke-marketplace-app/server-deployer/chart/triton/Chart.yaml +++ b/deploy/gke-marketplace-app/server-deployer/chart/triton/Chart.yaml @@ -25,7 +25,7 @@ # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. apiVersion: v1 -appVersion: "2.58" +appVersion: "2.59" description: Triton Inference Server name: triton-inference-server -version: 2.58.0 +version: 2.59.0 diff --git a/deploy/gke-marketplace-app/server-deployer/data-test/schema.yaml b/deploy/gke-marketplace-app/server-deployer/data-test/schema.yaml index 2c55441213..36ba6b6a63 100644 --- a/deploy/gke-marketplace-app/server-deployer/data-test/schema.yaml +++ b/deploy/gke-marketplace-app/server-deployer/data-test/schema.yaml @@ -27,7 +27,7 @@ x-google-marketplace: schemaVersion: v2 applicationApiVersion: v1beta1 - publishedVersion: '2.58.0' + publishedVersion: '2.59.0' publishedVersionMetadata: releaseNote: >- Initial release. diff --git a/deploy/gke-marketplace-app/server-deployer/schema.yaml b/deploy/gke-marketplace-app/server-deployer/schema.yaml index acaafab408..629303952d 100644 --- a/deploy/gke-marketplace-app/server-deployer/schema.yaml +++ b/deploy/gke-marketplace-app/server-deployer/schema.yaml @@ -27,7 +27,7 @@ x-google-marketplace: schemaVersion: v2 applicationApiVersion: v1beta1 - publishedVersion: '2.58.0' + publishedVersion: '2.59.0' publishedVersionMetadata: releaseNote: >- Initial release. From 545094c796c9918e85ef57c6cad47d1a6658a8f0 Mon Sep 17 00:00:00 2001 From: Yingge He <157551214+yinggeh@users.noreply.github.com> Date: Wed, 18 Jun 2025 14:00:35 -0700 Subject: [PATCH 04/14] ci: Fix L0_client_nobatch--base (#8259) --- qa/L0_client_nobatch/test.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qa/L0_client_nobatch/test.sh b/qa/L0_client_nobatch/test.sh index cb1a05d660..0c01bafb70 100755 --- a/qa/L0_client_nobatch/test.sh +++ b/qa/L0_client_nobatch/test.sh @@ -47,7 +47,7 @@ EXPECTED_NUM_TESTS="4" DATADIR=/data/inferenceserver/${REPO_VERSION} MODELDIR="${PWD}/qa_model_repository" -rm -rf ${MODELDIR} && cp -r "${DATADIR}/qa_model_repository/onnx_*" ${MODELDIR} # Note there is a coupling in ./client_test.py +rm -rf ${MODELDIR} && mkdir -p ${MODELDIR} && cp -r ${DATADIR}/qa_model_repository/onnx_* ${MODELDIR}/. # Note there is a coupling in ./client_test.py SERVER=/opt/tritonserver/bin/tritonserver SERVER_ARGS="--model-repository=${MODELDIR}" SERVER_LOG="./inference_server.log" From 624ac90d2f018de0233ae00af47551e37293d211 Mon Sep 17 00:00:00 2001 From: Misha Chornyi <99709299+mc-nv@users.noreply.github.com> Date: Thu, 26 Jun 2025 14:58:07 -0700 Subject: [PATCH 05/14] Update README.md for 2.59.0 / 25.06 release (#8268) --- README.md | 230 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 228 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 3acc072ce3..39de0a4222 100644 --- a/README.md +++ b/README.md @@ -27,5 +27,231 @@ --> [![License](https://img.shields.io/badge/License-BSD3-lightgrey.svg)](https://opensource.org/licenses/BSD-3-Clause) ->[!WARNING] -> You are currently on the `r25.06` branch which tracks under-development progress towards the next release. +# Triton Inference Server + +Triton Inference Server is an open source inference serving software that +streamlines AI inferencing. Triton enables teams to deploy any AI model from +multiple deep learning and machine learning frameworks, including TensorRT, +TensorFlow, PyTorch, ONNX, OpenVINO, Python, RAPIDS FIL, and more. Triton +Inference Server supports inference across cloud, data center, edge and embedded +devices on NVIDIA GPUs, x86 and ARM CPU, or AWS Inferentia. Triton Inference +Server delivers optimized performance for many query types, including real time, +batched, ensembles and audio/video streaming. Triton inference Server is part of +[NVIDIA AI Enterprise](https://www.nvidia.com/en-us/data-center/products/ai-enterprise/), +a software platform that accelerates the data science pipeline and streamlines +the development and deployment of production AI. + +Major features include: + +- [Supports multiple deep learning + frameworks](https://github.com/triton-inference-server/backend#where-can-i-find-all-the-backends-that-are-available-for-triton) +- [Supports multiple machine learning + frameworks](https://github.com/triton-inference-server/fil_backend) +- [Concurrent model + execution](docs/user_guide/architecture.md#concurrent-model-execution) +- [Dynamic batching](docs/user_guide/model_configuration.md#dynamic-batcher) +- [Sequence batching](docs/user_guide/model_configuration.md#sequence-batcher) and + [implicit state management](docs/user_guide/architecture.md#implicit-state-management) + for stateful models +- Provides [Backend API](https://github.com/triton-inference-server/backend) that + allows adding custom backends and pre/post processing operations +- Supports writing custom backends in python, a.k.a. + [Python-based backends.](https://github.com/triton-inference-server/backend/blob/r25.06/docs/python_based_backends.md#python-based-backends) +- Model pipelines using + [Ensembling](docs/user_guide/architecture.md#ensemble-models) or [Business + Logic Scripting + (BLS)](https://github.com/triton-inference-server/python_backend#business-logic-scripting) +- [HTTP/REST and GRPC inference + protocols](docs/customization_guide/inference_protocols.md) based on the community + developed [KServe + protocol](https://github.com/kserve/kserve/tree/master/docs/predict-api/v2) +- A [C API](docs/customization_guide/inference_protocols.md#in-process-triton-server-api) and + [Java API](docs/customization_guide/inference_protocols.md#java-bindings-for-in-process-triton-server-api) + allow Triton to link directly into your application for edge and other in-process use cases +- [Metrics](docs/user_guide/metrics.md) indicating GPU utilization, server + throughput, server latency, and more + +**New to Triton Inference Server?** Make use of +[these tutorials](https://github.com/triton-inference-server/tutorials) +to begin your Triton journey! + +Join the [Triton and TensorRT community](https://www.nvidia.com/en-us/deep-learning-ai/triton-tensorrt-newsletter/) and +stay current on the latest product updates, bug fixes, content, best practices, +and more. Need enterprise support? NVIDIA global support is available for Triton +Inference Server with the +[NVIDIA AI Enterprise software suite](https://www.nvidia.com/en-us/data-center/products/ai-enterprise/). + +## Serve a Model in 3 Easy Steps + +```bash +# Step 1: Create the example model repository +git clone -b r25.06 https://github.com/triton-inference-server/server.git +cd server/docs/examples +./fetch_models.sh + +# Step 2: Launch triton from the NGC Triton container +docker run --gpus=1 --rm --net=host -v ${PWD}/model_repository:/models nvcr.io/nvidia/tritonserver:25.06-py3 tritonserver --model-repository=/models --model-control-mode explicit --load-model densenet_onnx + +# Step 3: Sending an Inference Request +# In a separate console, launch the image_client example from the NGC Triton SDK container +docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:25.06-py3-sdk /workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg + +# Inference should return the following +Image '/workspace/images/mug.jpg': + 15.346230 (504) = COFFEE MUG + 13.224326 (968) = CUP + 10.422965 (505) = COFFEEPOT +``` +Please read the [QuickStart](docs/getting_started/quickstart.md) guide for additional information +regarding this example. The quickstart guide also contains an example of how to launch Triton on [CPU-only systems](docs/getting_started/quickstart.md#run-on-cpu-only-system). New to Triton and wondering where to get started? Watch the [Getting Started video](https://youtu.be/NQDtfSi5QF4). + +## Examples and Tutorials + +Check out [NVIDIA LaunchPad](https://www.nvidia.com/en-us/data-center/products/ai-enterprise-suite/trial/) +for free access to a set of hands-on labs with Triton Inference Server hosted on +NVIDIA infrastructure. + +Specific end-to-end examples for popular models, such as ResNet, BERT, and DLRM +are located in the +[NVIDIA Deep Learning Examples](https://github.com/NVIDIA/DeepLearningExamples) +page on GitHub. The +[NVIDIA Developer Zone](https://developer.nvidia.com/nvidia-triton-inference-server) +contains additional documentation, presentations, and examples. + +## Documentation + +### Build and Deploy + +The recommended way to build and use Triton Inference Server is with Docker +images. + +- [Install Triton Inference Server with Docker containers](docs/customization_guide/build.md#building-with-docker) (*Recommended*) +- [Install Triton Inference Server without Docker containers](docs/customization_guide/build.md#building-without-docker) +- [Build a custom Triton Inference Server Docker container](docs/customization_guide/compose.md) +- [Build Triton Inference Server from source](docs/customization_guide/build.md#building-on-unsupported-platforms) +- [Build Triton Inference Server for Windows 10](docs/customization_guide/build.md#building-for-windows-10) +- Examples for deploying Triton Inference Server with Kubernetes and Helm on [GCP](deploy/gcp/README.md), + [AWS](deploy/aws/README.md), and [NVIDIA FleetCommand](deploy/fleetcommand/README.md) +- [Secure Deployment Considerations](docs/customization_guide/deploy.md) + +### Using Triton + +#### Preparing Models for Triton Inference Server + +The first step in using Triton to serve your models is to place one or +more models into a [model repository](docs/user_guide/model_repository.md). Depending on +the type of the model and on what Triton capabilities you want to enable for +the model, you may need to create a [model +configuration](docs/user_guide/model_configuration.md) for the model. + +- [Add custom operations to Triton if needed by your model](docs/user_guide/custom_operations.md) +- Enable model pipelining with [Model Ensemble](docs/user_guide/architecture.md#ensemble-models) + and [Business Logic Scripting (BLS)](https://github.com/triton-inference-server/python_backend#business-logic-scripting) +- Optimize your models setting [scheduling and batching](docs/user_guide/architecture.md#models-and-schedulers) + parameters and [model instances](docs/user_guide/model_configuration.md#instance-groups). +- Use the [Model Analyzer tool](https://github.com/triton-inference-server/model_analyzer) + to help optimize your model configuration with profiling +- Learn how to [explicitly manage what models are available by loading and + unloading models](docs/user_guide/model_management.md) + +#### Configure and Use Triton Inference Server + +- Read the [Quick Start Guide](docs/getting_started/quickstart.md) to run Triton Inference + Server on both GPU and CPU +- Triton supports multiple execution engines, called + [backends](https://github.com/triton-inference-server/backend#where-can-i-find-all-the-backends-that-are-available-for-triton), including + [TensorRT](https://github.com/triton-inference-server/tensorrt_backend), + [TensorFlow](https://github.com/triton-inference-server/tensorflow_backend), + [PyTorch](https://github.com/triton-inference-server/pytorch_backend), + [ONNX](https://github.com/triton-inference-server/onnxruntime_backend), + [OpenVINO](https://github.com/triton-inference-server/openvino_backend), + [Python](https://github.com/triton-inference-server/python_backend), and more +- Not all the above backends are supported on every platform supported by Triton. + Look at the + [Backend-Platform Support Matrix](https://github.com/triton-inference-server/backend/blob/r25.06/docs/backend_platform_support_matrix.md) + to learn which backends are supported on your target platform. +- Learn how to [optimize performance](docs/user_guide/optimization.md) using the + [Performance Analyzer](https://github.com/triton-inference-server/perf_analyzer/blob/r25.06/README.md) + and + [Model Analyzer](https://github.com/triton-inference-server/model_analyzer) +- Learn how to [manage loading and unloading models](docs/user_guide/model_management.md) in + Triton +- Send requests directly to Triton with the [HTTP/REST JSON-based + or gRPC protocols](docs/customization_guide/inference_protocols.md#httprest-and-grpc-protocols) + +#### Client Support and Examples + +A Triton *client* application sends inference and other requests to Triton. The +[Python and C++ client libraries](https://github.com/triton-inference-server/client) +provide APIs to simplify this communication. + +- Review client examples for [C++](https://github.com/triton-inference-server/client/blob/r25.06/src/c%2B%2B/examples), + [Python](https://github.com/triton-inference-server/client/blob/r25.06/src/python/examples), + and [Java](https://github.com/triton-inference-server/client/blob/r25.06/src/java/src/main/java/triton/client/examples) +- Configure [HTTP](https://github.com/triton-inference-server/client#http-options) + and [gRPC](https://github.com/triton-inference-server/client#grpc-options) + client options +- Send input data (e.g. a jpeg image) directly to Triton in the [body of an HTTP + request without any additional metadata](https://github.com/triton-inference-server/server/blob/r25.06/docs/protocol/extension_binary_data.md#raw-binary-request) + +### Extend Triton + +[Triton Inference Server's architecture](docs/user_guide/architecture.md) is specifically +designed for modularity and flexibility + +- [Customize Triton Inference Server container](docs/customization_guide/compose.md) for your use case +- [Create custom backends](https://github.com/triton-inference-server/backend) + in either [C/C++](https://github.com/triton-inference-server/backend/blob/r25.06/README.md#triton-backend-api) + or [Python](https://github.com/triton-inference-server/python_backend) +- Create [decoupled backends and models](docs/user_guide/decoupled_models.md) that can send + multiple responses for a request or not send any responses for a request +- Use a [Triton repository agent](docs/customization_guide/repository_agents.md) to add functionality + that operates when a model is loaded and unloaded, such as authentication, + decryption, or conversion +- Deploy Triton on [Jetson and JetPack](docs/user_guide/jetson.md) +- [Use Triton on AWS + Inferentia](https://github.com/triton-inference-server/python_backend/tree/r25.06/inferentia) + +### Additional Documentation + +- [FAQ](docs/user_guide/faq.md) +- [User Guide](docs/README.md#user-guide) +- [Customization Guide](docs/README.md#customization-guide) +- [Release Notes](https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/index.html) +- [GPU, Driver, and CUDA Support +Matrix](https://docs.nvidia.com/deeplearning/dgx/support-matrix/index.html) + +## Contributing + +Contributions to Triton Inference Server are more than welcome. To +contribute please review the [contribution +guidelines](CONTRIBUTING.md). If you have a backend, client, +example or similar contribution that is not modifying the core of +Triton, then you should file a PR in the [contrib +repo](https://github.com/triton-inference-server/contrib). + +## Reporting problems, asking questions + +We appreciate any feedback, questions or bug reporting regarding this project. +When posting [issues in GitHub](https://github.com/triton-inference-server/server/issues), +follow the process outlined in the [Stack Overflow document](https://stackoverflow.com/help/mcve). +Ensure posted examples are: +- minimal – use as little code as possible that still produces the + same problem +- complete – provide all parts needed to reproduce the problem. Check + if you can strip external dependencies and still show the problem. The + less time we spend on reproducing problems the more time we have to + fix it +- verifiable – test the code you're about to provide to make sure it + reproduces the problem. Remove all other problems that are not + related to your request/question. + +For issues, please use the provided bug report and feature request templates. + +For questions, we recommend posting in our community +[GitHub Discussions.](https://github.com/triton-inference-server/server/discussions) + +## For more information + +Please refer to the [NVIDIA Developer Triton page](https://developer.nvidia.com/nvidia-triton-inference-server) +for more information. From 89888ced4b3a6ed9418b9758908510bcbc16e50b Mon Sep 17 00:00:00 2001 From: Misha Chornyi Date: Mon, 21 Jul 2025 13:44:11 -0700 Subject: [PATCH 06/14] prep: Update README.md for 25.07 --- README.md | 230 +----------------------------------------------------- 1 file changed, 2 insertions(+), 228 deletions(-) diff --git a/README.md b/README.md index 39de0a4222..4e3a4c33e4 100644 --- a/README.md +++ b/README.md @@ -27,231 +27,5 @@ --> [![License](https://img.shields.io/badge/License-BSD3-lightgrey.svg)](https://opensource.org/licenses/BSD-3-Clause) -# Triton Inference Server - -Triton Inference Server is an open source inference serving software that -streamlines AI inferencing. Triton enables teams to deploy any AI model from -multiple deep learning and machine learning frameworks, including TensorRT, -TensorFlow, PyTorch, ONNX, OpenVINO, Python, RAPIDS FIL, and more. Triton -Inference Server supports inference across cloud, data center, edge and embedded -devices on NVIDIA GPUs, x86 and ARM CPU, or AWS Inferentia. Triton Inference -Server delivers optimized performance for many query types, including real time, -batched, ensembles and audio/video streaming. Triton inference Server is part of -[NVIDIA AI Enterprise](https://www.nvidia.com/en-us/data-center/products/ai-enterprise/), -a software platform that accelerates the data science pipeline and streamlines -the development and deployment of production AI. - -Major features include: - -- [Supports multiple deep learning - frameworks](https://github.com/triton-inference-server/backend#where-can-i-find-all-the-backends-that-are-available-for-triton) -- [Supports multiple machine learning - frameworks](https://github.com/triton-inference-server/fil_backend) -- [Concurrent model - execution](docs/user_guide/architecture.md#concurrent-model-execution) -- [Dynamic batching](docs/user_guide/model_configuration.md#dynamic-batcher) -- [Sequence batching](docs/user_guide/model_configuration.md#sequence-batcher) and - [implicit state management](docs/user_guide/architecture.md#implicit-state-management) - for stateful models -- Provides [Backend API](https://github.com/triton-inference-server/backend) that - allows adding custom backends and pre/post processing operations -- Supports writing custom backends in python, a.k.a. - [Python-based backends.](https://github.com/triton-inference-server/backend/blob/r25.06/docs/python_based_backends.md#python-based-backends) -- Model pipelines using - [Ensembling](docs/user_guide/architecture.md#ensemble-models) or [Business - Logic Scripting - (BLS)](https://github.com/triton-inference-server/python_backend#business-logic-scripting) -- [HTTP/REST and GRPC inference - protocols](docs/customization_guide/inference_protocols.md) based on the community - developed [KServe - protocol](https://github.com/kserve/kserve/tree/master/docs/predict-api/v2) -- A [C API](docs/customization_guide/inference_protocols.md#in-process-triton-server-api) and - [Java API](docs/customization_guide/inference_protocols.md#java-bindings-for-in-process-triton-server-api) - allow Triton to link directly into your application for edge and other in-process use cases -- [Metrics](docs/user_guide/metrics.md) indicating GPU utilization, server - throughput, server latency, and more - -**New to Triton Inference Server?** Make use of -[these tutorials](https://github.com/triton-inference-server/tutorials) -to begin your Triton journey! - -Join the [Triton and TensorRT community](https://www.nvidia.com/en-us/deep-learning-ai/triton-tensorrt-newsletter/) and -stay current on the latest product updates, bug fixes, content, best practices, -and more. Need enterprise support? NVIDIA global support is available for Triton -Inference Server with the -[NVIDIA AI Enterprise software suite](https://www.nvidia.com/en-us/data-center/products/ai-enterprise/). - -## Serve a Model in 3 Easy Steps - -```bash -# Step 1: Create the example model repository -git clone -b r25.06 https://github.com/triton-inference-server/server.git -cd server/docs/examples -./fetch_models.sh - -# Step 2: Launch triton from the NGC Triton container -docker run --gpus=1 --rm --net=host -v ${PWD}/model_repository:/models nvcr.io/nvidia/tritonserver:25.06-py3 tritonserver --model-repository=/models --model-control-mode explicit --load-model densenet_onnx - -# Step 3: Sending an Inference Request -# In a separate console, launch the image_client example from the NGC Triton SDK container -docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:25.06-py3-sdk /workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg - -# Inference should return the following -Image '/workspace/images/mug.jpg': - 15.346230 (504) = COFFEE MUG - 13.224326 (968) = CUP - 10.422965 (505) = COFFEEPOT -``` -Please read the [QuickStart](docs/getting_started/quickstart.md) guide for additional information -regarding this example. The quickstart guide also contains an example of how to launch Triton on [CPU-only systems](docs/getting_started/quickstart.md#run-on-cpu-only-system). New to Triton and wondering where to get started? Watch the [Getting Started video](https://youtu.be/NQDtfSi5QF4). - -## Examples and Tutorials - -Check out [NVIDIA LaunchPad](https://www.nvidia.com/en-us/data-center/products/ai-enterprise-suite/trial/) -for free access to a set of hands-on labs with Triton Inference Server hosted on -NVIDIA infrastructure. - -Specific end-to-end examples for popular models, such as ResNet, BERT, and DLRM -are located in the -[NVIDIA Deep Learning Examples](https://github.com/NVIDIA/DeepLearningExamples) -page on GitHub. The -[NVIDIA Developer Zone](https://developer.nvidia.com/nvidia-triton-inference-server) -contains additional documentation, presentations, and examples. - -## Documentation - -### Build and Deploy - -The recommended way to build and use Triton Inference Server is with Docker -images. - -- [Install Triton Inference Server with Docker containers](docs/customization_guide/build.md#building-with-docker) (*Recommended*) -- [Install Triton Inference Server without Docker containers](docs/customization_guide/build.md#building-without-docker) -- [Build a custom Triton Inference Server Docker container](docs/customization_guide/compose.md) -- [Build Triton Inference Server from source](docs/customization_guide/build.md#building-on-unsupported-platforms) -- [Build Triton Inference Server for Windows 10](docs/customization_guide/build.md#building-for-windows-10) -- Examples for deploying Triton Inference Server with Kubernetes and Helm on [GCP](deploy/gcp/README.md), - [AWS](deploy/aws/README.md), and [NVIDIA FleetCommand](deploy/fleetcommand/README.md) -- [Secure Deployment Considerations](docs/customization_guide/deploy.md) - -### Using Triton - -#### Preparing Models for Triton Inference Server - -The first step in using Triton to serve your models is to place one or -more models into a [model repository](docs/user_guide/model_repository.md). Depending on -the type of the model and on what Triton capabilities you want to enable for -the model, you may need to create a [model -configuration](docs/user_guide/model_configuration.md) for the model. - -- [Add custom operations to Triton if needed by your model](docs/user_guide/custom_operations.md) -- Enable model pipelining with [Model Ensemble](docs/user_guide/architecture.md#ensemble-models) - and [Business Logic Scripting (BLS)](https://github.com/triton-inference-server/python_backend#business-logic-scripting) -- Optimize your models setting [scheduling and batching](docs/user_guide/architecture.md#models-and-schedulers) - parameters and [model instances](docs/user_guide/model_configuration.md#instance-groups). -- Use the [Model Analyzer tool](https://github.com/triton-inference-server/model_analyzer) - to help optimize your model configuration with profiling -- Learn how to [explicitly manage what models are available by loading and - unloading models](docs/user_guide/model_management.md) - -#### Configure and Use Triton Inference Server - -- Read the [Quick Start Guide](docs/getting_started/quickstart.md) to run Triton Inference - Server on both GPU and CPU -- Triton supports multiple execution engines, called - [backends](https://github.com/triton-inference-server/backend#where-can-i-find-all-the-backends-that-are-available-for-triton), including - [TensorRT](https://github.com/triton-inference-server/tensorrt_backend), - [TensorFlow](https://github.com/triton-inference-server/tensorflow_backend), - [PyTorch](https://github.com/triton-inference-server/pytorch_backend), - [ONNX](https://github.com/triton-inference-server/onnxruntime_backend), - [OpenVINO](https://github.com/triton-inference-server/openvino_backend), - [Python](https://github.com/triton-inference-server/python_backend), and more -- Not all the above backends are supported on every platform supported by Triton. - Look at the - [Backend-Platform Support Matrix](https://github.com/triton-inference-server/backend/blob/r25.06/docs/backend_platform_support_matrix.md) - to learn which backends are supported on your target platform. -- Learn how to [optimize performance](docs/user_guide/optimization.md) using the - [Performance Analyzer](https://github.com/triton-inference-server/perf_analyzer/blob/r25.06/README.md) - and - [Model Analyzer](https://github.com/triton-inference-server/model_analyzer) -- Learn how to [manage loading and unloading models](docs/user_guide/model_management.md) in - Triton -- Send requests directly to Triton with the [HTTP/REST JSON-based - or gRPC protocols](docs/customization_guide/inference_protocols.md#httprest-and-grpc-protocols) - -#### Client Support and Examples - -A Triton *client* application sends inference and other requests to Triton. The -[Python and C++ client libraries](https://github.com/triton-inference-server/client) -provide APIs to simplify this communication. - -- Review client examples for [C++](https://github.com/triton-inference-server/client/blob/r25.06/src/c%2B%2B/examples), - [Python](https://github.com/triton-inference-server/client/blob/r25.06/src/python/examples), - and [Java](https://github.com/triton-inference-server/client/blob/r25.06/src/java/src/main/java/triton/client/examples) -- Configure [HTTP](https://github.com/triton-inference-server/client#http-options) - and [gRPC](https://github.com/triton-inference-server/client#grpc-options) - client options -- Send input data (e.g. a jpeg image) directly to Triton in the [body of an HTTP - request without any additional metadata](https://github.com/triton-inference-server/server/blob/r25.06/docs/protocol/extension_binary_data.md#raw-binary-request) - -### Extend Triton - -[Triton Inference Server's architecture](docs/user_guide/architecture.md) is specifically -designed for modularity and flexibility - -- [Customize Triton Inference Server container](docs/customization_guide/compose.md) for your use case -- [Create custom backends](https://github.com/triton-inference-server/backend) - in either [C/C++](https://github.com/triton-inference-server/backend/blob/r25.06/README.md#triton-backend-api) - or [Python](https://github.com/triton-inference-server/python_backend) -- Create [decoupled backends and models](docs/user_guide/decoupled_models.md) that can send - multiple responses for a request or not send any responses for a request -- Use a [Triton repository agent](docs/customization_guide/repository_agents.md) to add functionality - that operates when a model is loaded and unloaded, such as authentication, - decryption, or conversion -- Deploy Triton on [Jetson and JetPack](docs/user_guide/jetson.md) -- [Use Triton on AWS - Inferentia](https://github.com/triton-inference-server/python_backend/tree/r25.06/inferentia) - -### Additional Documentation - -- [FAQ](docs/user_guide/faq.md) -- [User Guide](docs/README.md#user-guide) -- [Customization Guide](docs/README.md#customization-guide) -- [Release Notes](https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/index.html) -- [GPU, Driver, and CUDA Support -Matrix](https://docs.nvidia.com/deeplearning/dgx/support-matrix/index.html) - -## Contributing - -Contributions to Triton Inference Server are more than welcome. To -contribute please review the [contribution -guidelines](CONTRIBUTING.md). If you have a backend, client, -example or similar contribution that is not modifying the core of -Triton, then you should file a PR in the [contrib -repo](https://github.com/triton-inference-server/contrib). - -## Reporting problems, asking questions - -We appreciate any feedback, questions or bug reporting regarding this project. -When posting [issues in GitHub](https://github.com/triton-inference-server/server/issues), -follow the process outlined in the [Stack Overflow document](https://stackoverflow.com/help/mcve). -Ensure posted examples are: -- minimal – use as little code as possible that still produces the - same problem -- complete – provide all parts needed to reproduce the problem. Check - if you can strip external dependencies and still show the problem. The - less time we spend on reproducing problems the more time we have to - fix it -- verifiable – test the code you're about to provide to make sure it - reproduces the problem. Remove all other problems that are not - related to your request/question. - -For issues, please use the provided bug report and feature request templates. - -For questions, we recommend posting in our community -[GitHub Discussions.](https://github.com/triton-inference-server/server/discussions) - -## For more information - -Please refer to the [NVIDIA Developer Triton page](https://developer.nvidia.com/nvidia-triton-inference-server) -for more information. +>[!WARNING] +> You are currently on the `r25.07` branch which tracks under-development progress towards the next release. From 44e416840f0740fd2a89a47f9eea8d02142a8702 Mon Sep 17 00:00:00 2001 From: Misha Chornyi <99709299+mc-nv@users.noreply.github.com> Date: Wed, 23 Jul 2025 12:43:44 -0700 Subject: [PATCH 07/14] test: Remove shared memory key from the error response (#8269) (#8308) Co-authored-by: Yingge He <157551214+yinggeh@users.noreply.github.com> --- qa/python_models/bls_memory/model.py | 4 ++-- qa/python_models/bls_memory_async/model.py | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/qa/python_models/bls_memory/model.py b/qa/python_models/bls_memory/model.py index 69da4f440f..a55da15774 100644 --- a/qa/python_models/bls_memory/model.py +++ b/qa/python_models/bls_memory/model.py @@ -1,4 +1,4 @@ -# Copyright 2021-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# Copyright 2021-2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions @@ -60,7 +60,7 @@ def test_bls_out_of_memory(self): input0_np, infer_response = self._send_identity_tensor( tensor_size, self._is_decoupled ) - out_of_memory_message = "Failed to increase the shared memory pool size for key" + out_of_memory_message = "Failed to increase the shared memory pool size" if infer_response.has_error(): self.assertIn(out_of_memory_message, infer_response.error().message()) diff --git a/qa/python_models/bls_memory_async/model.py b/qa/python_models/bls_memory_async/model.py index d9e676b42e..c96073eac0 100644 --- a/qa/python_models/bls_memory_async/model.py +++ b/qa/python_models/bls_memory_async/model.py @@ -1,4 +1,4 @@ -# Copyright 2021-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# Copyright 2021-2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions @@ -53,7 +53,7 @@ async def test_bls_out_of_memory(): tensor_size = 256 * 1024 * 1024 input0_np, infer_response = await _send_identity_tensor(tensor_size, is_decoupled) - out_of_memory_message = "Failed to increase the shared memory pool size for key" + out_of_memory_message = "Failed to increase the shared memory pool size" if infer_response.has_error(): if not (out_of_memory_message in infer_response.error().message()): From c70ea0353e45b6fc5a41adab68a0cd1365ba8066 Mon Sep 17 00:00:00 2001 From: Misha Chornyi <99709299+mc-nv@users.noreply.github.com> Date: Wed, 23 Jul 2025 13:16:23 -0700 Subject: [PATCH 08/14] fix: Improve data type validation for classification (#8267) (#8309) Co-authored-by: Sai Kiran Polisetty --- qa/L0_parameters/class_count_test.py | 105 +++++++++++++++++++++++++++ qa/L0_parameters/test.sh | 59 ++++++++++++++- src/classification.cc | 16 +++- 3 files changed, 175 insertions(+), 5 deletions(-) create mode 100755 qa/L0_parameters/class_count_test.py diff --git a/qa/L0_parameters/class_count_test.py b/qa/L0_parameters/class_count_test.py new file mode 100755 index 0000000000..9e49ae4ae2 --- /dev/null +++ b/qa/L0_parameters/class_count_test.py @@ -0,0 +1,105 @@ +#!/usr/bin/env python3 + +# Copyright 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of NVIDIA CORPORATION nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY +# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY +# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +import sys + +sys.path.append("../common") + +import os +import unittest + +import numpy as np +import test_util as tu +import tritonclient.grpc as grpcclient +import tritonclient.http as httpclient +from tritonclient.utils import InferenceServerException + + +class ClassificationParameterTest(tu.TestResultCollector): + def setUp(self): + self.protocol = os.environ.get("CLIENT_TYPE", "http") + if self.protocol == "http": + self.client = httpclient.InferenceServerClient("localhost:8000") + else: + self.client = grpcclient.InferenceServerClient("localhost:8001") + + def _prepare_io(self, input_data, dtype): + if self.protocol == "http": + inputs = [httpclient.InferInput("INPUT0", input_data.shape, dtype)] + outputs = [httpclient.InferRequestedOutput(name="OUTPUT0", class_count=5)] + else: + inputs = [grpcclient.InferInput("INPUT0", input_data.shape, dtype)] + outputs = [grpcclient.InferRequestedOutput(name="OUTPUT0", class_count=5)] + inputs[0].set_data_from_numpy(input_data) + return inputs, outputs + + def test_classificattion(self): + shape = (1, 8) + dtype = "FP32" + model_name = "identity_fp32" + input_data = np.ones(shape, dtype=np.float32) + + inputs, outputs = self._prepare_io(input_data, dtype) + result = self.client.infer( + model_name=model_name, inputs=inputs, outputs=outputs + ) + output = result.get_output("OUTPUT0") + if self.protocol == "http": + output_dtype = output["datatype"] + else: + output_dtype = output.datatype + + self.assertEqual(output_dtype, "BYTES") + + # Validate shape matches to the class_count + output_data = result.as_numpy("OUTPUT0") + self.assertIsNotNone(output_data) + self.assertEqual(output_data.shape, (1, 5)) + + for res_str_bytes in np.nditer(output_data, flags=["refs_ok"]): + res_str = res_str_bytes.item().decode("utf-8") + self.assertTrue(res_str.startswith("1.000000:")) + + def test_classificattion_unsupported_data_type(self): + shape = (1, 8) + model_name = "identity_bytes" + dtype = "BYTES" + input_data = np.array([["test"] * shape[1]], dtype=object) + + inputs, outputs = self._prepare_io(input_data, dtype) + with self.assertRaises(InferenceServerException) as e: + self.client.infer(model_name=model_name, inputs=inputs, outputs=outputs) + + self.assertIn( + "class result not available for output due to unsupported type 'BYTES'", + str(e.exception), + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/qa/L0_parameters/test.sh b/qa/L0_parameters/test.sh index c53b02d4b7..63868237aa 100755 --- a/qa/L0_parameters/test.sh +++ b/qa/L0_parameters/test.sh @@ -1,5 +1,5 @@ #!/bin/bash -# Copyright 2023-2024, NVIDIA CORPORATION. All rights reserved. +# Copyright 2023-2025, NVIDIA CORPORATION. All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions @@ -97,10 +97,65 @@ for i in "${all_tests[@]}"; do wait $SERVER_PID done + +# Test Classification Extension +PYTHON_MODELS_DIR="${PYTHON_MODELS_DIR:-/opt/tritonserver/qa/python_models}" +MODELDIR="models" +TEST_RESULT_FILE="test_results.txt" +TEST_SCRIPT_PY="./class_count_test.py" + +rm -rf $MODELDIR +mkdir -p "${MODELDIR}/identity_fp32/1" +cp ${PYTHON_MODELS_DIR}/identity_fp32/config.pbtxt "${MODELDIR}/identity_fp32/" +cp ${PYTHON_MODELS_DIR}/identity_fp32/model.py "${MODELDIR}/identity_fp32/1/" + +mkdir -p "${MODELDIR}/identity_bytes/1" +cp ${PYTHON_MODELS_DIR}/identity_fp32/config.pbtxt "${MODELDIR}/identity_bytes/" +cp ${PYTHON_MODELS_DIR}/identity_fp32/model.py "${MODELDIR}/identity_bytes/1/" +(cd "${MODELDIR}/identity_bytes" && \ + sed -i 's/identity_fp32/identity_bytes/' config.pbtxt && \ + sed -i 's/TYPE_FP32/TYPE_STRING/' config.pbtxt ) + +SERVER_ARGS="--model-repository=`pwd`/${MODELDIR} --log-verbose=1" +for client_type in http grpc; do + export CLIENT_TYPE=$client_type + SERVER_LOG="./class_count_test_${client_type}_server.log" + CLIENT_LOG="./class_count_test_${client_type}_client.log" + rm -f $SERVER_LOG $CLIENT_LOG + run_server + if [ "$SERVER_PID" == "0" ]; then + echo -e "\n***\n*** Failed to start $SERVER\n***" + cat $SERVER_LOG + exit 1 + fi + + set +e + python3 $TEST_SCRIPT_PY -v >>"$CLIENT_LOG" 2>&1 + if [ $? -ne 0 ]; then + cat $CLIENT_LOG + echo -e "\n***\n*** Test Failed - class_count_${client_type}_test_client\n***" + RET=1 + else + check_test_results $TEST_RESULT_FILE 2 + if [ $? -ne 0 ]; then + cat $TEST_RESULT_FILE + echo -e "\n***\n*** Test Result Verification Failed - class_count_${client_type}_test_client\n***" + RET=1 + fi + fi + kill $SERVER_PID + wait $SERVER_PID + + if [ $? -ne 0 ]; then + echo -e "\n***\n*** Test Server shut down non-gracefully\n***" + RET=1 + fi + set -e +done + if [ $RET -eq 0 ]; then echo -e "\n***\n*** Test Passed\n***" else - cat $CLIENT_LOG echo -e "\n***\n*** Test FAILED\n***" fi diff --git a/src/classification.cc b/src/classification.cc index 2d8cd26b9e..828edf613d 100644 --- a/src/classification.cc +++ b/src/classification.cc @@ -1,4 +1,4 @@ -// Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved. +// Copyright (c) 2020-2025, NVIDIA CORPORATION. All rights reserved. // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions @@ -77,8 +77,18 @@ TopkClassifications( const TRITONSERVER_DataType datatype, const uint32_t req_class_count, std::vector* class_strs) { - const size_t element_cnt = - byte_size / TRITONSERVER_DataTypeByteSize(datatype); + const uint32_t dtype_byte_size = TRITONSERVER_DataTypeByteSize(datatype); + if (dtype_byte_size == 0) { + return TRITONSERVER_ErrorNew( + TRITONSERVER_ERROR_INVALID_ARG, + std::string( + std::string("class result not available for output due to " + "unsupported type '") + + std::string(TRITONSERVER_DataTypeString(datatype)) + "'") + .c_str()); + } + + const size_t element_cnt = byte_size / dtype_byte_size; switch (datatype) { case TRITONSERVER_TYPE_UINT8: From e2e2ae56d833d2ba3a6f2b7a6423a4b4ec7eb5e9 Mon Sep 17 00:00:00 2001 From: Misha Chornyi <99709299+mc-nv@users.noreply.github.com> Date: Wed, 23 Jul 2025 13:16:39 -0700 Subject: [PATCH 09/14] fix: Improve validation for system shared memory register (#8273) (#8310) Co-authored-by: Sai Kiran Polisetty --- qa/L0_shared_memory/shared_memory_test.py | 18 ++++++++++++++++++ qa/L0_shared_memory/test.sh | 1 + src/common.h | 4 ++++ src/shared_memory_manager.cc | 12 +++++++++++- 4 files changed, 34 insertions(+), 1 deletion(-) diff --git a/qa/L0_shared_memory/shared_memory_test.py b/qa/L0_shared_memory/shared_memory_test.py index 146025cf26..35667aacfa 100755 --- a/qa/L0_shared_memory/shared_memory_test.py +++ b/qa/L0_shared_memory/shared_memory_test.py @@ -509,6 +509,24 @@ def test_python_client_leak(self): "client memory usage is increasing", ) + def test_register_reserved_names(self): + """ + Test that registration fails if attempting to use a reserved + prefix for the shm key. + """ + # This matches kTritonSharedMemoryRegionPrefix in the server code. + reserved_prefix = "triton_python_backend_shm_region_" + + # The shared memory key cannot start with the reserved prefix. + shm_name = "my_test_shm_name" + shm_key = f"{reserved_prefix}_my_test_shm_key" + + with self.assertRaisesRegex( + utils.InferenceServerException, + f"cannot register shared memory region '{shm_name}' with key '{shm_key}' as the key contains the reserved prefix '{reserved_prefix}'", + ) as e: + self.triton_client.register_system_shared_memory(shm_name, shm_key, 10000) + def callback(user_data, result, error): if error: diff --git a/qa/L0_shared_memory/test.sh b/qa/L0_shared_memory/test.sh index 97b81397ec..55cdb80951 100755 --- a/qa/L0_shared_memory/test.sh +++ b/qa/L0_shared_memory/test.sh @@ -55,6 +55,7 @@ for i in \ test_infer_byte_size_out_of_bound \ test_infer_integer_overflow \ test_register_out_of_bound \ + test_register_reserved_names \ test_python_client_leak; do for client_type in http grpc; do SERVER_ARGS="--model-repository=`pwd`/models --log-verbose=1 ${SERVER_ARGS_EXTRA}" diff --git a/src/common.h b/src/common.h index fe7e3be4cd..7df618e965 100644 --- a/src/common.h +++ b/src/common.h @@ -45,6 +45,10 @@ constexpr char kContentEncodingHTTPHeader[] = "Content-Encoding"; constexpr char kContentTypeHeader[] = "Content-Type"; constexpr char kContentLengthHeader[] = "Content-Length"; +// This prefix is reserved for shm regions created internally by Triton +constexpr char kTritonSharedMemoryRegionPrefix[] = + "triton_python_backend_shm_region_"; + constexpr int MAX_GRPC_MESSAGE_SIZE = INT32_MAX; /// The value for a dimension in a shape that indicates that that diff --git a/src/shared_memory_manager.cc b/src/shared_memory_manager.cc index 17b0e43da0..80b739ee1b 100644 --- a/src/shared_memory_manager.cc +++ b/src/shared_memory_manager.cc @@ -354,6 +354,17 @@ SharedMemoryManager::RegisterSystemSharedMemory( const std::string& name, const std::string& shm_key, const size_t offset, const size_t byte_size) { + // Check if the shared memory key starts with the reserved prefix + if (shm_key.rfind(kTritonSharedMemoryRegionPrefix, 0) == 0) { + return TRITONSERVER_ErrorNew( + TRITONSERVER_ERROR_INVALID_ARG, + std::string( + "cannot register shared memory region '" + name + "' with key '" + + shm_key + "' as the key contains the reserved prefix '" + + kTritonSharedMemoryRegionPrefix + "'") + .c_str()); + } + std::lock_guard lock(mu_); if (shared_memory_map_.find(name) != shared_memory_map_.end()) { @@ -670,7 +681,6 @@ SharedMemoryManager::UnregisterAll(TRITONSERVER_MemoryType memory_type) ++next_it; if (it->second->kind_ == TRITONSERVER_MEMORY_GPU) { TRITONSERVER_Error* err = UnregisterHelper(it->first, memory_type); - ; if (err != nullptr) { unregister_fails.push_back(it->first); LOG_VERBOSE(1) << TRITONSERVER_ErrorMessage(err); From 81ec444ca8567a6a0f59054e91980d4fd1704289 Mon Sep 17 00:00:00 2001 From: Sai Kiran Polisetty Date: Fri, 25 Jul 2025 22:38:20 +0530 Subject: [PATCH 10/14] =?UTF-8?q?feat:=20Add=20support=20for=20`usage`=20i?= =?UTF-8?q?n=20the=20OpenAI=20frontend=20vLLM=20backend=20(#8=E2=80=A6=20(?= =?UTF-8?q?#8313)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- python/openai/README.md | 22 +- .../openai_frontend/engine/triton_engine.py | 90 +++++- .../openai_frontend/engine/utils/triton.py | 84 ++++- .../openai/openai_frontend/schemas/openai.py | 16 + python/openai/tests/test_chat_completions.py | 39 ++- python/openai/tests/test_completions.py | 35 +- python/openai/tests/test_lora.py | 12 + python/openai/tests/test_openai_client.py | 303 +++++++++++++++++- 8 files changed, 573 insertions(+), 28 deletions(-) diff --git a/python/openai/README.md b/python/openai/README.md index 5851918ff2..a84f2ac4f4 100644 --- a/python/openai/README.md +++ b/python/openai/README.md @@ -98,7 +98,7 @@ curl -s http://localhost:9000/v1/chat/completions -H 'Content-Type: application/ ```json { - "id": "cmpl-6930b296-7ef8-11ef-bdd1-107c6149ca79", + "id": "cmpl-0242093d-51ae-11f0-b339-e7480668bfbe", "choices": [ { "finish_reason": "stop", @@ -113,11 +113,15 @@ curl -s http://localhost:9000/v1/chat/completions -H 'Content-Type: application/ "logprobs": null } ], - "created": 1727679085, + "created": 1750846825, "model": "llama-3.1-8b-instruct", "system_fingerprint": null, "object": "chat.completion", - "usage": null + "usage": { + "completion_tokens": 7, + "prompt_tokens": 42, + "total_tokens": 49 + } } ``` @@ -138,20 +142,24 @@ curl -s http://localhost:9000/v1/completions -H 'Content-Type: application/json' ```json { - "id": "cmpl-d51df75c-7ef8-11ef-bdd1-107c6149ca79", + "id": "cmpl-58fba3a0-51ae-11f0-859d-e7480668bfbe", "choices": [ { "finish_reason": "stop", "index": 0, "logprobs": null, - "text": " a field of computer science that focuses on developing algorithms that allow computers to learn from" + "text": " an amazing field that can truly understand the hidden patterns that exist in the data," } ], - "created": 1727679266, + "created": 1750846970, "model": "llama-3.1-8b-instruct", "system_fingerprint": null, "object": "text_completion", - "usage": null + "usage": { + "completion_tokens": 16, + "prompt_tokens": 4, + "total_tokens": 20 + } } ``` diff --git a/python/openai/openai_frontend/engine/triton_engine.py b/python/openai/openai_frontend/engine/triton_engine.py index 32357cfdd4..e5955861e1 100644 --- a/python/openai/openai_frontend/engine/triton_engine.py +++ b/python/openai/openai_frontend/engine/triton_engine.py @@ -51,7 +51,9 @@ _create_trtllm_inference_request, _create_vllm_inference_request, _get_output, + _get_usage_from_response, _get_vllm_lora_names, + _StreamingUsageAccumulator, _validate_triton_responses_non_streaming, ) from schemas.openai import ( @@ -65,6 +67,7 @@ ChatCompletionStreamResponseDelta, ChatCompletionToolChoiceOption1, Choice, + CompletionUsage, CreateChatCompletionRequest, CreateChatCompletionResponse, CreateChatCompletionStreamResponse, @@ -225,6 +228,8 @@ async def chat( backend=metadata.backend, ) + usage = _get_usage_from_response(response, metadata.backend) + return CreateChatCompletionResponse( id=request_id, choices=[ @@ -239,6 +244,7 @@ async def chat( model=request.model, system_fingerprint=None, object=ObjectType.chat_completion, + usage=usage, ) def _get_chat_completion_response_message( @@ -311,7 +317,7 @@ async def completion( created = int(time.time()) if request.stream: return self._streaming_completion_iterator( - request_id, created, request.model, responses + request_id, created, request, responses, metadata.backend ) # Response validation with decoupled models in mind @@ -320,6 +326,8 @@ async def completion( response = responses[0] text = _get_output(response) + usage = _get_usage_from_response(response, metadata.backend) + choice = Choice( finish_reason=FinishReason.stop, index=0, @@ -333,6 +341,7 @@ async def completion( object=ObjectType.text_completion, created=created, model=request.model, + usage=usage, ) # TODO: This behavior should be tested further @@ -413,6 +422,7 @@ def _get_streaming_chat_response_chunk( request_id: str, created: int, model: str, + usage: Optional[CompletionUsage] = None, ) -> CreateChatCompletionStreamResponse: return CreateChatCompletionStreamResponse( id=request_id, @@ -421,6 +431,7 @@ def _get_streaming_chat_response_chunk( model=model, system_fingerprint=None, object=ObjectType.chat_completion_chunk, + usage=usage, ) def _get_first_streaming_chat_response( @@ -436,7 +447,7 @@ def _get_first_streaming_chat_response( finish_reason=None, ) chunk = self._get_streaming_chat_response_chunk( - choice, request_id, created, model + choice, request_id, created, model, usage=None ) return chunk @@ -462,6 +473,8 @@ async def _streaming_chat_iterator( ) previous_text = "" + include_usage = request.stream_options and request.stream_options.include_usage + usage_accumulator = _StreamingUsageAccumulator(backend) chunk = self._get_first_streaming_chat_response( request_id, created, model, role @@ -470,6 +483,8 @@ async def _streaming_chat_iterator( async for response in responses: delta_text = _get_output(response) + if include_usage: + usage_accumulator.update(response) ( response_delta, @@ -504,10 +519,25 @@ async def _streaming_chat_iterator( ) chunk = self._get_streaming_chat_response_chunk( - choice, request_id, created, model + choice, request_id, created, model, usage=None ) yield f"data: {chunk.model_dump_json(exclude_unset=True)}\n\n" + # Send the final usage chunk if requested via stream_options. + if include_usage: + usage_payload = usage_accumulator.get_final_usage() + if usage_payload: + final_usage_chunk = CreateChatCompletionStreamResponse( + id=request_id, + choices=[], + created=created, + model=model, + system_fingerprint=None, + object=ObjectType.chat_completion_chunk, + usage=usage_payload, + ) + yield f"data: {final_usage_chunk.model_dump_json(exclude_unset=True)}\n\n" + yield "data: [DONE]\n\n" def _get_streaming_response_delta( @@ -654,6 +684,18 @@ def _validate_chat_request( self._verify_chat_tool_call_settings(request=request) + if request.stream_options and not request.stream: + raise Exception("`stream_options` can only be used when `stream` is True") + + if ( + request.stream_options + and request.stream_options.include_usage + and metadata.backend != "vllm" + ): + raise Exception( + "`stream_options.include_usage` is currently only supported for the vLLM backend" + ) + def _verify_chat_tool_call_settings(self, request: CreateChatCompletionRequest): if ( request.tool_choice @@ -690,9 +732,21 @@ def _verify_chat_tool_call_settings(self, request: CreateChatCompletionRequest): ) async def _streaming_completion_iterator( - self, request_id: str, created: int, model: str, responses: AsyncIterable + self, + request_id: str, + created: int, + request: CreateCompletionRequest, + responses: AsyncIterable, + backend: str, ) -> AsyncIterator[str]: + model = request.model + include_usage = request.stream_options and request.stream_options.include_usage + usage_accumulator = _StreamingUsageAccumulator(backend) + async for response in responses: + if include_usage: + usage_accumulator.update(response) + text = _get_output(response) choice = Choice( finish_reason=FinishReason.stop if response.final else None, @@ -707,10 +761,26 @@ async def _streaming_completion_iterator( object=ObjectType.text_completion, created=created, model=model, + usage=None, ) yield f"data: {chunk.model_dump_json(exclude_unset=True)}\n\n" + # Send the final usage chunk if requested via stream_options. + if include_usage: + usage_payload = usage_accumulator.get_final_usage() + if usage_payload: + final_usage_chunk = CreateCompletionResponse( + id=request_id, + choices=[], + system_fingerprint=None, + object=ObjectType.text_completion, + created=created, + model=model, + usage=usage_payload, + ) + yield f"data: {final_usage_chunk.model_dump_json(exclude_unset=True)}\n\n" + yield "data: [DONE]\n\n" def _validate_completion_request( @@ -763,6 +833,18 @@ def _validate_completion_request( if request.logit_bias is not None or request.logprobs is not None: raise Exception("logit bias and log probs not supported") + if request.stream_options and not request.stream: + raise Exception("`stream_options` can only be used when `stream` is True") + + if ( + request.stream_options + and request.stream_options.include_usage + and metadata.backend != "vllm" + ): + raise Exception( + "`stream_options.include_usage` is currently only supported for the vLLM backend" + ) + def _should_stream_with_auto_tool_parsing( self, request: CreateChatCompletionRequest ): diff --git a/python/openai/openai_frontend/engine/utils/triton.py b/python/openai/openai_frontend/engine/utils/triton.py index 53f1af44d6..2c29e80443 100644 --- a/python/openai/openai_frontend/engine/utils/triton.py +++ b/python/openai/openai_frontend/engine/utils/triton.py @@ -27,7 +27,7 @@ import json import os import re -from dataclasses import asdict +from dataclasses import asdict, dataclass, field from typing import Iterable, List, Optional, Union import numpy as np @@ -36,6 +36,7 @@ from schemas.openai import ( ChatCompletionNamedToolChoice, ChatCompletionToolChoiceOption1, + CompletionUsage, CreateChatCompletionRequest, CreateCompletionRequest, ) @@ -100,6 +101,8 @@ def _create_vllm_inference_request( # Pass sampling_parameters as serialized JSON string input to support List # fields like 'stop' that aren't supported by TRITONSERVER_Parameters yet. inputs["sampling_parameters"] = [sampling_parameters] + inputs["return_num_input_tokens"] = np.bool_([True]) + inputs["return_num_output_tokens"] = np.bool_([True]) return model.create_request(inputs=inputs) @@ -184,6 +187,85 @@ def _to_string(tensor: tritonserver.Tensor) -> str: return _construct_string_from_pointer(tensor.data_ptr + 4, tensor.size - 4) +@dataclass +class _StreamingUsageAccumulator: + """Helper class to accumulate token usage from a streaming response.""" + + backend: str + prompt_tokens: int = 0 + completion_tokens: int = 0 + _prompt_tokens_set: bool = field(init=False, default=False) + + def update(self, response: tritonserver.InferenceResponse): + """Extracts usage from a response and updates the token counts.""" + usage = _get_usage_from_response(response, self.backend) + if usage: + # The prompt_tokens is received with every chunk but should only be set once. + if not self._prompt_tokens_set: + self.prompt_tokens = usage.prompt_tokens + self._prompt_tokens_set = True + self.completion_tokens += usage.completion_tokens + + def get_final_usage(self) -> Optional[CompletionUsage]: + """ + Returns the final populated CompletionUsage object if any tokens were tracked. + """ + # If _prompt_tokens_set is True, it means we have received and processed + # at least one valid usage payload. + if self._prompt_tokens_set: + return CompletionUsage( + prompt_tokens=self.prompt_tokens, + completion_tokens=self.completion_tokens, + total_tokens=self.prompt_tokens + self.completion_tokens, + ) + return None + + +def _get_usage_from_response( + response: tritonserver._api._response.InferenceResponse, + backend: str, +) -> Optional[CompletionUsage]: + """ + Extracts token usage statistics from a Triton inference response. + """ + # TODO: Remove this check once TRT-LLM backend supports both "num_input_tokens" + # and "num_output_tokens", and also update the test cases accordingly. + if backend != "vllm": + return None + + prompt_tokens = None + completion_tokens = None + + if ( + "num_input_tokens" in response.outputs + and "num_output_tokens" in response.outputs + ): + input_token_tensor = response.outputs["num_input_tokens"] + output_token_tensor = response.outputs["num_output_tokens"] + + if input_token_tensor.data_type == tritonserver.DataType.UINT32: + prompt_tokens_ptr = ctypes.cast( + input_token_tensor.data_ptr, ctypes.POINTER(ctypes.c_uint32) + ) + prompt_tokens = prompt_tokens_ptr[0] + + if output_token_tensor.data_type == tritonserver.DataType.UINT32: + completion_tokens_ptr = ctypes.cast( + output_token_tensor.data_ptr, ctypes.POINTER(ctypes.c_uint32) + ) + completion_tokens = completion_tokens_ptr[0] + + if prompt_tokens is not None and completion_tokens is not None: + total_tokens = prompt_tokens + completion_tokens + return CompletionUsage( + prompt_tokens=prompt_tokens, + completion_tokens=completion_tokens, + total_tokens=total_tokens, + ) + + return None + + # TODO: Use tritonserver.InferenceResponse when support is published def _get_output(response: tritonserver._api._response.InferenceResponse) -> str: if "text_output" in response.outputs: diff --git a/python/openai/openai_frontend/schemas/openai.py b/python/openai/openai_frontend/schemas/openai.py index a2438e8394..7ed137b8d0 100644 --- a/python/openai/openai_frontend/schemas/openai.py +++ b/python/openai/openai_frontend/schemas/openai.py @@ -133,6 +133,10 @@ class CreateCompletionRequest(BaseModel): False, description="Whether to stream back partial progress. If set, tokens will be sent as data-only [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#Event_stream_format) as they become available, with the stream terminated by a `data: [DONE]` message. [Example Python code](https://cookbook.openai.com/examples/how_to_stream_completions).\n", ) + stream_options: Optional[StreamOptions] = Field( + None, + description="Options for streaming responses. Only use when `stream` is set to `true`.", + ) suffix: Optional[str] = Field( None, description="The suffix that comes after a completion of inserted text.\n\nThis parameter is only supported for `gpt-3.5-turbo-instruct`.\n", @@ -467,6 +471,13 @@ class ResponseFormat(BaseModel): ) +class StreamOptions(BaseModel): + include_usage: Optional[bool] = Field( + False, + description="If enabled, an additional chunk is sent before the `data: [DONE]` message. That chunk’s `usage` field reports the total token usage for the request and its `choices` array is always empty. All other chunks include a `usage` field with a null value.", + ) + + class FunctionCall3(Enum): none = "none" auto = "auto" @@ -573,6 +584,7 @@ class CreateChatCompletionStreamResponse(BaseModel): object: Object4 = Field( ..., description="The object type, which is always `chat.completion.chunk`." ) + usage: Optional[CompletionUsage] = None class CreateChatCompletionImageResponse(BaseModel): @@ -885,6 +897,10 @@ class CreateChatCompletionRequest(BaseModel): False, description="If set, partial message deltas will be sent, like in ChatGPT. Tokens will be sent as data-only [server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#Event_stream_format) as they become available, with the stream terminated by a `data: [DONE]` message. [Example Python code](https://cookbook.openai.com/examples/how_to_stream_completions).\n", ) + stream_options: Optional[StreamOptions] = Field( + None, + description="Options for streaming responses. Only use when `stream` is set to `true`.", + ) temperature: Optional[confloat(ge=0.0, le=2.0)] = Field( 0.7, description="What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic.\n\nWe generally recommend altering this or `top_p` but not both.\n", diff --git a/python/openai/tests/test_chat_completions.py b/python/openai/tests/test_chat_completions.py index 532d898788..8f8c85d444 100644 --- a/python/openai/tests/test_chat_completions.py +++ b/python/openai/tests/test_chat_completions.py @@ -41,7 +41,9 @@ class TestChatCompletions: def client(self, fastapi_client_class_scope): yield fastapi_client_class_scope - def test_chat_completions_defaults(self, client, model: str, messages: List[dict]): + def test_chat_completions_defaults( + self, client, model: str, messages: List[dict], backend: str + ): response = client.post( "/v1/chat/completions", json={"model": model, "messages": messages}, @@ -51,8 +53,12 @@ def test_chat_completions_defaults(self, client, model: str, messages: List[dict message = response.json()["choices"][0]["message"] assert message["content"].strip() assert message["role"] == "assistant" - # "usage" currently not supported - assert not response.json()["usage"] + + usage = response.json().get("usage") + if backend == "vllm": + assert usage is not None + else: + assert usage is None def test_chat_completions_system_prompt(self, client, model: str): # NOTE: Currently just sanity check that there are no issues when a @@ -497,9 +503,30 @@ def test_request_logprobs(self): def test_request_logit_bias(self): pass - @pytest.mark.skip(reason="Not Implemented Yet") - def test_usage_response(self): - pass + def test_usage_response( + self, client, model: str, messages: List[dict], backend: str + ): + if backend != "vllm": + pytest.skip( + "Usage reporting is currently available only for the vLLM backend." + ) + + response = client.post( + "/v1/chat/completions", + json={"model": model, "messages": messages}, + ) + + assert response.status_code == 200 + usage = response.json().get("usage") + assert usage is not None + assert isinstance(usage["prompt_tokens"], int) + assert isinstance(usage["completion_tokens"], int) + assert isinstance(usage["total_tokens"], int) + assert usage["prompt_tokens"] > 0 + assert usage["completion_tokens"] > 0 + assert ( + usage["total_tokens"] == usage["prompt_tokens"] + usage["completion_tokens"] + ) # For tests that won't use the same pytest fixture for server startup across diff --git a/python/openai/tests/test_completions.py b/python/openai/tests/test_completions.py index 1a411a0124..9ec3ffe7f7 100644 --- a/python/openai/tests/test_completions.py +++ b/python/openai/tests/test_completions.py @@ -35,7 +35,7 @@ class TestCompletions: def client(self, fastapi_client_class_scope): yield fastapi_client_class_scope - def test_completions_defaults(self, client, model: str, prompt: str): + def test_completions_defaults(self, client, model: str, prompt: str, backend: str): response = client.post( "/v1/completions", json={"model": model, "prompt": prompt}, @@ -46,8 +46,12 @@ def test_completions_defaults(self, client, model: str, prompt: str): # NOTE: Could be improved to look for certain quality of response, # or tested with dummy identity model. assert response.json()["choices"][0]["text"].strip() - # "usage" currently not supported - assert not response.json()["usage"] + + usage = response.json().get("usage") + if backend == "vllm": + assert usage is not None + else: + assert usage is None @pytest.mark.parametrize( "sampling_parameter, value", @@ -367,6 +371,25 @@ def test_lora(self): def test_multi_lora(self): pass - @pytest.mark.skip(reason="Not Implemented Yet") - def test_usage_response(self): - pass + def test_usage_response(self, client, model: str, prompt: str, backend: str): + if backend != "vllm": + pytest.skip( + "Usage reporting is currently available only for the vLLM backend." + ) + + response = client.post( + "/v1/completions", + json={"model": model, "prompt": prompt}, + ) + + assert response.status_code == 200 + usage = response.json().get("usage") + assert usage is not None + assert isinstance(usage["prompt_tokens"], int) + assert isinstance(usage["completion_tokens"], int) + assert isinstance(usage["total_tokens"], int) + assert usage["prompt_tokens"] > 0 + assert usage["completion_tokens"] > 0 + assert ( + usage["total_tokens"] == usage["prompt_tokens"] + usage["completion_tokens"] + ) diff --git a/python/openai/tests/test_lora.py b/python/openai/tests/test_lora.py index 2c89789bbc..4a5ff10361 100644 --- a/python/openai/tests/test_lora.py +++ b/python/openai/tests/test_lora.py @@ -131,6 +131,18 @@ def _create_model_repository_mock_llm(self): name: "exclude_input_in_output" data_type: TYPE_BOOL dims: [ 1 ] + }, + { + name: "return_num_input_tokens" + data_type: TYPE_BOOL + dims: [1] + optional: true + }, + { + name: "return_num_output_tokens" + data_type: TYPE_BOOL + dims: [1] + optional: true } ] output [ diff --git a/python/openai/tests/test_openai_client.py b/python/openai/tests/test_openai_client.py index 618e4052ed..9d00343415 100644 --- a/python/openai/tests/test_openai_client.py +++ b/python/openai/tests/test_openai_client.py @@ -49,7 +49,7 @@ def test_openai_client_models(self, client: openai.OpenAI, backend: str): raise Exception(f"Unexpected backend {backend=}") def test_openai_client_completion( - self, client: openai.OpenAI, model: str, prompt: str + self, client: openai.OpenAI, model: str, prompt: str, backend: str ): completion = client.completions.create( prompt=prompt, @@ -60,8 +60,20 @@ def test_openai_client_completion( assert completion.choices[0].text assert completion.choices[0].finish_reason == "stop" + usage = completion.usage + if backend == "vllm": + assert usage is not None + assert isinstance(usage.prompt_tokens, int) + assert isinstance(usage.completion_tokens, int) + assert isinstance(usage.total_tokens, int) + assert usage.prompt_tokens > 0 + assert usage.completion_tokens > 0 + assert usage.total_tokens == usage.prompt_tokens + usage.completion_tokens + else: + assert usage is None + def test_openai_client_chat_completion( - self, client: openai.OpenAI, model: str, messages: List[dict] + self, client: openai.OpenAI, model: str, messages: List[dict], backend: str ): chat_completion = client.chat.completions.create( messages=messages, @@ -72,6 +84,18 @@ def test_openai_client_chat_completion( assert chat_completion.choices[0].message.content assert chat_completion.choices[0].finish_reason == "stop" + usage = chat_completion.usage + if backend == "vllm": + assert usage is not None + assert isinstance(usage.prompt_tokens, int) + assert isinstance(usage.completion_tokens, int) + assert isinstance(usage.total_tokens, int) + assert usage.prompt_tokens > 0 + assert usage.completion_tokens > 0 + assert usage.total_tokens == usage.prompt_tokens + usage.completion_tokens + else: + assert usage is None + @pytest.mark.parametrize("echo", [False, True]) def test_openai_client_completion_echo( self, client: openai.OpenAI, echo: bool, backend: str, model: str, prompt: str @@ -117,7 +141,7 @@ async def test_openai_client_models(self, client: openai.AsyncOpenAI, backend: s @pytest.mark.asyncio async def test_openai_client_completion( - self, client: openai.AsyncOpenAI, model: str, prompt: str + self, client: openai.AsyncOpenAI, model: str, prompt: str, backend: str ): completion = await client.completions.create( prompt=prompt, @@ -128,9 +152,21 @@ async def test_openai_client_completion( assert completion.choices[0].text assert completion.choices[0].finish_reason == "stop" + usage = completion.usage + if backend == "vllm": + assert usage is not None + assert isinstance(usage.prompt_tokens, int) + assert isinstance(usage.completion_tokens, int) + assert isinstance(usage.total_tokens, int) + assert usage.prompt_tokens > 0 + assert usage.completion_tokens > 0 + assert usage.total_tokens == usage.prompt_tokens + usage.completion_tokens + else: + assert usage is None + @pytest.mark.asyncio async def test_openai_client_chat_completion( - self, client: openai.AsyncOpenAI, model: str, messages: List[dict] + self, client: openai.AsyncOpenAI, model: str, messages: List[dict], backend: str ): chat_completion = await client.chat.completions.create( messages=messages, @@ -139,6 +175,19 @@ async def test_openai_client_chat_completion( assert chat_completion.choices[0].message.content assert chat_completion.choices[0].finish_reason == "stop" + + usage = chat_completion.usage + if backend == "vllm": + assert usage is not None + assert isinstance(usage.prompt_tokens, int) + assert isinstance(usage.completion_tokens, int) + assert isinstance(usage.total_tokens, int) + assert usage.prompt_tokens > 0 + assert usage.completion_tokens > 0 + assert usage.total_tokens == usage.prompt_tokens + usage.completion_tokens + else: + assert usage is None + print(f"Chat completion results: {chat_completion}") @pytest.mark.asyncio @@ -235,6 +284,7 @@ async def test_chat_streaming( chunks.append(delta.content) if chunk.choices[0].finish_reason is not None: finish_reason_count += 1 + assert chunk.usage is None # finish reason should only return in last block assert finish_reason_count == 1 @@ -245,3 +295,248 @@ async def test_chat_streaming( assert len(chunks) > 1 streamed_output = "".join(chunks) assert streamed_output == output + + @pytest.mark.asyncio + async def test_chat_streaming_usage_option( + self, client: openai.AsyncOpenAI, model: str, messages: List[dict], backend: str + ): + if backend != "vllm": + pytest.skip( + "Usage reporting is currently available only for the vLLM backend." + ) + + seed = 0 + temperature = 0.0 + max_tokens = 16 + + # Get usage and content from a non-streaming call + stream_false = await client.chat.completions.create( + model=model, + messages=messages, + max_tokens=max_tokens, + temperature=temperature, + seed=seed, + stream=False, + ) + usage_stream_false = stream_false.usage + stream_false_output = stream_false.choices[0].message.content + assert usage_stream_false is not None + assert stream_false_output is not None + + # First, run with include_usage=False. + stream_options_false = await client.chat.completions.create( + model=model, + messages=messages, + max_tokens=max_tokens, + temperature=temperature, + seed=seed, + stream=True, + stream_options={"include_usage": False}, + ) + chunks_false = [chunk async for chunk in stream_options_false] + for chunk in chunks_false: + assert chunk.usage is None, "Usage should be null when include_usage=False" + stream_options_false_output = "".join( + c.choices[0].delta.content + for c in chunks_false + if c.choices and c.choices[0].delta.content + ) + + # Now, run with include_usage=True. + stream_options_true = await client.chat.completions.create( + model=model, + messages=messages, + max_tokens=max_tokens, + temperature=temperature, + seed=seed, + stream=True, + stream_options={"include_usage": True}, + ) + chunks_true = [chunk async for chunk in stream_options_true] + content_chunks = [c for c in chunks_true if c.usage is None] + + # Verify that we received exactly one extra chunk. + assert len(chunks_true) == len(chunks_false) + 1 + + # Verify content is consistent + stream_options_true_output = "".join( + c.choices[0].delta.content + for c in content_chunks + if c.choices and c.choices[0].delta.content + ) + assert stream_options_true_output == stream_false_output + assert stream_options_true_output == stream_options_false_output + + # Verify the final chunk has usage data and empty choices. + final_chunk = chunks_true[-1] + assert final_chunk.usage is not None + assert len(final_chunk.choices) == 0 + usage_stream_options_true = final_chunk.usage + assert ( + isinstance(usage_stream_options_true.prompt_tokens, int) + and usage_stream_options_true.prompt_tokens > 0 + ) + assert ( + isinstance(usage_stream_options_true.completion_tokens, int) + and usage_stream_options_true.completion_tokens > 0 + ) + assert ( + usage_stream_options_true.total_tokens + == usage_stream_options_true.prompt_tokens + + usage_stream_options_true.completion_tokens + ) + + # Verify other chunks have no usage data. + for chunk in chunks_true[:-1]: + assert chunk.usage is None + + # Assert usage is consistent between streaming and non-streaming calls + assert usage_stream_false.model_dump() == usage_stream_options_true.model_dump() + + @pytest.mark.asyncio + async def test_completion_streaming_usage_option( + self, client: openai.AsyncOpenAI, model: str, prompt: str, backend: str + ): + if backend != "vllm": + pytest.skip( + "Usage reporting is currently available only for the vLLM backend." + ) + + seed = 0 + temperature = 0.0 + max_tokens = 16 + + # Get usage and content from a non-streaming call + stream_false = await client.completions.create( + model=model, + prompt=prompt, + max_tokens=max_tokens, + temperature=temperature, + stream=False, + seed=seed, + ) + usage_stream_false = stream_false.usage + stream_false_output = stream_false.choices[0].text + assert usage_stream_false is not None + assert stream_false_output is not None + + # First, run with include_usage=False. + stream_options_false = await client.completions.create( + model=model, + prompt=prompt, + max_tokens=max_tokens, + temperature=temperature, + seed=seed, + stream=True, + stream_options={"include_usage": False}, + ) + chunks_false = [chunk async for chunk in stream_options_false] + for chunk in chunks_false: + assert chunk.usage is None + stream_options_false_output = "".join( + c.choices[0].text for c in chunks_false if c.choices and c.choices[0].text + ) + + # Now, run with include_usage=True. + stream_options_true = await client.completions.create( + model=model, + prompt=prompt, + max_tokens=max_tokens, + temperature=temperature, + stream=True, + seed=seed, + stream_options={"include_usage": True}, + ) + chunks_true = [chunk async for chunk in stream_options_true] + content_chunks = [c for c in chunks_true if c.usage is None] + + # Verify that we received exactly one extra chunk. + assert len(chunks_true) == len(chunks_false) + 1 + + # Verify content is consistent + stream_options_true_output = "".join( + c.choices[0].text for c in content_chunks if c.choices and c.choices[0].text + ) + assert stream_options_true_output == stream_false_output + assert stream_options_true_output == stream_options_false_output + + # Verify the final chunk has usage data and empty choices. + final_chunk = chunks_true[-1] + assert final_chunk.usage is not None + assert len(final_chunk.choices) == 0 + usage_stream_options_true = final_chunk.usage + assert ( + isinstance(usage_stream_options_true.prompt_tokens, int) + and usage_stream_options_true.prompt_tokens > 0 + ) + assert ( + isinstance(usage_stream_options_true.completion_tokens, int) + and usage_stream_options_true.completion_tokens > 0 + ) + assert ( + usage_stream_options_true.total_tokens + == usage_stream_options_true.prompt_tokens + + usage_stream_options_true.completion_tokens + ) + + # Verify other chunks have no usage data. + for chunk in chunks_true[:-1]: + assert chunk.usage is None + + # Assert usage is consistent between streaming and non-streaming calls + assert usage_stream_false.model_dump() == usage_stream_options_true.model_dump() + + @pytest.mark.asyncio + async def test_stream_options_without_streaming( + self, client: openai.AsyncOpenAI, model: str, prompt: str + ): + with pytest.raises(openai.BadRequestError) as e: + await client.completions.create( + model=model, + prompt=prompt, + stream=False, + stream_options={"include_usage": True}, + ) + assert "`stream_options` can only be used when `stream` is True" in str(e.value) + + with pytest.raises(openai.BadRequestError) as e: + await client.chat.completions.create( + model=model, + messages=[{"role": "user", "content": prompt}], + stream=False, + stream_options={"include_usage": True}, + ) + assert "`stream_options` can only be used when `stream` is True" in str(e.value) + + @pytest.mark.asyncio + async def test_streaming_usage_unsupported_backend( + self, client: openai.AsyncOpenAI, model: str, messages: List[dict], backend: str + ): + if backend == "vllm": + pytest.skip( + "This test is for backends that do not support usage reporting." + ) + + with pytest.raises(openai.BadRequestError) as e: + await client.completions.create( + model=model, + prompt="Test prompt", + stream=True, + stream_options={"include_usage": True}, + ) + assert ( + "`stream_options.include_usage` is currently only supported for the vLLM backend" + in str(e.value) + ) + + with pytest.raises(openai.BadRequestError) as e: + await client.chat.completions.create( + model=model, + messages=messages, + stream=True, + stream_options={"include_usage": True}, + ) + assert ( + "`stream_options.include_usage` is currently only supported for the vLLM backend" + in str(e.value) + ) From 3772013bf3d218a911dbd8edfc74dc37f5023d62 Mon Sep 17 00:00:00 2001 From: Misha Chornyi <99709299+mc-nv@users.noreply.github.com> Date: Fri, 25 Jul 2025 10:31:47 -0700 Subject: [PATCH 11/14] TPRD-1638: adding missed argument (#8312) --- Dockerfile.sdk | 61 ++++++++----------- TRITON_VERSION | 2 +- build.py | 6 +- deploy/aws/values.yaml | 2 +- deploy/fleetcommand/Chart.yaml | 2 +- deploy/fleetcommand/values.yaml | 6 +- deploy/gcp/values.yaml | 2 +- .../perf-analyzer-script/triton_client.yaml | 2 +- .../server-deployer/build_and_push.sh | 4 +- .../server-deployer/chart/triton/Chart.yaml | 2 +- .../server-deployer/chart/triton/values.yaml | 6 +- .../server-deployer/data-test/schema.yaml | 2 +- .../server-deployer/schema.yaml | 4 +- .../gke-marketplace-app/trt-engine/README.md | 6 +- deploy/k8s-onprem/values.yaml | 2 +- deploy/oci/values.yaml | 2 +- docs/introduction/compatibility.md | 1 + python/openai/README.md | 2 +- qa/common/gen_jetson_trt_models | 2 +- qa/common/gen_qa_custom_ops | 2 +- qa/common/gen_qa_model_repository | 2 +- 21 files changed, 57 insertions(+), 63 deletions(-) diff --git a/Dockerfile.sdk b/Dockerfile.sdk index d1c4b5a189..e24d5c1982 100644 --- a/Dockerfile.sdk +++ b/Dockerfile.sdk @@ -29,7 +29,7 @@ # # Base image on the minimum Triton container -ARG BASE_IMAGE=nvcr.io/nvidia/tritonserver:25.06-py3-min +ARG BASE_IMAGE=nvcr.io/nvidia/tritonserver:25.07-py3-min ARG TRITON_CLIENT_REPO_SUBDIR=clientrepo ARG TRITON_PA_REPO_SUBDIR=perfanalyzerrepo @@ -60,44 +60,35 @@ ENV PIP_BREAK_SYSTEM_PACKAGES=1 RUN apt-get update && \ apt-get install -y --no-install-recommends \ - ca-certificates \ - software-properties-common \ autoconf \ automake \ build-essential \ + ca-certificates \ curl \ git \ gperf \ libb64-dev \ libgoogle-perftools-dev \ - libopencv-dev \ libopencv-core-dev \ + libopencv-dev \ libssl-dev \ libtool \ + maven \ + openjdk-11-jdk \ pkg-config \ python3 \ - python3-pip \ python3-dev \ - python3-wheel \ + python3-pdfkit \ + python3-pip \ python3-setuptools \ + python3-wheel \ rapidjson-dev \ + software-properties-common \ vim \ - wget \ - python3-pdfkit \ - openjdk-11-jdk \ - maven && \ - pip3 install --upgrade "grpcio-tools<1.68" - -# Client build requires recent version of CMake (FetchContent required) -# Using CMAKE installation instruction from:: https://apt.kitware.com/ -RUN apt update -q=2 \ - && apt install -y gpg wget \ - && wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | gpg --dearmor - | tee /usr/share/keyrings/kitware-archive-keyring.gpg >/dev/null \ - && . /etc/os-release \ - && echo "deb [signed-by=/usr/share/keyrings/kitware-archive-keyring.gpg] https://apt.kitware.com/ubuntu/ $UBUNTU_CODENAME main" | tee /etc/apt/sources.list.d/kitware.list >/dev/null \ - && apt-get update -q=2 \ - && apt-get install -y --no-install-recommends cmake=3.28.3* cmake-data=3.28.3* \ - && cmake --version + wget && \ + pip3 install --upgrade "grpcio-tools<1.68" cmake==3.28.3 + +ENV CMAKE_POLICY_MINIMUM_REQUIRED=3.5 # Build expects "python" executable (not python3). RUN rm -f /usr/bin/python && \ @@ -137,8 +128,7 @@ RUN cmake -DCMAKE_INSTALL_PREFIX=/workspace/install \ -DTRITON_ENABLE_JAVA_HTTP=ON \ -DTRITON_ENABLE_EXAMPLES=ON -DTRITON_ENABLE_TESTS=ON \ -DTRITON_ENABLE_GPU=${TRITON_ENABLE_GPU} /workspace/client -RUN make -j16 cc-clients java-clients && \ - rm -fr ~/.m2 +RUN cmake --build . -v --parallel --target cc-clients java-clients # TODO: PA will rebuild the CC clients since it depends on it. # This should be optimized so that we do not have to build @@ -156,6 +146,7 @@ RUN if [ "$TRITON_PERF_ANALYZER_BUILD" = "1" ]; then \ -DTRITON_COMMON_REPO_TAG=${TRITON_COMMON_REPO_TAG} \ -DTRITON_CORE_REPO_TAG=${TRITON_CORE_REPO_TAG} \ -DTRITON_CLIENT_REPO_TAG=${TRITON_CLIENT_REPO_TAG} \ + -DTRITON_THIRD_PARTY_REPO_TAG=${TRITON_THIRD_PARTY_REPO_TAG} \ -DTRITON_ENABLE_PERF_ANALYZER_C_API=ON \ -DTRITON_ENABLE_PERF_ANALYZER_TFS=ON \ -DTRITON_ENABLE_PERF_ANALYZER_TS=ON \ @@ -167,7 +158,7 @@ RUN if [ "$TRITON_PERF_ANALYZER_BUILD" = "1" ]; then \ -DTRITON_PACKAGE_PERF_ANALYZER=ON \ -DTRITON_ENABLE_GPU=${TRITON_ENABLE_GPU} \ /workspace/perf_analyzer && \ - make -j16 perf-analyzer python-clients && \ + cmake --build . -v --parallel --target perf-analyzer python-clients && \ pip3 install build && \ cd /workspace/perf_analyzer/genai-perf && \ python3 -m build --wheel --outdir /workspace/install/python; \ @@ -180,12 +171,13 @@ RUN if [ "$TRITON_PERF_ANALYZER_BUILD" = "1" ]; then \ -DTRITON_REPO_ORGANIZATION=${TRITON_REPO_ORGANIZATION} \ -DTRITON_COMMON_REPO_TAG=${TRITON_COMMON_REPO_TAG} \ -DTRITON_CLIENT_REPO_TAG=${TRITON_CLIENT_REPO_TAG} \ + -DTRITON_THIRD_PARTY_REPO_TAG=${TRITON_THIRD_PARTY_REPO_TAG} \ -DTRITON_ENABLE_PYTHON_HTTP=ON \ -DTRITON_ENABLE_PYTHON_GRPC=ON \ -DTRITON_PACKAGE_PERF_ANALYZER=ON \ -DTRITON_ENABLE_GPU=${TRITON_ENABLE_GPU} \ /workspace/perf_analyzer && \ - make -j16 python-clients && \ + cmake --build . -v --parallel --target python-clients && \ mkdir -p /workspace/install/python && \ cp /workspace/perf_analyzer/genai_perf-*.whl /workspace/install/python/; \ fi @@ -216,26 +208,27 @@ ARG TRITON_ENABLE_GPU RUN apt-get update && \ apt-get install -y --no-install-recommends \ - software-properties-common \ curl \ + default-jdk \ git \ gperf \ libb64-dev \ libgoogle-perftools-dev \ - libopencv-dev \ libopencv-core-dev \ + libopencv-dev \ libssl-dev \ libtool \ + maven \ + perl \ python3 \ - python3-pip \ python3-dev \ - python3-wheel \ + python3-pdfkit \ + python3-pip \ python3-setuptools \ + python3-wheel \ + software-properties-common \ vim \ - wget \ - python3-pdfkit \ - maven \ - default-jdk && \ + wget && \ pip3 install "grpcio<1.68" "grpcio-tools<1.68" WORKDIR /workspace diff --git a/TRITON_VERSION b/TRITON_VERSION index e7a9ca6a62..3b4ce49e5d 100644 --- a/TRITON_VERSION +++ b/TRITON_VERSION @@ -1 +1 @@ -2.59.0 +2.59.1 diff --git a/build.py b/build.py index 5ced5327f2..f848ff05f8 100755 --- a/build.py +++ b/build.py @@ -71,9 +71,9 @@ # DEFAULT_TRITON_VERSION_MAP = { - "release_version": "2.59.0", - "triton_container_version": "25.06", - "upstream_container_version": "25.06", + "release_version": "2.59.1", + "triton_container_version": "25.07", + "upstream_container_version": "25.07", "ort_version": "1.22.0", "ort_openvino_version": "2025.1.0", "standalone_openvino_version": "2025.1.0", diff --git a/deploy/aws/values.yaml b/deploy/aws/values.yaml index c24b0ed7b8..eb9afb78e0 100644 --- a/deploy/aws/values.yaml +++ b/deploy/aws/values.yaml @@ -27,7 +27,7 @@ replicaCount: 1 image: - imageName: nvcr.io/nvidia/tritonserver:25.06-py3 + imageName: nvcr.io/nvidia/tritonserver:25.07-py3 pullPolicy: IfNotPresent modelRepositoryPath: s3://triton-inference-server-repository/model_repository numGpus: 1 diff --git a/deploy/fleetcommand/Chart.yaml b/deploy/fleetcommand/Chart.yaml index b819cbb4a5..1b1af554ee 100644 --- a/deploy/fleetcommand/Chart.yaml +++ b/deploy/fleetcommand/Chart.yaml @@ -26,7 +26,7 @@ apiVersion: v1 # appVersion is the Triton version; update when changing release -appVersion: "2.59.0" +appVersion: "2.59.1" description: Triton Inference Server (Fleet Command) name: triton-inference-server # version is the Chart version; update when changing anything in the chart diff --git a/deploy/fleetcommand/values.yaml b/deploy/fleetcommand/values.yaml index d2300ef421..03569ab343 100644 --- a/deploy/fleetcommand/values.yaml +++ b/deploy/fleetcommand/values.yaml @@ -27,7 +27,7 @@ replicaCount: 1 image: - imageName: nvcr.io/nvidia/tritonserver:25.06-py3 + imageName: nvcr.io/nvidia/tritonserver:25.07-py3 pullPolicy: IfNotPresent numGpus: 1 serverCommand: tritonserver @@ -47,13 +47,13 @@ image: # # To set model control mode, uncomment and configure below # TODO: Fix the following url, it is invalid - # See https://github.com/triton-inference-server/server/blob/r25.06/docs/user_guide/model_management.md + # See https://github.com/triton-inference-server/server/blob/r25.07/docs/user_guide/model_management.md # for more details #- --model-control-mode=explicit|poll|none # # Additional server args # - # see https://github.com/triton-inference-server/server/blob/r25.06/README.md + # see https://github.com/triton-inference-server/server/blob/r25.07/README.md # for more details service: diff --git a/deploy/gcp/values.yaml b/deploy/gcp/values.yaml index 73948fb39d..55d0919205 100644 --- a/deploy/gcp/values.yaml +++ b/deploy/gcp/values.yaml @@ -27,7 +27,7 @@ replicaCount: 1 image: - imageName: nvcr.io/nvidia/tritonserver:25.06-py3 + imageName: nvcr.io/nvidia/tritonserver:25.07-py3 pullPolicy: IfNotPresent modelRepositoryPath: gs://triton-inference-server-repository/model_repository numGpus: 1 diff --git a/deploy/gke-marketplace-app/benchmark/perf-analyzer-script/triton_client.yaml b/deploy/gke-marketplace-app/benchmark/perf-analyzer-script/triton_client.yaml index 8be23e0f2c..951f04898e 100644 --- a/deploy/gke-marketplace-app/benchmark/perf-analyzer-script/triton_client.yaml +++ b/deploy/gke-marketplace-app/benchmark/perf-analyzer-script/triton_client.yaml @@ -33,7 +33,7 @@ metadata: namespace: default spec: containers: - - image: nvcr.io/nvidia/tritonserver:25.06-py3-sdk + - image: nvcr.io/nvidia/tritonserver:25.07-py3-sdk imagePullPolicy: Always name: nv-triton-client securityContext: diff --git a/deploy/gke-marketplace-app/server-deployer/build_and_push.sh b/deploy/gke-marketplace-app/server-deployer/build_and_push.sh index b31804b9a2..659e3a40eb 100755 --- a/deploy/gke-marketplace-app/server-deployer/build_and_push.sh +++ b/deploy/gke-marketplace-app/server-deployer/build_and_push.sh @@ -28,8 +28,8 @@ export REGISTRY=gcr.io/$(gcloud config get-value project | tr ':' '/') export APP_NAME=tritonserver export MAJOR_VERSION=2.59 -export MINOR_VERSION=2.59.0 -export NGC_VERSION=25.06-py3 +export MINOR_VERSION=2.59.1 +export NGC_VERSION=25.07-py3 docker pull nvcr.io/nvidia/$APP_NAME:$NGC_VERSION diff --git a/deploy/gke-marketplace-app/server-deployer/chart/triton/Chart.yaml b/deploy/gke-marketplace-app/server-deployer/chart/triton/Chart.yaml index 3206248935..218a45c7a5 100644 --- a/deploy/gke-marketplace-app/server-deployer/chart/triton/Chart.yaml +++ b/deploy/gke-marketplace-app/server-deployer/chart/triton/Chart.yaml @@ -28,4 +28,4 @@ apiVersion: v1 appVersion: "2.59" description: Triton Inference Server name: triton-inference-server -version: 2.59.0 +version: 2.59.1 diff --git a/deploy/gke-marketplace-app/server-deployer/chart/triton/values.yaml b/deploy/gke-marketplace-app/server-deployer/chart/triton/values.yaml index 52b1ab2f21..94e5132d4d 100644 --- a/deploy/gke-marketplace-app/server-deployer/chart/triton/values.yaml +++ b/deploy/gke-marketplace-app/server-deployer/chart/triton/values.yaml @@ -31,14 +31,14 @@ maxReplicaCount: 3 tritonProtocol: HTTP # HPA GPU utilization autoscaling target HPATargetAverageValue: 85 -modelRepositoryPath: gs://triton_sample_models/25.06 -publishedVersion: '2.59.0' +modelRepositoryPath: gs://triton_sample_models/25.07 +publishedVersion: '2.59.1' gcpMarketplace: true image: registry: gcr.io repository: nvidia-ngc-public/tritonserver - tag: 25.06-py3 + tag: 25.07-py3 pullPolicy: IfNotPresent # modify the model repository here to match your GCP storage bucket numGpus: 1 diff --git a/deploy/gke-marketplace-app/server-deployer/data-test/schema.yaml b/deploy/gke-marketplace-app/server-deployer/data-test/schema.yaml index 36ba6b6a63..edfcbe164b 100644 --- a/deploy/gke-marketplace-app/server-deployer/data-test/schema.yaml +++ b/deploy/gke-marketplace-app/server-deployer/data-test/schema.yaml @@ -27,7 +27,7 @@ x-google-marketplace: schemaVersion: v2 applicationApiVersion: v1beta1 - publishedVersion: '2.59.0' + publishedVersion: '2.59.1' publishedVersionMetadata: releaseNote: >- Initial release. diff --git a/deploy/gke-marketplace-app/server-deployer/schema.yaml b/deploy/gke-marketplace-app/server-deployer/schema.yaml index 629303952d..ab21637bf3 100644 --- a/deploy/gke-marketplace-app/server-deployer/schema.yaml +++ b/deploy/gke-marketplace-app/server-deployer/schema.yaml @@ -27,7 +27,7 @@ x-google-marketplace: schemaVersion: v2 applicationApiVersion: v1beta1 - publishedVersion: '2.59.0' + publishedVersion: '2.59.1' publishedVersionMetadata: releaseNote: >- Initial release. @@ -89,7 +89,7 @@ properties: modelRepositoryPath: type: string title: Bucket where models are stored. Please make sure the user/service account to create the GKE app has permission to this GCS bucket. Read Triton documentation on configs and formatting details, supporting TensorRT, TensorFlow, Pytorch, Onnx ... etc. - default: gs://triton_sample_models/25.06 + default: gs://triton_sample_models/25.07 image.ldPreloadPath: type: string title: Leave this empty by default. Triton allows users to create custom layers for backend such as TensorRT plugin or Tensorflow custom ops, the compiled shared library must be provided via LD_PRELOAD environment variable. diff --git a/deploy/gke-marketplace-app/trt-engine/README.md b/deploy/gke-marketplace-app/trt-engine/README.md index dba63cf63e..c80da18f89 100644 --- a/deploy/gke-marketplace-app/trt-engine/README.md +++ b/deploy/gke-marketplace-app/trt-engine/README.md @@ -33,7 +33,7 @@ ``` docker run --gpus all -it --network host \ --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 \ - -v ~:/scripts nvcr.io/nvidia/tensorrt:25.06-py3 + -v ~:/scripts nvcr.io/nvidia/tensorrt:25.07-py3 pip install onnx six torch tf2onnx tensorflow @@ -57,7 +57,7 @@ mkdir -p engines python3 builder.py -m models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/model.ckpt -o engines/bert_large_int8_bs1_s128.engine -b 1 -s 128 -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/ -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/vocab.txt --int8 --fp16 --strict --calib-num 1 -iln -imh -gsutil cp bert_large_int8_bs1_s128.engine gs://triton_sample_models/25.06/bert/1/model.plan +gsutil cp bert_large_int8_bs1_s128.engine gs://triton_sample_models/25.07/bert/1/model.plan ``` -For each Triton upgrade, container version used to generate the model, and the model path in GCS `gs://triton_sample_models/25.06/` should be updated accordingly with the correct version. +For each Triton upgrade, container version used to generate the model, and the model path in GCS `gs://triton_sample_models/25.07/` should be updated accordingly with the correct version. diff --git a/deploy/k8s-onprem/values.yaml b/deploy/k8s-onprem/values.yaml index ac6d7a3480..84d6c62f28 100644 --- a/deploy/k8s-onprem/values.yaml +++ b/deploy/k8s-onprem/values.yaml @@ -30,7 +30,7 @@ tags: openshift: false image: - imageName: nvcr.io/nvidia/tritonserver:25.06-py3 + imageName: nvcr.io/nvidia/tritonserver:25.07-py3 pullPolicy: IfNotPresent modelRepositoryServer: < Replace with the IP Address of your file server > modelRepositoryPath: /srv/models diff --git a/deploy/oci/values.yaml b/deploy/oci/values.yaml index be6eb13a0e..093791e1be 100644 --- a/deploy/oci/values.yaml +++ b/deploy/oci/values.yaml @@ -27,7 +27,7 @@ replicaCount: 1 image: - imageName: nvcr.io/nvidia/tritonserver:25.06-py3 + imageName: nvcr.io/nvidia/tritonserver:25.07-py3 pullPolicy: IfNotPresent modelRepositoryPath: s3://https://.compat.objectstorage..oraclecloud.com:443/triton-inference-server-repository numGpus: 1 diff --git a/docs/introduction/compatibility.md b/docs/introduction/compatibility.md index 4332d4c8ae..450944756d 100644 --- a/docs/introduction/compatibility.md +++ b/docs/introduction/compatibility.md @@ -38,6 +38,7 @@ | Triton release version | NGC Tag | Python version | Torch version | TensorRT version | TensorRT-LLM version | CUDA version | CUDA Driver version | Size | | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 25.07 | nvcr.io/nvidia/tritonserver:25.07-trtllm-python-py3 | Python 3.12.3 | 2.7.0a0+79aa17489c.nv25.4 | 10.10.0.31 | 0.20.0 | 12.9.0.036 | 575.51.03 | 18.3G | | 25.06 | nvcr.io/nvidia/tritonserver:25.06-trtllm-python-py3 | Python 3.12.3 | 2.7.0a0+79aa17489c.nv25.4 | 10.10.0.31 | 0.20.0 | 12.9.0.036 | 575.51.03 | 18.3G | | 25.05 | nvcr.io/nvidia/tritonserver:25.05-trtllm-python-py3 | Python 3.12.3 | 2.7.0a0+7c8ec84dab.nv25.3 | 10.9.0.34 | 0.19.0 | 12.8.1.012 | 570.124.06 | 17G | | 25.04 | nvcr.io/nvidia/tritonserver:25.04-trtllm-python-py3 | Python 3.12.3 | 2.7.0a0+7c8ec84dab.nv25.3 | 10.9.0.34 | 0.18.2 | 12.8.1.012 | 570.124.06 | 17G | diff --git a/python/openai/README.md b/python/openai/README.md index a84f2ac4f4..4951ab7cd7 100644 --- a/python/openai/README.md +++ b/python/openai/README.md @@ -51,7 +51,7 @@ docker run -it --net=host --gpus all --rm \ -v ${HOME}/.cache/huggingface:/root/.cache/huggingface \ -e HF_TOKEN \ - nvcr.io/nvidia/tritonserver:25.06-vllm-python-py3 + nvcr.io/nvidia/tritonserver:25.07-vllm-python-py3 ``` 2. Launch the OpenAI-compatible Triton Inference Server: diff --git a/qa/common/gen_jetson_trt_models b/qa/common/gen_jetson_trt_models index b081fe9c4e..5bbfb4c74f 100755 --- a/qa/common/gen_jetson_trt_models +++ b/qa/common/gen_jetson_trt_models @@ -34,7 +34,7 @@ # Make all generated files accessible outside of container umask 0000 # Set the version of the models -TRITON_VERSION=${TRITON_VERSION:=25.06} +TRITON_VERSION=${TRITON_VERSION:=25.07} # Set the CUDA device to use CUDA_DEVICE=${RUNNER_ID:=0} # Set TensorRT image diff --git a/qa/common/gen_qa_custom_ops b/qa/common/gen_qa_custom_ops index 108b1daa53..6b3f349f44 100755 --- a/qa/common/gen_qa_custom_ops +++ b/qa/common/gen_qa_custom_ops @@ -37,7 +37,7 @@ ## ############################################################################ -TRITON_VERSION=${TRITON_VERSION:=25.06} +TRITON_VERSION=${TRITON_VERSION:=25.07} NVIDIA_UPSTREAM_VERSION=${NVIDIA_UPSTREAM_VERSION:=$TRITON_VERSION} PYTORCH_IMAGE=${PYTORCH_IMAGE:=nvcr.io/nvidia/pytorch:$NVIDIA_UPSTREAM_VERSION-py3} UBUNTU_IMAGE=${UBUNTU_IMAGE:=ubuntu:24.04} diff --git a/qa/common/gen_qa_model_repository b/qa/common/gen_qa_model_repository index 2deeb97da5..8660124fc5 100755 --- a/qa/common/gen_qa_model_repository +++ b/qa/common/gen_qa_model_repository @@ -48,7 +48,7 @@ ## ############################################################################ -TRITON_VERSION=${TRITON_VERSION:=25.06} +TRITON_VERSION=${TRITON_VERSION:=25.07} # ONNX. Use ONNX_OPSET 0 to use the default for ONNX version ONNX_VERSION=1.16.1 From bbb37953cd9cf008e9e67d327fa18a03b5cea9d1 Mon Sep 17 00:00:00 2001 From: Misha Chornyi <99709299+mc-nv@users.noreply.github.com> Date: Fri, 25 Jul 2025 16:49:23 -0700 Subject: [PATCH 12/14] refactor: Enhance memory allocation security in HTTP and Sagemaker request handler (#8305) (#8314) Co-authored-by: Yingge He <157551214+yinggeh@users.noreply.github.com> --- qa/L0_http/http_request_many_chunks.py | 153 +++++ qa/L0_http/test.sh | 29 +- .../sagemaker_request_many_chunks.py | 91 +++ qa/L0_sagemaker/test.sh | 47 ++ src/http_server.cc | 547 ++++++++---------- src/http_server.h | 30 +- src/sagemaker_server.cc | 83 +-- 7 files changed, 606 insertions(+), 374 deletions(-) create mode 100755 qa/L0_http/http_request_many_chunks.py create mode 100755 qa/L0_sagemaker/sagemaker_request_many_chunks.py diff --git a/qa/L0_http/http_request_many_chunks.py b/qa/L0_http/http_request_many_chunks.py new file mode 100755 index 0000000000..a67d9a82c5 --- /dev/null +++ b/qa/L0_http/http_request_many_chunks.py @@ -0,0 +1,153 @@ +#!/usr/bin/python +# Copyright 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of NVIDIA CORPORATION nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY +# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY +# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +import socket +import unittest + + +class HTTPRequestManyChunksTest(unittest.TestCase): + def setUp(self): + self._model_name = "simple" + self._local_host = "localhost" + self._http_port = 8000 + self._malicious_chunk_count = ( + 1000000 # large enough to cause a stack overflow if using alloca() + ) + self._parse_error = ( + "failed to parse the request JSON buffer: Invalid value. at 0" + ) + + def send_chunked_request( + self, header: str, chunk_count: int, expected_response: str + ): + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + header = ( + f"{header}" + f"Host: {self._local_host}:{self._http_port}\r\n" + f"Content-Type: application/octet-stream\r\n" + f"Transfer-Encoding: chunked\r\n" + f"Connection: close\r\n" + f"\r\n" + ) + try: + s.connect((self._local_host, self._http_port)) + # HTTP request with chunked encoding + s.sendall((header.encode())) + + # Send chunked payload + for _ in range(chunk_count): + s.send(b"1\r\nA\r\n") + # End chunked encoding + s.sendall(b"0\r\n\r\n") + + # Receive response + response = b"" + while True: + try: + chunk = s.recv(4096) + if not chunk: + break + response += chunk + except socket.timeout: + break + self.assertIn(expected_response, response.decode()) + except Exception as e: + raise (e) + finally: + s.close() + + def test_infer(self): + request_header = ( + f"POST /v2/models/{self._model_name}/infer HTTP/1.1\r\n" + f"Inference-Header-Content-Length: 0\r\n" + ) + + self.send_chunked_request( + request_header, + self._malicious_chunk_count, + "Raw request must only have 1 input (found 1) to be deduced but got 2 inputs in 'simple' model configuration", + ) + + def test_registry_index(self): + request_header = f"POST /v2/repository/index HTTP/1.1\r\n" + + self.send_chunked_request( + request_header, self._malicious_chunk_count, self._parse_error + ) + + def test_model_control(self): + load_request_header = ( + f"POST /v2/repository/models/{self._model_name}/load HTTP/1.1\r\n" + ) + unload_request_header = load_request_header.replace("/load", "/unload") + + self.send_chunked_request( + load_request_header, self._malicious_chunk_count, self._parse_error + ) + self.send_chunked_request( + unload_request_header, self._malicious_chunk_count, self._parse_error + ) + + def test_trace(self): + request_header = ( + f"POST /v2/models/{self._model_name}/trace/setting HTTP/1.1\r\n" + ) + + self.send_chunked_request( + request_header, self._malicious_chunk_count, self._parse_error + ) + + def test_logging(self): + request_header = f"POST /v2/logging HTTP/1.1\r\n" + + self.send_chunked_request( + request_header, self._malicious_chunk_count, self._parse_error + ) + + def test_system_shm_register(self): + request_header = f"POST /v2/systemsharedmemory/region/test_system_shm_register/register HTTP/1.1\r\n" + + self.send_chunked_request( + request_header, self._malicious_chunk_count, self._parse_error + ) + + def test_cuda_shm_register(self): + request_header = f"POST /v2/cudasharedmemory/region/test_cuda_shm_register/register HTTP/1.1\r\n" + + self.send_chunked_request( + request_header, self._malicious_chunk_count, self._parse_error + ) + + def test_generate(self): + request_header = f"POST /v2/models/{self._model_name}/generate HTTP/1.1\r\n" + self.send_chunked_request( + request_header, self._malicious_chunk_count, self._parse_error + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/qa/L0_http/test.sh b/qa/L0_http/test.sh index a96bd6a1c6..0fc25501a3 100755 --- a/qa/L0_http/test.sh +++ b/qa/L0_http/test.sh @@ -40,6 +40,7 @@ fi export CUDA_VISIBLE_DEVICES=0 +source ../common/util.sh RET=0 CLIENT_PLUGIN_TEST="./http_client_plugin_test.py" @@ -129,7 +130,6 @@ set -e CLIENT_LOG=`pwd`/client.log SERVER_ARGS="--backend-directory=${BACKEND_DIR} --model-repository=${MODELDIR}" -source ../common/util.sh run_server if [ "$SERVER_PID" == "0" ]; then @@ -855,7 +855,32 @@ elif [ `grep -c "Error: --http-max-input-size must be greater than 0." ${SERVER_ RET=1 fi -### +### Test HTTP Requests Containing Many Chunks ### +MODELDIR="`pwd`/models" +REQUEST_MANY_CHUNKS_PY="http_request_many_chunks.py" +CLIENT_LOG="./client.http_request_many_chunks.log" +SERVER_ARGS="--model-repository=${MODELDIR} --log-verbose=1 --model-control-mode=explicit --load-model=simple" +SERVER_LOG="./inference_server_request_many_chunks.log" + +run_server +if [ "$SERVER_PID" == "0" ]; then + echo -e "\n***\n*** Failed to start $SERVER\n***" + cat $SERVER_LOG + exit 1 +fi + +set +e +python $REQUEST_MANY_CHUNKS_PY -v >> ${CLIENT_LOG} 2>&1 +if [ $? -ne 0 ]; then + echo -e "\n***\n*** HTTP Request Many Chunks Test Failed\n***" + cat $SERVER_LOG + cat $CLIENT_LOG + RET=1 +fi +set -e + +kill $SERVER_PID +wait $SERVER_PID if [ $RET -eq 0 ]; then echo -e "\n***\n*** Test Passed\n***" diff --git a/qa/L0_sagemaker/sagemaker_request_many_chunks.py b/qa/L0_sagemaker/sagemaker_request_many_chunks.py new file mode 100755 index 0000000000..415ddcba28 --- /dev/null +++ b/qa/L0_sagemaker/sagemaker_request_many_chunks.py @@ -0,0 +1,91 @@ +#!/usr/bin/python +# Copyright 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of NVIDIA CORPORATION nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY +# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY +# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +import socket +import unittest + + +class SagemakerRequestManyChunksTest(unittest.TestCase): + def setUp(self): + self._local_host = "localhost" + self._sagemaker_port = 8080 + self._malicious_chunk_count = ( + 1000000 # large enough to cause a stack overflow if using alloca() + ) + + def send_chunked_request( + self, header: str, chunk_count: int, expected_response: str + ): + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + header = ( + f"{header}" + f"Host: {self._local_host}:{self._sagemaker_port}\r\n" + f"Content-Type: application/octet-stream\r\n" + f"Transfer-Encoding: chunked\r\n" + f"Connection: close\r\n" + f"\r\n" + ) + try: + s.connect((self._local_host, self._sagemaker_port)) + # HTTP request with chunked encoding + s.sendall((header.encode())) + + # Send chunked payload + for _ in range(chunk_count): + s.send(b"1\r\nA\r\n") + # End chunked encoding + s.sendall(b"0\r\n\r\n") + + # Receive response + response = b"" + while True: + try: + chunk = s.recv(4096) + if not chunk: + break + response += chunk + except socket.timeout: + break + self.assertIn(expected_response, response.decode()) + except Exception as e: + raise (e) + finally: + s.close() + + def test_load_model(self): + request_header = ( + f"POST /models HTTP/1.1\r\n" f"X-Amzn-SageMaker-Target-Model: ZZZZZZZ\r\n" + ) + self.send_chunked_request( + request_header, + self._malicious_chunk_count, + "failed to parse the request JSON buffer: Invalid value. at 0", + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/qa/L0_sagemaker/test.sh b/qa/L0_sagemaker/test.sh index dde4c794d8..c86b278e38 100755 --- a/qa/L0_sagemaker/test.sh +++ b/qa/L0_sagemaker/test.sh @@ -565,6 +565,53 @@ kill $SERVER_PID wait $SERVE_PID # MME end +### Test Sagemaker Requests Containing Many Chunks ### +rm -rf models && mkdir models && \ + cp -r $DATADIR/qa_model_repository/onnx_int32_int32_int32 models/sm_model && \ + rm -r models/sm_model/2 && rm -r models/sm_model/3 && \ + sed -i "s/onnx_int32_int32_int32/sm_model/" models/sm_model/config.pbtxt + +export SAGEMAKER_TRITON_DEFAULT_MODEL_NAME=sm_model +REQUEST_MANY_CHUNKS_PY="sagemaker_request_many_chunks.py" +CLIENT_LOG="./client.sagemaker_request_many_chunks.log" +SERVER_LOG="./server.sagemaker_request_many_chunks.log" + +serve > $SERVER_LOG 2>&1 & +SERVE_PID=$! +# Obtain Triton PID in such way as $! will return the script PID +sleep 1 +SERVER_PID=`ps | grep tritonserver | awk '{ printf $1 }'` +sagemaker_wait_for_server_ready $SERVER_PID 10 +if [ "$WAIT_RET" != "0" ]; then + echo -e "\n***\n*** Failed to start $SERVER\n***" + kill $SERVER_PID || true + cat $SERVER_LOG + exit 1 +fi + +# Ping +set +e +code=`curl -s -w %{http_code} -o ./ping.out localhost:8080/ping` +set -e +if [ "$code" != "200" ]; then + cat ./ping.out + echo -e "\n***\n*** Test Failed\n***" + RET=1 +fi + +set +e +python $REQUEST_MANY_CHUNKS_PY >>$CLIENT_LOG 2>&1 +if [ $? -ne 0 ]; then + echo -e "\n***\n*** Sagemaker Request Many Chunks Test Failed\n***" + cat $SERVER_LOG + cat $CLIENT_LOG + RET=1 +fi +set -e + +kill $SERVER_PID +wait $SERVE_PID + unlink /opt/ml/model rm -rf /opt/ml/model diff --git a/src/http_server.cc b/src/http_server.cc index e3ee3efe17..f62f73c948 100644 --- a/src/http_server.cc +++ b/src/http_server.cc @@ -1053,62 +1053,6 @@ ValidateOutputParameter(triton::common::TritonJson::Value& io) return nullptr; // success } -TRITONSERVER_Error* -EVBufferToJson( - triton::common::TritonJson::Value* document, evbuffer_iovec* v, int* v_idx, - const size_t length, int n) -{ - size_t offset = 0, remaining_length = length; - char* json_base; - std::vector json_buffer; - - // No need to memcpy when number of iovecs is 1 - if ((n > 0) && (v[0].iov_len >= remaining_length)) { - json_base = static_cast(v[0].iov_base); - if (v[0].iov_len > remaining_length) { - v[0].iov_base = static_cast(json_base + remaining_length); - v[0].iov_len -= remaining_length; - remaining_length = 0; - } else if (v[0].iov_len == remaining_length) { - remaining_length = 0; - *v_idx += 1; - } - } else { - json_buffer.resize(length); - json_base = json_buffer.data(); - while ((remaining_length > 0) && (*v_idx < n)) { - char* base = static_cast(v[*v_idx].iov_base); - size_t base_size; - if (v[*v_idx].iov_len > remaining_length) { - base_size = remaining_length; - v[*v_idx].iov_base = static_cast(base + remaining_length); - v[*v_idx].iov_len -= remaining_length; - remaining_length = 0; - } else { - base_size = v[*v_idx].iov_len; - remaining_length -= v[*v_idx].iov_len; - *v_idx += 1; - } - - memcpy(json_base + offset, base, base_size); - offset += base_size; - } - } - - if (remaining_length != 0) { - return TRITONSERVER_ErrorNew( - TRITONSERVER_ERROR_INVALID_ARG, - std::string( - "unexpected size for request JSON, expecting " + - std::to_string(remaining_length) + " more bytes") - .c_str()); - } - - RETURN_IF_ERR(document->Parse(json_base, length)); - - return nullptr; // success -} - std::string CompressionTypeUsed(const std::string accept_encoding) { @@ -1432,34 +1376,16 @@ HTTPAPIServer::HandleRepositoryIndex( } TRITONSERVER_Error* err = nullptr; - - struct evbuffer_iovec* v = nullptr; - int v_idx = 0; - int n = evbuffer_peek(req->buffer_in, -1, NULL, NULL, 0); - if (n > 0) { - v = static_cast( - alloca(sizeof(struct evbuffer_iovec) * n)); - if (evbuffer_peek(req->buffer_in, -1, NULL, v, n) != n) { - err = TRITONSERVER_ErrorNew( - TRITONSERVER_ERROR_INTERNAL, - "unexpected error getting registry index request body"); - } - } - + triton::common::TritonJson::Value index_request; bool ready = false; + size_t buffer_len = 0; + RETURN_AND_RESPOND_IF_ERR( + req, EVRequestToJson(req, "registry index", &index_request, &buffer_len)); - if (err == nullptr) { - // If no request json then just use all default values. - size_t buffer_len = evbuffer_get_length(req->buffer_in); - if (buffer_len > 0) { - triton::common::TritonJson::Value index_request; - err = EVBufferToJson(&index_request, v, &v_idx, buffer_len, n); - if (err == nullptr) { - triton::common::TritonJson::Value ready_json; - if (index_request.Find("ready", &ready_json)) { - err = ready_json.AsBool(&ready); - } - } + if (buffer_len > 0) { + triton::common::TritonJson::Value ready_json; + if (index_request.Find("ready", &ready_json)) { + err = ready_json.AsBool(&ready); } } @@ -1508,19 +1434,6 @@ HTTPAPIServer::HandleRepositoryControl( "'repository_name' specification is not supported"); } else { if (action == "load") { - struct evbuffer_iovec* v = nullptr; - int v_idx = 0; - int n = evbuffer_peek(req->buffer_in, -1, NULL, NULL, 0); - if (n > 0) { - v = static_cast( - alloca(sizeof(struct evbuffer_iovec) * n)); - if (evbuffer_peek(req->buffer_in, -1, NULL, v, n) != n) { - RETURN_AND_RESPOND_IF_ERR( - req, TRITONSERVER_ErrorNew( - TRITONSERVER_ERROR_INTERNAL, - "unexpected error getting load model request buffers")); - } - } static auto param_deleter = [](std::vector* params) { if (params != nullptr) { @@ -1538,15 +1451,15 @@ HTTPAPIServer::HandleRepositoryControl( std::list> binary_files; // WAR for the const-ness check std::vector const_params; - size_t buffer_len = evbuffer_get_length(req->buffer_in); - if (buffer_len > 0) { - triton::common::TritonJson::Value request; - RETURN_AND_RESPOND_IF_ERR( - req, EVBufferToJson(&request, v, &v_idx, buffer_len, n)); + triton::common::TritonJson::Value load_request; + size_t buffer_len = 0; + RETURN_AND_RESPOND_IF_ERR( + req, EVRequestToJson(req, "load model", &load_request, &buffer_len)); + if (buffer_len > 0) { // Parse request body for parameters triton::common::TritonJson::Value param_json; - if (request.Find("parameters", ¶m_json)) { + if (load_request.Find("parameters", ¶m_json)) { // Iterate over each member in 'param_json' std::vector members; RETURN_AND_RESPOND_IF_ERR(req, param_json.Members(&members)); @@ -1592,37 +1505,25 @@ HTTPAPIServer::HandleRepositoryControl( // Check if the dependent models should be removed bool unload_dependents = false; { - struct evbuffer_iovec* v = nullptr; - int v_idx = 0; - int n = evbuffer_peek(req->buffer_in, -1, NULL, NULL, 0); - if (n > 0) { - v = static_cast( - alloca(sizeof(struct evbuffer_iovec) * n)); - if (evbuffer_peek(req->buffer_in, -1, NULL, v, n) != n) { - err = TRITONSERVER_ErrorNew( - TRITONSERVER_ERROR_INTERNAL, - "unexpected error getting model control request body"); - } - } + triton::common::TritonJson::Value control_request; + size_t buffer_len = 0; + RETURN_AND_RESPOND_IF_ERR( + req, EVRequestToJson( + req, "unload model", &control_request, &buffer_len)); - size_t buffer_len = evbuffer_get_length(req->buffer_in); if (buffer_len > 0) { - triton::common::TritonJson::Value control_request; - err = EVBufferToJson(&control_request, v, &v_idx, buffer_len, n); - if (err == nullptr) { - triton::common::TritonJson::Value params_json; - if (control_request.Find("parameters", ¶ms_json)) { - triton::common::TritonJson::Value ud_json; - if (params_json.Find("unload_dependents", &ud_json)) { - auto parse_err = ud_json.AsBool(&unload_dependents); - if (parse_err != nullptr) { - err = TRITONSERVER_ErrorNew( - TRITONSERVER_ErrorCode(parse_err), - (std::string("Unable to parse 'unload_dependents': ") + - TRITONSERVER_ErrorMessage(parse_err)) - .c_str()); - TRITONSERVER_ErrorDelete(parse_err); - } + triton::common::TritonJson::Value params_json; + if (control_request.Find("parameters", ¶ms_json)) { + triton::common::TritonJson::Value ud_json; + if (params_json.Find("unload_dependents", &ud_json)) { + auto parse_err = ud_json.AsBool(&unload_dependents); + if (parse_err != nullptr) { + err = TRITONSERVER_ErrorNew( + TRITONSERVER_ErrorCode(parse_err), + (std::string("Unable to parse 'unload_dependents': ") + + TRITONSERVER_ErrorMessage(parse_err)) + .c_str()); + TRITONSERVER_ErrorDelete(parse_err); } } } @@ -1868,42 +1769,27 @@ HTTPAPIServer::HandleTrace(evhtp_request_t* req, const std::string& model_name) // Perform trace setting update if requested if (req->method == htp_method_POST) { - struct evbuffer_iovec* v = nullptr; - int v_idx = 0; - int n = evbuffer_peek(req->buffer_in, -1, NULL, NULL, 0); - if (n > 0) { - v = static_cast( - alloca(sizeof(struct evbuffer_iovec) * n)); - if (evbuffer_peek(req->buffer_in, -1, NULL, v, n) != n) { - RETURN_AND_RESPOND_IF_ERR( - req, TRITONSERVER_ErrorNew( - TRITONSERVER_ERROR_INTERNAL, - "unexpected error getting trace request buffers")); - } - } - - triton::common::TritonJson::Value request; - size_t buffer_len = evbuffer_get_length(req->buffer_in); + triton::common::TritonJson::Value trace_request; RETURN_AND_RESPOND_IF_ERR( - req, EVBufferToJson(&request, v, &v_idx, buffer_len, n)); + req, EVRequestToJsonAllowsEmpty(req, "trace", &trace_request)); TraceManager::NewSetting new_setting; triton::common::TritonJson::Value setting_json; - if (request.Find("trace_file", &setting_json)) { + if (trace_request.Find("trace_file", &setting_json)) { RETURN_AND_RESPOND_IF_ERR( req, TRITONSERVER_ErrorNew( TRITONSERVER_ERROR_UNSUPPORTED, "trace file location can not be updated through network " "protocol")); } - if (request.Find("trace_level", &setting_json)) { + if (trace_request.Find("trace_level", &setting_json)) { if (setting_json.IsNull()) { new_setting.clear_level_ = true; } else { triton::common::TritonJson::Value level_array; RETURN_AND_RESPOND_IF_ERR( - req, request.MemberAsArray("trace_level", &level_array)); + req, trace_request.MemberAsArray("trace_level", &level_array)); for (size_t i = 0; i < level_array.ArraySize(); ++i) { std::string level_str; RETURN_AND_RESPOND_IF_ERR( @@ -1930,7 +1816,7 @@ HTTPAPIServer::HandleTrace(evhtp_request_t* req, const std::string& model_name) } } } - if (request.Find("trace_rate", &setting_json)) { + if (trace_request.Find("trace_rate", &setting_json)) { if (setting_json.IsNull()) { new_setting.clear_rate_ = true; } else { @@ -1963,7 +1849,7 @@ HTTPAPIServer::HandleTrace(evhtp_request_t* req, const std::string& model_name) } } } - if (request.Find("trace_count", &setting_json)) { + if (trace_request.Find("trace_count", &setting_json)) { if (setting_json.IsNull()) { new_setting.clear_count_ = true; } else { @@ -2005,7 +1891,7 @@ HTTPAPIServer::HandleTrace(evhtp_request_t* req, const std::string& model_name) } } } - if (request.Find("log_frequency", &setting_json)) { + if (trace_request.Find("log_frequency", &setting_json)) { if (setting_json.IsNull()) { new_setting.clear_log_frequency_ = true; } else { @@ -2128,28 +2014,13 @@ HTTPAPIServer::HandleLogging(evhtp_request_t* req) #ifdef TRITON_ENABLE_LOGGING // Perform log setting update if requested if (req->method == htp_method_POST) { - struct evbuffer_iovec* v = nullptr; - int v_idx = 0; - int n = evbuffer_peek(req->buffer_in, -1, NULL, NULL, 0); - if (n > 0) { - v = static_cast( - alloca(sizeof(struct evbuffer_iovec) * n)); - if (evbuffer_peek(req->buffer_in, -1, NULL, v, n) != n) { - RETURN_AND_RESPOND_IF_ERR( - req, - TRITONSERVER_ErrorNew( - TRITONSERVER_ERROR_INTERNAL, - "unexpected error getting dynamic logging request buffers")); - } - } - triton::common::TritonJson::Value request; - size_t buffer_len = evbuffer_get_length(req->buffer_in); + triton::common::TritonJson::Value log_request; RETURN_AND_RESPOND_IF_ERR( - req, EVBufferToJson(&request, v, &v_idx, buffer_len, n)); + req, EVRequestToJsonAllowsEmpty(req, "dynamic logging", &log_request)); // Server and Core repos do not have the same Logger object // Each update must be applied to both server and core repo versions triton::common::TritonJson::Value setting_json; - if (request.Find("log_file", &setting_json)) { + if (log_request.Find("log_file", &setting_json)) { if (!setting_json.IsNull()) { RETURN_AND_RESPOND_IF_ERR( req, TRITONSERVER_ErrorNew( @@ -2158,7 +2029,7 @@ HTTPAPIServer::HandleLogging(evhtp_request_t* req) "protocol")); } } - if (request.Find("log_info", &setting_json)) { + if (log_request.Find("log_info", &setting_json)) { if (!setting_json.IsNull()) { bool log_info_status; RETURN_AND_RESPOND_IF_ERR(req, setting_json.AsBool(&log_info_status)); @@ -2166,7 +2037,7 @@ HTTPAPIServer::HandleLogging(evhtp_request_t* req) TRITONSERVER_ServerOptionsSetLogInfo(nullptr, log_info_status); } } - if (request.Find("log_warning", &setting_json)) { + if (log_request.Find("log_warning", &setting_json)) { if (!setting_json.IsNull()) { bool log_warn_status; RETURN_AND_RESPOND_IF_ERR(req, setting_json.AsBool(&log_warn_status)); @@ -2174,7 +2045,7 @@ HTTPAPIServer::HandleLogging(evhtp_request_t* req) TRITONSERVER_ServerOptionsSetLogWarn(nullptr, log_warn_status); } } - if (request.Find("log_error", &setting_json)) { + if (log_request.Find("log_error", &setting_json)) { if (!setting_json.IsNull()) { bool log_error_status; RETURN_AND_RESPOND_IF_ERR(req, setting_json.AsBool(&log_error_status)); @@ -2182,7 +2053,7 @@ HTTPAPIServer::HandleLogging(evhtp_request_t* req) TRITONSERVER_ServerOptionsSetLogError(nullptr, log_error_status); } } - if (request.Find("log_verbose_level", &setting_json)) { + if (log_request.Find("log_verbose_level", &setting_json)) { if (!setting_json.IsNull()) { uint64_t verbose_level; RETURN_AND_RESPOND_IF_ERR(req, setting_json.AsUInt(&verbose_level)); @@ -2191,7 +2062,7 @@ HTTPAPIServer::HandleLogging(evhtp_request_t* req) nullptr, static_cast(verbose_level)); } } - if (request.Find("log_format", &setting_json)) { + if (log_request.Find("log_format", &setting_json)) { if (!setting_json.IsNull()) { std::string log_format_parse; RETURN_AND_RESPOND_IF_ERR( @@ -2310,63 +2181,46 @@ HTTPAPIServer::HandleSystemSharedMemory( TRITONSERVER_ERROR_INVALID_ARG, "'region name' is necessary to register system shared memory region"); } else { - struct evbuffer_iovec* v = nullptr; - int v_idx = 0; - int n = evbuffer_peek(req->buffer_in, -1, NULL, NULL, 0); - if (n > 0) { - v = static_cast( - alloca(sizeof(struct evbuffer_iovec) * n)); - if (evbuffer_peek(req->buffer_in, -1, NULL, v, n) != n) { - err = TRITONSERVER_ErrorNew( - TRITONSERVER_ERROR_INTERNAL, - "unexpected error getting register request buffers"); - } + triton::common::TritonJson::Value register_request; + triton::common::TritonJson::Value key_json; + RETURN_AND_RESPOND_IF_ERR( + req, EVRequestToJsonAllowsEmpty(req, action, ®ister_request)); + if (!register_request.Find("key", &key_json)) { + err = TRITONSERVER_ErrorNew( + TRITONSERVER_ERROR_INVALID_ARG, + "Shared memory register request has no 'key' field"); } + const char* shm_key = nullptr; if (err == nullptr) { - triton::common::TritonJson::Value register_request; - size_t buffer_len = evbuffer_get_length(req->buffer_in); - err = EVBufferToJson(®ister_request, v, &v_idx, buffer_len, n); - if (err == nullptr) { - triton::common::TritonJson::Value key_json; - if (!register_request.Find("key", &key_json)) { - err = TRITONSERVER_ErrorNew( - TRITONSERVER_ERROR_INVALID_ARG, - "Shared memory register request has no 'key' field"); - } - - const char* shm_key = nullptr; - if (err == nullptr) { - size_t shm_key_len; - err = key_json.AsString(&shm_key, &shm_key_len); - } - - uint64_t offset = 0; - if (err == nullptr) { - triton::common::TritonJson::Value offset_json; - if (register_request.Find("offset", &offset_json)) { - err = offset_json.AsUInt(&offset); - } - } + size_t shm_key_len; + err = key_json.AsString(&shm_key, &shm_key_len); + } - uint64_t byte_size = 0; - if (err == nullptr) { - triton::common::TritonJson::Value byte_size_json; - if (!register_request.Find("byte_size", &byte_size_json)) { - err = TRITONSERVER_ErrorNew( - TRITONSERVER_ERROR_INVALID_ARG, - "Shared memory register request has no 'byte_size' field"); - } else { - err = byte_size_json.AsUInt(&byte_size); - } - } + uint64_t offset = 0; + if (err == nullptr) { + triton::common::TritonJson::Value offset_json; + if (register_request.Find("offset", &offset_json)) { + err = offset_json.AsUInt(&offset); + } + } - if (err == nullptr) { - err = shm_manager_->RegisterSystemSharedMemory( - region_name, shm_key, offset, byte_size); - } + uint64_t byte_size = 0; + if (err == nullptr) { + triton::common::TritonJson::Value byte_size_json; + if (!register_request.Find("byte_size", &byte_size_json)) { + err = TRITONSERVER_ErrorNew( + TRITONSERVER_ERROR_INVALID_ARG, + "Shared memory register request has no 'byte_size' field"); + } else { + err = byte_size_json.AsUInt(&byte_size); } } + + if (err == nullptr) { + err = shm_manager_->RegisterSystemSharedMemory( + region_name, shm_key, offset, byte_size); + } } } else if (action == "unregister") { if (region_name.empty()) { @@ -2417,67 +2271,50 @@ HTTPAPIServer::HandleCudaSharedMemory( "'region name' is necessary to register cuda shared memory region"); } else { #ifdef TRITON_ENABLE_GPU - struct evbuffer_iovec* v = nullptr; - int v_idx = 0; - int n = evbuffer_peek(req->buffer_in, -1, NULL, NULL, 0); - if (n > 0) { - v = static_cast( - alloca(sizeof(struct evbuffer_iovec) * n)); - if (evbuffer_peek(req->buffer_in, -1, NULL, v, n) != n) { - err = TRITONSERVER_ErrorNew( - TRITONSERVER_ERROR_INTERNAL, - "unexpected error getting register request buffers"); - } + triton::common::TritonJson::Value register_request; + RETURN_AND_RESPOND_IF_ERR( + req, EVRequestToJsonAllowsEmpty(req, action, ®ister_request)); + const char* b64_handle = nullptr; + size_t b64_handle_len = 0; + triton::common::TritonJson::Value raw_handle_json; + if (!register_request.Find("raw_handle", &raw_handle_json)) { + err = TRITONSERVER_ErrorNew( + TRITONSERVER_ERROR_INVALID_ARG, + "Shared memory register request has no 'raw_handle' field"); + } else { + err = + raw_handle_json.MemberAsString("b64", &b64_handle, &b64_handle_len); } - if (err == nullptr) { - triton::common::TritonJson::Value register_request; - size_t buffer_len = evbuffer_get_length(req->buffer_in); - err = EVBufferToJson(®ister_request, v, &v_idx, buffer_len, n); - if (err == nullptr) { - const char* b64_handle = nullptr; - size_t b64_handle_len = 0; - triton::common::TritonJson::Value raw_handle_json; - if (!register_request.Find("raw_handle", &raw_handle_json)) { - err = TRITONSERVER_ErrorNew( - TRITONSERVER_ERROR_INVALID_ARG, - "Shared memory register request has no 'raw_handle' field"); - } else { - err = raw_handle_json.MemberAsString( - "b64", &b64_handle, &b64_handle_len); - } - uint64_t byte_size = 0; - if (err == nullptr) { - err = register_request.MemberAsUInt("byte_size", &byte_size); - } + uint64_t byte_size = 0; + if (err == nullptr) { + err = register_request.MemberAsUInt("byte_size", &byte_size); + } - uint64_t device_id = 0; - if (err == nullptr) { - err = register_request.MemberAsUInt("device_id", &device_id); - } + uint64_t device_id = 0; + if (err == nullptr) { + err = register_request.MemberAsUInt("device_id", &device_id); + } - if (err == nullptr) { - size_t decoded_size; - std::vector raw_handle; - RETURN_AND_RESPOND_IF_ERR( - req, DecodeBase64( - b64_handle, b64_handle_len, raw_handle, decoded_size, - "raw_handle")); + if (err == nullptr) { + size_t decoded_size; + std::vector raw_handle; + RETURN_AND_RESPOND_IF_ERR( + req, DecodeBase64( + b64_handle, b64_handle_len, raw_handle, decoded_size, + "raw_handle")); - if (decoded_size != sizeof(cudaIpcMemHandle_t)) { - err = TRITONSERVER_ErrorNew( - TRITONSERVER_ERROR_INVALID_ARG, - "'raw_handle' must be a valid base64 encoded " - "cudaIpcMemHandle_t"); - } else { - raw_handle.resize(sizeof(cudaIpcMemHandle_t)); - err = shm_manager_->RegisterCUDASharedMemory( - region_name.c_str(), - reinterpret_cast( - raw_handle.data()), - byte_size, device_id); - } - } + if (decoded_size != sizeof(cudaIpcMemHandle_t)) { + err = TRITONSERVER_ErrorNew( + TRITONSERVER_ERROR_INVALID_ARG, + "'raw_handle' must be a valid base64 encoded " + "cudaIpcMemHandle_t"); + } else { + raw_handle.resize(sizeof(cudaIpcMemHandle_t)); + err = shm_manager_->RegisterCUDASharedMemory( + region_name.c_str(), + reinterpret_cast(raw_handle.data()), + byte_size, device_id); } } #else @@ -3028,25 +2865,50 @@ HTTPAPIServer::ParseJsonTritonRequestID( return nullptr; // Success } -// TODO: Can refactor other non-inference routes to re-use this helper instead. TRITONSERVER_Error* -HTTPAPIServer::EVRequestToJson( - evhtp_request_t* req, triton::common::TritonJson::Value* request_json_ptr) +HTTPAPIServer::EVRequestToJsonImpl( + evhtp_request_t* req, std::string_view request_kind, bool allows_empty_body, + triton::common::TritonJson::Value* request_json, size_t* buffer_len) { struct evbuffer_iovec* v = nullptr; int v_idx = 0; + std::vector v_vec; + int n = evbuffer_peek(req->buffer_in, -1, NULL, NULL, 0); if (n > 0) { - v = static_cast( - alloca(sizeof(struct evbuffer_iovec) * n)); + try { + v_vec = std::vector(n); + } + catch (const std::bad_alloc& e) { + // Handle memory allocation failure + return TRITONSERVER_ErrorNew( + TRITONSERVER_ERROR_INVALID_ARG, + (std::string("Memory allocation failed for evbuffer: ") + e.what()) + .c_str()); + } + catch (const std::exception& e) { + // Catch any other std exceptions + return TRITONSERVER_ErrorNew( + TRITONSERVER_ERROR_INTERNAL, + (std::string("Exception while creating evbuffer vector: ") + e.what()) + .c_str()); + } + + v = v_vec.data(); if (evbuffer_peek(req->buffer_in, -1, NULL, v, n) != n) { return TRITONSERVER_ErrorNew( TRITONSERVER_ERROR_INTERNAL, - "Unexpected error getting request buffers"); + ("Unexpected error getting " + std::string(request_kind) + + " request buffers") + .c_str()); } } - size_t buffer_len = evbuffer_get_length(req->buffer_in); - RETURN_IF_ERR(EVBufferToJson(request_json_ptr, v, &v_idx, buffer_len, n)); + + *buffer_len = evbuffer_get_length(req->buffer_in); + if (allows_empty_body || *buffer_len > 0) { + RETURN_IF_ERR(EVBufferToJson(request_json, v, &v_idx, *buffer_len, n)); + } + return nullptr; // success } @@ -3063,11 +2925,29 @@ HTTPAPIServer::EVBufferToInput( // body. struct evbuffer_iovec* v = nullptr; int v_idx = 0; + std::vector v_vec; int n = evbuffer_peek(input_buffer, -1, NULL, NULL, 0); if (n > 0) { - v = static_cast( - alloca(sizeof(struct evbuffer_iovec) * n)); + try { + v_vec = std::vector(n); + } + catch (const std::bad_alloc& e) { + // Handle memory allocation failure + return TRITONSERVER_ErrorNew( + TRITONSERVER_ERROR_INVALID_ARG, + (std::string("Memory allocation failed for evbuffer: ") + e.what()) + .c_str()); + } + catch (const std::exception& e) { + // Catch any other std exceptions + return TRITONSERVER_ErrorNew( + TRITONSERVER_ERROR_INTERNAL, + (std::string("Exception while creating evbuffer vector: ") + e.what()) + .c_str()); + } + + v = v_vec.data(); if (evbuffer_peek(input_buffer, -1, NULL, v, n) != n) { return TRITONSERVER_ErrorNew( TRITONSERVER_ERROR_INTERNAL, @@ -3120,10 +3000,30 @@ HTTPAPIServer::EVBufferToRawInput( } else { struct evbuffer_iovec* v = nullptr; int v_idx = 0; + std::vector v_vec; + int n = evbuffer_peek(input_buffer, -1, NULL, NULL, 0); if (n > 0) { - v = static_cast( - alloca(sizeof(struct evbuffer_iovec) * n)); + try { + v_vec = std::vector(n); + } + catch (const std::bad_alloc& e) { + // Handle memory allocation failure + return TRITONSERVER_ErrorNew( + TRITONSERVER_ERROR_INVALID_ARG, + (std::string("Memory allocation failed for evbuffer: ") + e.what()) + .c_str()); + } + catch (const std::exception& e) { + // Catch any other std exceptions + return TRITONSERVER_ErrorNew( + TRITONSERVER_ERROR_INTERNAL, + (std::string("Exception while creating evbuffer vector: ") + + e.what()) + .c_str()); + } + + v = v_vec.data(); if (evbuffer_peek(input_buffer, -1, NULL, v, n) != n) { return TRITONSERVER_ErrorNew( TRITONSERVER_ERROR_INTERNAL, @@ -3155,6 +3055,62 @@ HTTPAPIServer::EVBufferToRawInput( return nullptr; // success } +TRITONSERVER_Error* +HTTPAPIServer::EVBufferToJson( + triton::common::TritonJson::Value* document, evbuffer_iovec* v, int* v_idx, + const size_t length, int n) +{ + size_t offset = 0, remaining_length = length; + char* json_base; + std::vector json_buffer; + + // No need to memcpy when number of iovecs is 1 + if ((n > 0) && (v[0].iov_len >= remaining_length)) { + json_base = static_cast(v[0].iov_base); + if (v[0].iov_len > remaining_length) { + v[0].iov_base = static_cast(json_base + remaining_length); + v[0].iov_len -= remaining_length; + remaining_length = 0; + } else if (v[0].iov_len == remaining_length) { + remaining_length = 0; + *v_idx += 1; + } + } else { + json_buffer.resize(length); + json_base = json_buffer.data(); + while ((remaining_length > 0) && (*v_idx < n)) { + char* base = static_cast(v[*v_idx].iov_base); + size_t base_size; + if (v[*v_idx].iov_len > remaining_length) { + base_size = remaining_length; + v[*v_idx].iov_base = static_cast(base + remaining_length); + v[*v_idx].iov_len -= remaining_length; + remaining_length = 0; + } else { + base_size = v[*v_idx].iov_len; + remaining_length -= v[*v_idx].iov_len; + *v_idx += 1; + } + + memcpy(json_base + offset, base, base_size); + offset += base_size; + } + } + + if (remaining_length != 0) { + return TRITONSERVER_ErrorNew( + TRITONSERVER_ERROR_INVALID_ARG, + std::string( + "unexpected size for request JSON, expecting " + + std::to_string(remaining_length) + " more bytes") + .c_str()); + } + + RETURN_IF_ERR(document->Parse(json_base, length)); + + return nullptr; // success +} + struct HeaderSearchPayload { HeaderSearchPayload( const re2::RE2& regex, TRITONSERVER_InferenceRequest* request) @@ -3429,7 +3385,8 @@ HTTPAPIServer::HandleGenerate( // arbitrary JSON message convoluted (added key is reference to a string and // thus the string must live as long as the JSON message). triton::common::TritonJson::Value request; - RETURN_AND_CALLBACK_IF_ERR(EVRequestToJson(req, &request), error_callback); + RETURN_AND_CALLBACK_IF_ERR( + EVRequestToJsonAllowsEmpty(req, "generate", &request), error_callback); RETURN_AND_CALLBACK_IF_ERR( ParseJsonTritonRequestID(request, irequest), error_callback); diff --git a/src/http_server.h b/src/http_server.h index 877327b4d5..c5ed0eb6de 100644 --- a/src/http_server.h +++ b/src/http_server.h @@ -607,9 +607,34 @@ class HTTPAPIServer : public HTTPServer { std::map* input_metadata, triton::common::TritonJson::Value* meta_data_root); + // Internal utility method for parsing evhtp request to JSON + // Should not be called directly - use EVRequestToJson or + // EVRequestToJsonAllowsEmpty instead + TRITONSERVER_Error* EVRequestToJsonImpl( + evhtp_request_t* req, std::string_view request_kind, + bool allows_empty_body, triton::common::TritonJson::Value* request_json, + size_t* buffer_len); + // Parses full evhtp request and its evbuffers into JSON. + TRITONSERVER_Error* EVRequestToJsonAllowsEmpty( + evhtp_request_t* req, std::string_view request_kind, + triton::common::TritonJson::Value* request_json) + { + size_t buffer_len = 0; + TRITONSERVER_Error* err = + EVRequestToJsonImpl(req, request_kind, true, request_json, &buffer_len); + return err; + } + TRITONSERVER_Error* EVRequestToJson( - evhtp_request_t* req, triton::common::TritonJson::Value* request_json); + evhtp_request_t* req, std::string_view request_kind, + triton::common::TritonJson::Value* request_json, size_t* buffer_len) + { + TRITONSERVER_Error* err = + EVRequestToJsonImpl(req, request_kind, false, request_json, buffer_len); + return err; + } + // Parses evhtp request buffers into Triton Inference Request. TRITONSERVER_Error* EVRequestToTritonRequest( evhtp_request_t* req, const std::string& model_name, @@ -622,6 +647,9 @@ class HTTPAPIServer : public HTTPServer { TRITONSERVER_Error* EVBufferToRawInput( const std::string& model_name, TRITONSERVER_InferenceRequest* irequest, evbuffer* input_buffer, InferRequestClass* infer_req); + TRITONSERVER_Error* EVBufferToJson( + triton::common::TritonJson::Value* document, evbuffer_iovec* v, + int* v_idx, const size_t length, int n); // Helpers for parsing JSON requests for Triton-specific fields diff --git a/src/sagemaker_server.cc b/src/sagemaker_server.cc index 52074f2b9d..49b8a3ba06 100644 --- a/src/sagemaker_server.cc +++ b/src/sagemaker_server.cc @@ -54,63 +54,6 @@ EVBufferAddErrorJson(evbuffer* buffer, TRITONSERVER_Error* err) evbuffer_add(buffer, buffer_json.Base(), buffer_json.Size()); } - -TRITONSERVER_Error* -EVBufferToJson( - triton::common::TritonJson::Value* document, evbuffer_iovec* v, int* v_idx, - const size_t length, int n) -{ - size_t offset = 0, remaining_length = length; - char* json_base; - std::vector json_buffer; - - // No need to memcpy when number of iovecs is 1 - if ((n > 0) && (v[0].iov_len >= remaining_length)) { - json_base = static_cast(v[0].iov_base); - if (v[0].iov_len > remaining_length) { - v[0].iov_base = static_cast(json_base + remaining_length); - v[0].iov_len -= remaining_length; - remaining_length = 0; - } else if (v[0].iov_len == remaining_length) { - remaining_length = 0; - *v_idx += 1; - } - } else { - json_buffer.resize(length); - json_base = json_buffer.data(); - while ((remaining_length > 0) && (*v_idx < n)) { - char* base = static_cast(v[*v_idx].iov_base); - size_t base_size; - if (v[*v_idx].iov_len > remaining_length) { - base_size = remaining_length; - v[*v_idx].iov_base = static_cast(base + remaining_length); - v[*v_idx].iov_len -= remaining_length; - remaining_length = 0; - } else { - base_size = v[*v_idx].iov_len; - remaining_length -= v[*v_idx].iov_len; - *v_idx += 1; - } - - memcpy(json_base + offset, base, base_size); - offset += base_size; - } - } - - if (remaining_length != 0) { - return TRITONSERVER_ErrorNew( - TRITONSERVER_ERROR_INVALID_ARG, - std::string( - "unexpected size for request JSON, expecting " + - std::to_string(remaining_length) + " more bytes") - .c_str()); - } - - RETURN_IF_ERR(document->Parse(json_base, length)); - - return nullptr; // success -} - } // namespace @@ -266,7 +209,9 @@ SagemakerAPIServer::Handle(evhtp_request_t* req) std::unordered_map parse_load_map; ParseSageMakerRequest(req, &parse_load_map, "load"); - SageMakerMMELoadModel(req, parse_load_map); + if (!parse_load_map.empty()) { + SageMakerMMELoadModel(req, parse_load_map); + } return; } break; @@ -320,29 +265,15 @@ SagemakerAPIServer::ParseSageMakerRequest( std::unordered_map* parse_map, const std::string& action) { - struct evbuffer_iovec* v = nullptr; - int v_idx = 0; - int n = evbuffer_peek(req->buffer_in, -1, NULL, NULL, 0); - if (n > 0) { - v = static_cast( - alloca(sizeof(struct evbuffer_iovec) * n)); - if (evbuffer_peek(req->buffer_in, -1, NULL, v, n) != n) { - HTTP_RESPOND_IF_ERR( - req, TRITONSERVER_ErrorNew( - TRITONSERVER_ERROR_INTERNAL, - "unexpected error getting load model request buffers")); - } - } + size_t buffer_len; + triton::common::TritonJson::Value request; + HTTP_RESPOND_IF_ERR( + req, EVRequestToJson(req, "load model", &request, &buffer_len)); std::string model_name_string; std::string url_string; - size_t buffer_len = evbuffer_get_length(req->buffer_in); if (buffer_len > 0) { - triton::common::TritonJson::Value request; - HTTP_RESPOND_IF_ERR( - req, EVBufferToJson(&request, v, &v_idx, buffer_len, n)); - triton::common::TritonJson::Value url; triton::common::TritonJson::Value model_name; From 559f87bd1bfdcb02b3e6005ff7d6b32d236b0f53 Mon Sep 17 00:00:00 2001 From: Misha Chornyi <99709299+mc-nv@users.noreply.github.com> Date: Tue, 29 Jul 2025 13:06:30 -0700 Subject: [PATCH 13/14] Update README.md (#8318) --- README.md | 230 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 228 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 4e3a4c33e4..753afac84d 100644 --- a/README.md +++ b/README.md @@ -27,5 +27,231 @@ --> [![License](https://img.shields.io/badge/License-BSD3-lightgrey.svg)](https://opensource.org/licenses/BSD-3-Clause) ->[!WARNING] -> You are currently on the `r25.07` branch which tracks under-development progress towards the next release. +# Triton Inference Server + +Triton Inference Server is an open source inference serving software that +streamlines AI inferencing. Triton enables teams to deploy any AI model from +multiple deep learning and machine learning frameworks, including TensorRT, +TensorFlow, PyTorch, ONNX, OpenVINO, Python, RAPIDS FIL, and more. Triton +Inference Server supports inference across cloud, data center, edge and embedded +devices on NVIDIA GPUs, x86 and ARM CPU, or AWS Inferentia. Triton Inference +Server delivers optimized performance for many query types, including real time, +batched, ensembles and audio/video streaming. Triton inference Server is part of +[NVIDIA AI Enterprise](https://www.nvidia.com/en-us/data-center/products/ai-enterprise/), +a software platform that accelerates the data science pipeline and streamlines +the development and deployment of production AI. + +Major features include: + +- [Supports multiple deep learning + frameworks](https://github.com/triton-inference-server/backend#where-can-i-find-all-the-backends-that-are-available-for-triton) +- [Supports multiple machine learning + frameworks](https://github.com/triton-inference-server/fil_backend) +- [Concurrent model + execution](docs/user_guide/architecture.md#concurrent-model-execution) +- [Dynamic batching](docs/user_guide/model_configuration.md#dynamic-batcher) +- [Sequence batching](docs/user_guide/model_configuration.md#sequence-batcher) and + [implicit state management](docs/user_guide/architecture.md#implicit-state-management) + for stateful models +- Provides [Backend API](https://github.com/triton-inference-server/backend) that + allows adding custom backends and pre/post processing operations +- Supports writing custom backends in python, a.k.a. + [Python-based backends.](https://github.com/triton-inference-server/backend/blob/r25.07/docs/python_based_backends.md#python-based-backends) +- Model pipelines using + [Ensembling](docs/user_guide/architecture.md#ensemble-models) or [Business + Logic Scripting + (BLS)](https://github.com/triton-inference-server/python_backend#business-logic-scripting) +- [HTTP/REST and GRPC inference + protocols](docs/customization_guide/inference_protocols.md) based on the community + developed [KServe + protocol](https://github.com/kserve/kserve/tree/master/docs/predict-api/v2) +- A [C API](docs/customization_guide/inference_protocols.md#in-process-triton-server-api) and + [Java API](docs/customization_guide/inference_protocols.md#java-bindings-for-in-process-triton-server-api) + allow Triton to link directly into your application for edge and other in-process use cases +- [Metrics](docs/user_guide/metrics.md) indicating GPU utilization, server + throughput, server latency, and more + +**New to Triton Inference Server?** Make use of +[these tutorials](https://github.com/triton-inference-server/tutorials) +to begin your Triton journey! + +Join the [Triton and TensorRT community](https://www.nvidia.com/en-us/deep-learning-ai/triton-tensorrt-newsletter/) and +stay current on the latest product updates, bug fixes, content, best practices, +and more. Need enterprise support? NVIDIA global support is available for Triton +Inference Server with the +[NVIDIA AI Enterprise software suite](https://www.nvidia.com/en-us/data-center/products/ai-enterprise/). + +## Serve a Model in 3 Easy Steps + +```bash +# Step 1: Create the example model repository +git clone -b r25.07 https://github.com/triton-inference-server/server.git +cd server/docs/examples +./fetch_models.sh + +# Step 2: Launch triton from the NGC Triton container +docker run --gpus=1 --rm --net=host -v ${PWD}/model_repository:/models nvcr.io/nvidia/tritonserver:25.07-py3 tritonserver --model-repository=/models --model-control-mode explicit --load-model densenet_onnx + +# Step 3: Sending an Inference Request +# In a separate console, launch the image_client example from the NGC Triton SDK container +docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:25.07-py3-sdk /workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg + +# Inference should return the following +Image '/workspace/images/mug.jpg': + 15.346230 (504) = COFFEE MUG + 13.224326 (968) = CUP + 10.422965 (505) = COFFEEPOT +``` +Please read the [QuickStart](docs/getting_started/quickstart.md) guide for additional information +regarding this example. The quickstart guide also contains an example of how to launch Triton on [CPU-only systems](docs/getting_started/quickstart.md#run-on-cpu-only-system). New to Triton and wondering where to get started? Watch the [Getting Started video](https://youtu.be/NQDtfSi5QF4). + +## Examples and Tutorials + +Check out [NVIDIA LaunchPad](https://www.nvidia.com/en-us/data-center/products/ai-enterprise-suite/trial/) +for free access to a set of hands-on labs with Triton Inference Server hosted on +NVIDIA infrastructure. + +Specific end-to-end examples for popular models, such as ResNet, BERT, and DLRM +are located in the +[NVIDIA Deep Learning Examples](https://github.com/NVIDIA/DeepLearningExamples) +page on GitHub. The +[NVIDIA Developer Zone](https://developer.nvidia.com/nvidia-triton-inference-server) +contains additional documentation, presentations, and examples. + +## Documentation + +### Build and Deploy + +The recommended way to build and use Triton Inference Server is with Docker +images. + +- [Install Triton Inference Server with Docker containers](docs/customization_guide/build.md#building-with-docker) (*Recommended*) +- [Install Triton Inference Server without Docker containers](docs/customization_guide/build.md#building-without-docker) +- [Build a custom Triton Inference Server Docker container](docs/customization_guide/compose.md) +- [Build Triton Inference Server from source](docs/customization_guide/build.md#building-on-unsupported-platforms) +- [Build Triton Inference Server for Windows 10](docs/customization_guide/build.md#building-for-windows-10) +- Examples for deploying Triton Inference Server with Kubernetes and Helm on [GCP](deploy/gcp/README.md), + [AWS](deploy/aws/README.md), and [NVIDIA FleetCommand](deploy/fleetcommand/README.md) +- [Secure Deployment Considerations](docs/customization_guide/deploy.md) + +### Using Triton + +#### Preparing Models for Triton Inference Server + +The first step in using Triton to serve your models is to place one or +more models into a [model repository](docs/user_guide/model_repository.md). Depending on +the type of the model and on what Triton capabilities you want to enable for +the model, you may need to create a [model +configuration](docs/user_guide/model_configuration.md) for the model. + +- [Add custom operations to Triton if needed by your model](docs/user_guide/custom_operations.md) +- Enable model pipelining with [Model Ensemble](docs/user_guide/architecture.md#ensemble-models) + and [Business Logic Scripting (BLS)](https://github.com/triton-inference-server/python_backend#business-logic-scripting) +- Optimize your models setting [scheduling and batching](docs/user_guide/architecture.md#models-and-schedulers) + parameters and [model instances](docs/user_guide/model_configuration.md#instance-groups). +- Use the [Model Analyzer tool](https://github.com/triton-inference-server/model_analyzer) + to help optimize your model configuration with profiling +- Learn how to [explicitly manage what models are available by loading and + unloading models](docs/user_guide/model_management.md) + +#### Configure and Use Triton Inference Server + +- Read the [Quick Start Guide](docs/getting_started/quickstart.md) to run Triton Inference + Server on both GPU and CPU +- Triton supports multiple execution engines, called + [backends](https://github.com/triton-inference-server/backend#where-can-i-find-all-the-backends-that-are-available-for-triton), including + [TensorRT](https://github.com/triton-inference-server/tensorrt_backend), + [TensorFlow](https://github.com/triton-inference-server/tensorflow_backend), + [PyTorch](https://github.com/triton-inference-server/pytorch_backend), + [ONNX](https://github.com/triton-inference-server/onnxruntime_backend), + [OpenVINO](https://github.com/triton-inference-server/openvino_backend), + [Python](https://github.com/triton-inference-server/python_backend), and more +- Not all the above backends are supported on every platform supported by Triton. + Look at the + [Backend-Platform Support Matrix](https://github.com/triton-inference-server/backend/blob/r25.07/docs/backend_platform_support_matrix.md) + to learn which backends are supported on your target platform. +- Learn how to [optimize performance](docs/user_guide/optimization.md) using the + [Performance Analyzer](https://github.com/triton-inference-server/perf_analyzer/blob/r25.07/README.md) + and + [Model Analyzer](https://github.com/triton-inference-server/model_analyzer) +- Learn how to [manage loading and unloading models](docs/user_guide/model_management.md) in + Triton +- Send requests directly to Triton with the [HTTP/REST JSON-based + or gRPC protocols](docs/customization_guide/inference_protocols.md#httprest-and-grpc-protocols) + +#### Client Support and Examples + +A Triton *client* application sends inference and other requests to Triton. The +[Python and C++ client libraries](https://github.com/triton-inference-server/client) +provide APIs to simplify this communication. + +- Review client examples for [C++](https://github.com/triton-inference-server/client/blob/r25.07/src/c%2B%2B/examples), + [Python](https://github.com/triton-inference-server/client/blob/r25.07/src/python/examples), + and [Java](https://github.com/triton-inference-server/client/blob/r25.07/src/java/src/main/java/triton/client/examples) +- Configure [HTTP](https://github.com/triton-inference-server/client#http-options) + and [gRPC](https://github.com/triton-inference-server/client#grpc-options) + client options +- Send input data (e.g. a jpeg image) directly to Triton in the [body of an HTTP + request without any additional metadata](https://github.com/triton-inference-server/server/blob/r25.07/docs/protocol/extension_binary_data.md#raw-binary-request) + +### Extend Triton + +[Triton Inference Server's architecture](docs/user_guide/architecture.md) is specifically +designed for modularity and flexibility + +- [Customize Triton Inference Server container](docs/customization_guide/compose.md) for your use case +- [Create custom backends](https://github.com/triton-inference-server/backend) + in either [C/C++](https://github.com/triton-inference-server/backend/blob/r25.07/README.md#triton-backend-api) + or [Python](https://github.com/triton-inference-server/python_backend) +- Create [decoupled backends and models](docs/user_guide/decoupled_models.md) that can send + multiple responses for a request or not send any responses for a request +- Use a [Triton repository agent](docs/customization_guide/repository_agents.md) to add functionality + that operates when a model is loaded and unloaded, such as authentication, + decryption, or conversion +- Deploy Triton on [Jetson and JetPack](docs/user_guide/jetson.md) +- [Use Triton on AWS + Inferentia](https://github.com/triton-inference-server/python_backend/tree/r25.07/inferentia) + +### Additional Documentation + +- [FAQ](docs/user_guide/faq.md) +- [User Guide](docs/README.md#user-guide) +- [Customization Guide](docs/README.md#customization-guide) +- [Release Notes](https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/index.html) +- [GPU, Driver, and CUDA Support +Matrix](https://docs.nvidia.com/deeplearning/dgx/support-matrix/index.html) + +## Contributing + +Contributions to Triton Inference Server are more than welcome. To +contribute please review the [contribution +guidelines](CONTRIBUTING.md). If you have a backend, client, +example or similar contribution that is not modifying the core of +Triton, then you should file a PR in the [contrib +repo](https://github.com/triton-inference-server/contrib). + +## Reporting problems, asking questions + +We appreciate any feedback, questions or bug reporting regarding this project. +When posting [issues in GitHub](https://github.com/triton-inference-server/server/issues), +follow the process outlined in the [Stack Overflow document](https://stackoverflow.com/help/mcve). +Ensure posted examples are: +- minimal – use as little code as possible that still produces the + same problem +- complete – provide all parts needed to reproduce the problem. Check + if you can strip external dependencies and still show the problem. The + less time we spend on reproducing problems the more time we have to + fix it +- verifiable – test the code you're about to provide to make sure it + reproduces the problem. Remove all other problems that are not + related to your request/question. + +For issues, please use the provided bug report and feature request templates. + +For questions, we recommend posting in our community +[GitHub Discussions.](https://github.com/triton-inference-server/server/discussions) + +## For more information + +Please refer to the [NVIDIA Developer Triton page](https://developer.nvidia.com/nvidia-triton-inference-server) +for more information. From b258a4055a37b28138505e74b1edf42b54cd84cc Mon Sep 17 00:00:00 2001 From: Misha Chornyi Date: Tue, 29 Jul 2025 16:48:04 -0700 Subject: [PATCH 14/14] Restore default branch name --- README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 410d9100bb..48bd0baf1b 100644 --- a/README.md +++ b/README.md @@ -61,7 +61,7 @@ Major features include: - Provides [Backend API](https://github.com/triton-inference-server/backend) that allows adding custom backends and pre/post processing operations - Supports writing custom backends in python, a.k.a. - [Python-based backends.](https://github.com/triton-inference-server/backend/blob/r25.07/docs/python_based_backends.md#python-based-backends) + [Python-based backends.](https://github.com/triton-inference-server/backend/blob/main/docs/python_based_backends.md#python-based-backends) - Model pipelines using [Ensembling](docs/user_guide/architecture.md#ensemble-models) or [Business Logic Scripting @@ -173,10 +173,10 @@ configuration](docs/user_guide/model_configuration.md) for the model. [Python](https://github.com/triton-inference-server/python_backend), and more - Not all the above backends are supported on every platform supported by Triton. Look at the - [Backend-Platform Support Matrix](https://github.com/triton-inference-server/backend/blob/r25.07/docs/backend_platform_support_matrix.md) + [Backend-Platform Support Matrix](https://github.com/triton-inference-server/backend/blob/main/docs/backend_platform_support_matrix.md) to learn which backends are supported on your target platform. - Learn how to [optimize performance](docs/user_guide/optimization.md) using the - [Performance Analyzer](https://github.com/triton-inference-server/perf_analyzer/blob/r25.07/README.md) + [Performance Analyzer](https://github.com/triton-inference-server/perf_analyzer/blob/main/README.md) and [Model Analyzer](https://github.com/triton-inference-server/model_analyzer) - Learn how to [manage loading and unloading models](docs/user_guide/model_management.md) in @@ -190,14 +190,14 @@ A Triton *client* application sends inference and other requests to Triton. The [Python and C++ client libraries](https://github.com/triton-inference-server/client) provide APIs to simplify this communication. -- Review client examples for [C++](https://github.com/triton-inference-server/client/blob/r25.07/src/c%2B%2B/examples), - [Python](https://github.com/triton-inference-server/client/blob/r25.07/src/python/examples), - and [Java](https://github.com/triton-inference-server/client/blob/r25.07/src/java/src/main/java/triton/client/examples) +- Review client examples for [C++](https://github.com/triton-inference-server/client/blob/main/src/c%2B%2B/examples), + [Python](https://github.com/triton-inference-server/client/blob/main/src/python/examples), + and [Java](https://github.com/triton-inference-server/client/blob/main/src/java/src/main/java/triton/client/examples) - Configure [HTTP](https://github.com/triton-inference-server/client#http-options) and [gRPC](https://github.com/triton-inference-server/client#grpc-options) client options - Send input data (e.g. a jpeg image) directly to Triton in the [body of an HTTP - request without any additional metadata](https://github.com/triton-inference-server/server/blob/r25.07/docs/protocol/extension_binary_data.md#raw-binary-request) + request without any additional metadata](https://github.com/triton-inference-server/server/blob/main/docs/protocol/extension_binary_data.md#raw-binary-request) ### Extend Triton @@ -206,7 +206,7 @@ designed for modularity and flexibility - [Customize Triton Inference Server container](docs/customization_guide/compose.md) for your use case - [Create custom backends](https://github.com/triton-inference-server/backend) - in either [C/C++](https://github.com/triton-inference-server/backend/blob/r25.07/README.md#triton-backend-api) + in either [C/C++](https://github.com/triton-inference-server/backend/blob/main/README.md#triton-backend-api) or [Python](https://github.com/triton-inference-server/python_backend) - Create [decoupled backends and models](docs/user_guide/decoupled_models.md) that can send multiple responses for a request or not send any responses for a request @@ -215,7 +215,7 @@ designed for modularity and flexibility decryption, or conversion - Deploy Triton on [Jetson and JetPack](docs/user_guide/jetson.md) - [Use Triton on AWS - Inferentia](https://github.com/triton-inference-server/python_backend/tree/r25.07/inferentia) + Inferentia](https://github.com/triton-inference-server/python_backend/tree/main/inferentia) ### Additional Documentation