Skip to content

Commit 437e7e2

Browse files
slorello89slorello89
authored
Merge pull request #7 from triton-inference-server/feature/tls
Feature/tls
2 parents 111f831 + ff9df76 commit 437e7e2

18 files changed

Lines changed: 1271 additions & 7 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
name: "CI"
2+
3+
on:
4+
push:
5+
branches: ['main']
6+
pull_request:
7+
branches: ['main']
8+
jobs:
9+
tls:
10+
runs-on: ubuntu-latest
11+
steps:
12+
- uses: actions/checkout@v2
13+
- name: install dependencies
14+
run: sudo apt update && sudo apt install -y rapidjson-dev
15+
- name: build
16+
run: bash ./build.sh
17+
- name: prepare models
18+
run: cd examples && source fetch_model.sh && cd ..
19+
- name: build test certs
20+
run: |
21+
source ./examples/tls/gen-certs.sh
22+
chmod +r ./certs/ca.crt ./certs/redis.crt ./certs/redis.key
23+
- name: mv certs
24+
run: mv certs examples/tls/
25+
- name: docker-compose up
26+
run: docker-compose -f ./examples/tls/docker-compose.yml up -d
27+
- name: execute
28+
run: docker-compose -f ./examples/tls/docker-compose.yml run client

‎.gitignore‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,7 @@
33
*.so
44
.idea
55
cmake-build-debug
6-
third-party
6+
third-party
7+
examples/tls/certs
8+
/examples/model_repository/densenet_onnx/1
9+
/examples/model_repository/inception_graphdef/1

‎CMakeLists.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@ execute_process(
7474
## Add redis++ library to the project
7575
find_library(REDISPP redis++ PATHS build/install/lib NO_DEFAULT_PATH REQUIRED)
7676
find_library(HIREDIS hiredis PATHS build/install/lib NO_DEFAULT_PATH REQUIRED)
77+
find_library(HIREDIS_SSL hiredis_ssl PATHS build/install/lib NO_DEFAULT_PATH REQUIRED)
7778

7879
#
7980
# Shared library implementing the Triton Cache API
@@ -111,6 +112,7 @@ target_link_libraries(
111112
triton-redis-cache
112113
PUBLIC
113114
${HIREDIS}
115+
${HIREDIS_SSL}
114116
${REDISPP}
115117
PRIVATE
116118
triton-core-serverapi # from repo-core

‎Dockerfile‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
FROM nvcr.io/nvidia/tritonserver:23.05-py3
1+
FROM nvcr.io/nvidia/tritonserver:23.06-py3
22

3-
RUN mkdir /opt/tritonserver/caches/redis
43
COPY ./build/install/caches/redis/libtritoncache_redis.so /opt/tritonserver/caches/redis

‎README.md‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,23 @@ tritonserver --cache-config redis,host=redis-host --cache-config redis,port=6379
108108

109109
Optionally you may configure your `user`/`password` via environment variables. The corresponding `user` environment variable is `TRITONCACHE_REDIS_USERNAME` whereas the corresponding `password` environment variable is `TRITONCACHE_REDIS_PASSWORD`.
110110

111+
### TLS
112+
113+
Transport Layer Security (TLS) can be enabled in Redis and within the Triton Redis Cache, to do so you will need a TLS
114+
enabled version of Redis, e.g. [OSS Redis](https://redis.io/docs/management/security/encryption/) or
115+
[Redis Enterprise](https://docs.redis.com/latest/rs/security/tls/enable-tls/). You will also need to configure Triton Server to use TLS with Redis
116+
through the following `--cache-config` TLS options.
117+
118+
#### Configuration Items for TLS
119+
120+
| Configuration Option | Required | Description |
121+
|----------------------|----------|-------------------------------------------------------|
122+
| tls_enabled | Yes | set to `true` to enable TLS |
123+
| cert | no | The certificate to use for TLS. |
124+
| key | no | The certificate key to use for TLS. |
125+
| cacert | No | The Certificate Authority certificate to use for TLS. |
126+
| sni | No | Server name indication for TLS. |
127+
111128
## Monitoring and Observability
112129

113130
There are many ways to go about monitoring what's going on in Redis. One popular mode is to export metrics data from Redis to Prometheus, and use Grafana to observe them.
@@ -120,6 +137,7 @@ There are many ways to go about monitoring what's going on in Redis. One popular
120137
You can try out the Redis Cache with Triton in docker:
121138

122139
* clone this repo: `git clone https://github.com/triton-inference-server/redis_cache`
140+
* follow build instructions enumerated [above](https://github.com/triton-inference-server/redis_cache#build-the-cache)
123141
* clone the Triton server repo: `git clone https://github.com/triton-inference-server`
124142
* Add the following to: `docs/examples/model_repository/densenet_onnx/config.pbtxt`
125143
```
@@ -140,7 +158,7 @@ Password: <MY API KEY>
140158
> NOTE: Username: $oauthtoken in this context means that your username is literally $oauthtoken - your API key serves as the unique part of your credentials
141159
* run `docker-compose build`
142160
* run `docker-compose up`
143-
* In a separate terminal run `docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:23.03-py3-sdk`
161+
* In a separate terminal run `docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:23.06-py3-sdk`
144162
* Run `/workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg`
145163
* on the first run - this will miss the cache
146164
* subsequent runs will pull the inference out of the cache

‎build.sh‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#!/bin/bash
2+
mkdir -p build
3+
cd build
4+
cmake -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install ..
5+
make install

‎build_deps.sh‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,8 @@ else
2222
fi
2323
cd hiredis
2424

25-
LIBRARY_PATH=lib CC=gcc CXX=g++ make PREFIX="$(pwd)/../../build/install" static -j 4
26-
LIBRARY_PATH=lib CC=gcc CXX=g++ make PREFIX="$(pwd)/../../build/install" install
25+
LIBRARY_PATH=lib CC=gcc CXX=g++ make PREFIX="$(pwd)/../../build/install" USE_SSL=1 static -j 4
26+
LIBRARY_PATH=lib CC=gcc CXX=g++ make PREFIX="$(pwd)/../../build/install" USE_SSL=1 install
2727
cd ../
2828
# delete shared libraries
2929
rm ../build/install/lib/*.so
@@ -44,7 +44,7 @@ else
4444
mkdir compile
4545
cd compile
4646

47-
$CMAKE -DCMAKE_BUILD_TYPE=Release -DREDIS_PLUS_PLUS_BUILD_TEST=OFF -DREDIS_PLUS_PLUS_BUILD_SHARED=OFF -DCMAKE_PREFIX_PATH="$(pwd)../../../build/install/lib/" -DCMAKE_INSTALL_PREFIX="$(pwd)/../../../build/install" -DCMAKE_CXX_STANDARD=17 ..
47+
$CMAKE -DCMAKE_BUILD_TYPE=Release -DREDIS_PLUS_PLUS_BUILD_TEST=OFF -DREDIS_PLUS_PLUS_BUILD_SHARED=OFF -DCMAKE_PREFIX_PATH="$(pwd)../../../build/install/lib/" -DCMAKE_INSTALL_PREFIX="$(pwd)/../../../build/install" -DCMAKE_CXX_STANDARD=17 -DREDIS_PLUS_PLUS_USE_TLS=ON ..
4848
CC=gcc CXX=g++ make -j 4
4949
CC=gcc CXX=g++ make install
5050
cd ../../

‎examples/fetch_model.sh‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
#!/bin/bash
2+
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
3+
#
4+
# Redistribution and use in source and binary forms, with or without
5+
# modification, are permitted provided that the following conditions
6+
# are met:
7+
# * Redistributions of source code must retain the above copyright
8+
# notice, this list of conditions and the following disclaimer.
9+
# * Redistributions in binary form must reproduce the above copyright
10+
# notice, this list of conditions and the following disclaimer in the
11+
# documentation and/or other materials provided with the distribution.
12+
# * Neither the name of NVIDIA CORPORATION nor the names of its
13+
# contributors may be used to endorse or promote products derived
14+
# from this software without specific prior written permission.
15+
#
16+
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
17+
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
18+
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
19+
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
20+
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
21+
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
22+
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
23+
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
24+
# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
25+
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
26+
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
27+
28+
set -ex
29+
30+
# ONNX densenet
31+
mkdir -p model_repository/densenet_onnx/1
32+
wget -O model_repository/densenet_onnx/1/model.onnx \
33+
https://contentmamluswest001.blob.core.windows.net/content/14b2744cf8d6418c87ffddc3f3127242/9502630827244d60a1214f250e3bbca7/08aed7327d694b8dbaee2c97b8d0fcba/densenet121-1.2.onnx

‎examples/model_repository/densenet_onnx/config.pbtxt‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: "densenet_onnx"
2+
platform: "onnxruntime_onnx"
3+
max_batch_size : 0
4+
response_cache {
5+
enable: true
6+
}
7+
input [
8+
{
9+
name: "data_0"
10+
data_type: TYPE_FP32
11+
format: FORMAT_NCHW
12+
dims: [ 3, 224, 224 ]
13+
reshape { shape: [ 1, 3, 224, 224 ] }
14+
}
15+
]
16+
output [
17+
{
18+
name: "fc6_1"
19+
data_type: TYPE_FP32
20+
dims: [ 1000 ]
21+
reshape { shape: [ 1, 1000, 1, 1 ] }
22+
label_filename: "densenet_labels.txt"
23+
}
24+
]

0 commit comments

Comments
 (0)