Merge RAPIDS-Triton into developer tools

.github/CODEOWNERS

@@ -0,0 +1,2 @@
+* @tanmayv25 @GuanLuo
+/backend @wphicks @dantegd @divyegala

.gitignore

@@ -1,3 +1,2 @@
-/build
-/.vscode
-*.so
+backend/cpp/build
+backend/qa/logs

README.md

@@ -1 +1,46 @@
-# triton_developer_tools
+# Triton Developer Tools
+This repository contains tools to make it easier to develop custom C++ backends
+for NVIDIA's Triton Inference Server and to use Triton as a library within an
+existing C++ project. It consists of two sub-projects:
+- [Triton Backend Developer Tools](https://github.com/triton-inference-server/developer_tools/tree/main/backend): A header-only library to make it quick and easy to add high-performance custom functionality to Triton at the C++ level
+- Triton Library Developer Tools: Coming soon, this library will allow easy
+  integration of Triton as a library within other C++ projects
+
+## Backend Developer Tools
+
+### Why might I want to use Triton's Backend Developer Tools?
+- You have an inference model which you wish to serve in Triton, but Triton
+  does not currently support your model type
+- You want to add specialized data manipulation logic to Triton and require
+  performance beyond what the Python backend can provide
+- You want to ensure that your custom C++ backend stays up-to-date with the
+  latest Triton performance features
+
+### Why might I _not_ want to use Triton's Backend Developer Tools?
+- The functionality you're looking for is already provided by an existing
+  Triton backend
+- You want to write your additional logic in Python rather than C++
+
+### Example
+To create a custom backend with Triton's Backend Developer Tools, you can
+make use of the Backend Template (TODO(wphicks): link) and have a working
+Triton backend in minutes. For example, consider a simple example of a backend
+that simply returns any array of floats provided as input. In order to create
+such a backend, the only code we would have to write would be the
+following `predict` method:
+```
+  void predict(dev_tools::Batch& batch) const {
+    dev_tools::Tensor<float> input = get_input<float>(batch, "input__0");
+    dev_tools::Tensor<float> output = get_output<float>(batch, "output__0");
+
+    dev_tools::copy(output, input);
+
+    output.finalize();
+  }
+```
+For a complete walkthrough of this example and much more detail on how to add
+your own functionality to Triton, check out the Backend Developer Tools docs
+TODO(wphicks): Link.
+
+## Library Developer Tools
+TODO(wphicks)

backend/.dockerignore

@@ -0,0 +1 @@
+cpp/build

backend/CONTRIBUTING.md

@@ -0,0 +1,93 @@
+<!--
+Copyright (c) 2021, NVIDIA CORPORATION.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<!-- TODO(wphicks): Add more detail -->
+# Contributing to RAPIDS-Triton
+
+You can help improve RAPIDS-Triton in any of the following ways:
+- Submitting a bug report, feature request or documentation issue
+- Proposing and implementing a new feature
+- Implementing a feature or bug-fix for an outstanding issue
+
+## Bug reports
+When submitting a bug report, please include a *minimum* *reproducible*
+example. Ideally, this should be a snippet of code that other developers can
+copy, paste, and immediately run to try to reproduce the error. Please:
+- Do include import statements and any other code necessary to immediately run
+  your example
+- Avoid examples that require other developers to download models or data
+  unless you cannot reproduce the problem with synthetically-generated data
+
+## Code Contributions
+To contribute code to this project, please follow these steps:
+1. Find an issue to work on or submit an issue documenting the problem you
+   would like to work on.
+2. Comment on the issue saying that you plan to work on it.
+3. Review the conventions below for information to help you make your changes
+   in a way that is consistent with the rest of the codebase.
+4. Code!
+5. Create your pull request.
+6. Wait for other developers to review your code and update your PR as needed.
+7. Once a PR is approved, it will be merged into the main branch.
+
+### Coding Conventions
+* RAPIDS-Triton follows [Almost Always Auto
+  (AAA)](https://herbsutter.com/2013/08/12/gotw-94-solution-aaa-style-almost-always-auto/)
+  style. Please maintain this style in any contributions, with the possible
+  exception of some docs, where type information may be helpful for new users
+  trying to understand a snippet in isolation.
+* Avoid raw loops where possible.
+* C++ versions of types should be used instead of C versions except when
+  interfacing with C code (e.g. use `std::size_t` instead of `size_t`).
+* Avoid using output pointers in function signatures. Prefer instead to
+  actually return the value computed by the function and take advantage of
+  return value optimization and move semantics.
+
+### Signing Your Work
+* We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
+  * Any contribution which contains commits that are not Signed-Off will not be accepted.
+* To sign off on a commit you simply use the `--signoff` (or `-s`) option when committing your changes:
+  ```bash
+  $ git commit -s -m "Add cool feature."
+  ```
+  This will append the following to your commit message:
+  ```
+  Signed-off-by: Your Name <your@email.com>
+  ```
+* Full text of the DCO:
+  ```
+    Developer Certificate of Origin
+    Version 1.1
+
+    Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
+    1 Letterman Drive
+    Suite D4700
+    San Francisco, CA, 94129
+
+    Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
+
+
+    Developer's Certificate of Origin 1.1
+
+    By making a contribution to this project, I certify that:
+
+    (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or
+
+    (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or
+
+    (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.
+
+    (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved.
+  ```

backend/Dockerfile

@@ -0,0 +1,128 @@
+# Copyright (c) 2021-2022, NVIDIA CORPORATION.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+###########################################################################################
+# Arguments for controlling build details
+###########################################################################################
+# Version of Triton to use
+ARG TRITON_VERSION=22.12
+# Base container image
+ARG BASE_IMAGE=nvcr.io/nvidia/tritonserver:${TRITON_VERSION}-py3
+# Whether or not to build indicated components
+ARG BUILD_TESTS=OFF
+ARG BUILD_EXAMPLE=ON
+# Whether or not to enable GPU build
+ARG TRITON_ENABLE_GPU=ON
+
+FROM ${BASE_IMAGE} as base
+
+ENV PATH="/root/miniconda3/bin:${PATH}"
+
+RUN apt-get update \
+    && apt-get install --no-install-recommends -y wget patchelf \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+
+ENV PYTHONDONTWRITEBYTECODE=true
+
+RUN wget \
+    https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
+    && mkdir /root/.conda \
+    && bash Miniconda3-latest-Linux-x86_64.sh -b \
+    && rm -f Miniconda3-latest-Linux-x86_64.sh 
+
+COPY ./conda/environments/triton_backend.yml /environment.yml
+
+RUN conda env update -f /environment.yml \
+    && rm /environment.yml \
+    && conda clean -afy \
+    && find /root/miniconda3/ -follow -type f -name '*.pyc' -delete \
+    && find /root/miniconda3/ -follow -type f -name '*.js.map' -delete
+
+ENV PYTHONDONTWRITEBYTECODE=false
+
+SHELL ["conda", "run", "--no-capture-output", "-n", "triton_backend_dev", "/bin/bash", "-c"]
+
+FROM base as build-stage
+
+COPY ./cpp /triton_backend
+
+ARG TRITON_VERSION
+ENV TRITON_VERSION=$TRITON_VERSION
+
+ARG BUILD_TYPE=Release
+ENV BUILD_TYPE=$BUILD_TYPE
+ARG BUILD_TESTS
+ENV BUILD_TESTS=$BUILD_TESTS
+ARG BUILD_EXAMPLE
+ENV BUILD_EXAMPLE=$BUILD_EXAMPLE
+ARG TRITON_ENABLE_GPU
+ENV TRITON_ENABLE_GPU=$TRITON_ENABLE_GPU
+
+RUN mkdir /triton_backend/build
+
+WORKDIR /triton_backend/build
+
+RUN cmake \
+      -GNinja \
+      -DCMAKE_BUILD_TYPE="${BUILD_TYPE}" \
+      -DBUILD_TESTS="${BUILD_TESTS}" \
+      -DBUILD_EXAMPLE="${BUILD_EXAMPLE}" \
+      -DTRITON_ENABLE_GPU="${TRITON_ENABLE_GPU}" \
+      ..
+
+ENV CCACHE_DIR=/ccache
+
+RUN --mount=type=cache,target=/ccache/ ninja install
+
+FROM base as test-install
+
+COPY ./conda/environments/triton_backend_test.yml /environment.yml
+
+RUN conda env update -f /environment.yml \
+    && rm /environment.yml \
+    && conda clean -afy \
+    && find /root/miniconda3/ -follow -type f -name '*.pyc' -delete \
+    && find /root/miniconda3/ -follow -type f -name '*.js.map' -delete
+
+COPY ./python /triton_backend
+
+RUN conda run -n triton_backend_test pip install /triton_backend \
+ && rm -rf /triton_backend
+
+FROM build-stage as test-stage
+
+COPY --from=test-install /root/miniconda3 /root/miniconda3
+
+ENV TEST_EXE=/triton_backend/build/test_triton_backend
+
+COPY qa /qa
+
+ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "triton_backend_test", "/bin/bash", "/qa/entrypoint.sh"]
+
+FROM ${BASE_IMAGE}
+
+RUN mkdir /models
+
+# Remove existing backend install
+RUN if [ -d /opt/tritonserver/backends/dev_tools-identity ]; \
+    then \
+      rm -rf /opt/tritonserver/backends/dev_tools-identity/*; \
+    fi
+
+COPY --from=build-stage \
+  /opt/tritonserver/backends/dev_tools-identity \
+  /opt/tritonserver/backends/dev_tools-identity
+
+ENTRYPOINT ["tritonserver", "--model-repository=/models"]