Skip to content

Commit 702c6da

Browse files
whoisjwhoisj
authored
docs: correct model management (#8562)
1 parent abbb415 commit 702c6da

1 file changed

Lines changed: 90 additions & 65 deletions

File tree

‎docs/user_guide/model_management.md‎

Lines changed: 90 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
<!--
2-
# Copyright 2018-2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# Copyright 2018-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
33
#
44
# Redistribution and use in source and binary forms, with or without
55
# modification, are permitted provided that the following conditions
@@ -34,65 +34,79 @@ The model control mode determines how changes to the model repository are handle
3434

3535
## Model Control Mode `NONE`
3636

37-
Triton attempts to load all models in the model repository at startup.
38-
Models that Triton is not able to load will be marked as "UNAVAILABLE" and will not be available for inferencing.
37+
- Triton attempts to load all models in the model repository at startup.
38+
Models that Triton is not able to load will be marked as "UNAVAILABLE" and will not be available for inferencing.
3939

40-
Changes to the model repository while the server is running will be ignored.
41-
Model load and unload requests using the [model control protocol](../protocol/extension_model_repository.md) will have no affect and will return an error response.
40+
- Changes to the model repository while the server is running will be ignored.
41+
Model load and unload requests using the [model control protocol](../protocol/extension_model_repository.md) will have no affect and will return an error response.
42+
43+
- This model control mode is selected by specifying `--model-control-mode=none` when starting Triton.
4244

43-
This model control mode is selected by specifying `--model-control-mode=none` when starting Triton.
44-
This is the default model control mode.
45-
Changing the model repository while Triton is running must be done carefully, as explained in [Modifying the Model Repository](#modifying-the-model-repository).
45+
- This is the default model control mode.
46+
47+
> [!IMPORTANT]
48+
> Changing the model repository while Triton is running must be done carefully, as explained in [Modifying the Model Repository](#modifying-the-model-repository).
4649
4750
## Model Control Mode `EXPLICIT`
4851

49-
At startup, Triton loads only those models specified explicitly with the `--load-model` command-line option.
50-
To load ALL models at startup, specify `--load-model=*` as the ONLY `--load-model` argument.
51-
Specifying `--load-model=*` in conjunction with another `--load-model` argument will result in error.
52-
If `--load-model` is not specified then no models are loaded at startup.
53-
Models that Triton is not able to load will be marked as "UNAVAILABLE" and will not be available for inferencing.
52+
- At startup, Triton loads only those models specified explicitly with the `--load-model` command-line option.
5453

55-
After startup, all model load and unload actions must be initiated explicitly by using the [model control protocol](../protocol/extension_model_repository.md).
56-
The response status of the model control request indicates success or failure of the load or unload action.
57-
When attempting to reload an already loaded model, if the reload fails for any reason the already loaded model will be unchanged and will remain loaded.
58-
If the reload succeeds, the newly loaded model will replace the already loaded model without any loss in availability for the model.
54+
- To load ALL models at startup, specify `--load-model=*` as the ONLY `--load-model` argument.
55+
Specifying `--load-model=*` in conjunction with another `--load-model` argument will result in error.
5956

60-
This model control mode is enabled by specifying `--model-control-mode=explicit`.
61-
Changing the model repository while Triton is running must be done carefully, as explained in
62-
[Modifying the Model Repository](#modifying-the-model-repository).
57+
- If `--load-model` is not specified then no models are loaded at startup.
58+
Models that Triton is not able to load will be marked as "UNAVAILABLE" and will not be available for inferencing.
59+
60+
- After startup, all model load and unload actions must be initiated explicitly by using the [model control protocol](../protocol/extension_model_repository.md).
61+
The response status of the model control request indicates success or failure of the load or unload action.
62+
When attempting to reload an already loaded model, the existing model should be explicitly unloaded prior to the updated version being loaded.
63+
64+
- This model control mode is enabled by specifying `--model-control-mode=explicit`.
65+
66+
### Using Alternate Memory Allocation Libraries
67+
68+
If you are seeing some memory growth when using the [model control protocol](../protocol/extension_model_repository.md) for loading and unloading models, it is possible that it's not an actual memory leak but some system's `malloc` heuristics that causes memory to be unable to be released back to the OS right away.
69+
70+
To improve memory performance, you can consider switching from `malloc` to [`tcmalloc`](https://github.com/google/tcmalloc) or to [`jemalloc`](https://github.com/jemalloc/jemalloc) by setting the `LD_PRELOAD` environment variable when running Triton, as shown below:
71+
72+
- Using `tcmalloc`:
73+
74+
```bash
75+
LD_PRELOAD=/usr/lib/$(uname -m)-linux-gnu/libtcmalloc.so.4:${LD_PRELOAD} tritonserver --model-repository=/models ...
76+
```
77+
78+
- Using `jemalloc`:
79+
80+
```bash
81+
PRELOAD=/usr/lib/$(uname -m)-linux-gnu/libtcmalloc.so:${LD_PRELOAD} tritonserver --model-repository=/models ...
82+
```
6383

64-
If you are seeing some memory growth when using the [model control protocol](../protocol/extension_model_repository.md) for loading and unloading models, it is possible that it's not an actual memory leak but some system's malloc heuristics that causes memory to be unable to be released back to the OS right away.
65-
To improve memory performance, you can consider switching from malloc to [tcmalloc](https://github.com/google/tcmalloc) or to [jemalloc](https://github.com/jemalloc/jemalloc) by setting the `LD_PRELOAD` environment variable when running Triton, as shown below:
66-
```
67-
# Using tcmalloc
68-
LD_PRELOAD=/usr/lib/$(uname -m)-linux-gnu/libtcmalloc.so.4:${LD_PRELOAD} tritonserver --model-repository=/models ...
69-
```
70-
```
71-
# Using jemalloc
72-
LD_PRELOAD=/usr/lib/$(uname -m)-linux-gnu/libjemalloc.so:${LD_PRELOAD} tritonserver --model-repository=/models ...
73-
```
7484
We recommend experimenting with both `tcmalloc` and `jemalloc` to determine which one works better for your use case,
7585
as they have different strategies for memory allocation and deallocation and may perform differently depending on the workload.
7686

7787
Both `tcmalloc` and `jemalloc` libraries are already installed within the Triton container.
7888
However, if you need to install them, you can do so using the following commands:
79-
```
80-
# Install tcmalloc
81-
apt-get install gperf libgoogle-perftools-dev
82-
```
83-
```
84-
# Install jemalloc
85-
apt-get install libjemalloc-dev
86-
```
89+
90+
- Install `tcmalloc`:
91+
92+
```bash
93+
apt-get install gperf libgoogle-perftools-dev
94+
```
95+
96+
- Install `jemalloc`:
97+
98+
```bash
99+
apt-get install libjemalloc-dev
100+
```
87101

88102
## Model Control Mode `POLL`
89103

90-
Triton attempts to load all models in the model repository at startup.
91-
Models that Triton is not able to load will be marked as "UNAVAILABLE" and will not be available for inferencing.
104+
- Triton attempts to load all models in the model repository at startup.
105+
Models that Triton is not able to load will be marked as "UNAVAILABLE" and will not be available for inferencing.
92106

93-
Changes to the model repository will be detected and Triton will attempt to load and unload models as necessary based on those changes.
94-
When attempting to reload an already loaded model, if the reload fails for any reason the already loaded model will be unchanged and will remain loaded.
95-
If the reload succeeds, the newly loaded model will replace the already loaded model without any loss of availability for the model.
107+
- Changes to the model repository will be detected and Triton will attempt to load and unload models as necessary based on those changes.
108+
- When reloading a model fails, the already loaded model will be unchanged and will remain loaded.
109+
- When reloading a model succeeds, the existing loaded model will be replaced by a newly loaded instance without loss of availability.
96110

97111
Changes to the model repository may not be detected immediately because Triton polls the repository periodically.
98112
You can control the polling interval with the `--repository-poll-secs` option.
@@ -112,20 +126,20 @@ Changing the model repository while Triton is running must be done carefully, as
112126

113127
In `POLL` mode Triton responds to the following model repository changes:
114128

115-
* Versions may be added and removed from models by adding and removing the corresponding version subdirectory.
129+
- Versions may be added and removed from models by adding and removing the corresponding version subdirectory.
116130
Triton will allow in-flight requests to complete even if they are using a removed version of the model.
117131
New requests for a removed model version will fail.
118-
Depending on the model's [version policy](model_configuration.md#version-policy), changes to the available versions may change which model version is served by default.
132+
Depending on the model's [version policy](model_configuration.md#version-policy), changes to the available versions may change which model version is served by default.
119133

120-
* Existing models can be removed from the repository by removing the corresponding model directory.
134+
- Existing models can be removed from the repository by removing the corresponding model directory.
121135
Triton will allow in-flight requests to any version of the removed model to complete.
122136
New requests for a removed model will fail.
123137

124-
* New models can be added to the repository by adding a new model directory.
138+
- New models can be added to the repository by adding a new model directory.
125139

126-
* The [model configuration file](model_configuration.md) (config.pbtxt) can be changed and Triton will unload and reload the model to pick up the new model configuration.
140+
- The [model configuration file](model_configuration.md) (config.pbtxt) can be changed and Triton will unload and reload the model to pick up the new model configuration.
127141

128-
* Label(s) files providing labels for outputs that represent classifications can be added, removed, or modified and Triton will unload and reload the model to pick up the new labels.
142+
- Label(s) files providing labels for outputs that represent classifications can be added, removed, or modified and Triton will unload and reload the model to pick up the new labels.
129143
If a label file is added or removed, the corresponding edit to the `label_filename` property of the output it corresponds to in the [model configuration](model_configuration.md) must be performed at the same time.
130144

131145
## Modifying the Model Repository
@@ -134,35 +148,46 @@ Each model in a model repository [resides in its own sub-directory](model_reposi
134148
The activity allowed on the contents of a model's sub-directory varies depending on how Triton is using that model.
135149
The state of a model can be determined by using the [model metadata](../customization_guide/inference_protocols.md#inference-protocols-and-apis) or [repository index](../protocol/extension_model_repository.md#index) APIs.
136150

137-
* If the model is actively loading or unloading, no files or directories within that sub-directory must be added, removed or modified.
151+
- When a model is actively loading or unloading, no files or directories within that sub-directory must be added, removed, or modified.
152+
153+
- When a model has never been loaded or has been completely unloaded, then its entire model sub-directory can be removed or its contents modified.
138154

139-
* If the model has never been loaded or has been completely unloaded, then the entire model sub-directory can be removed or any of its contents can be added, removed or modified.
155+
- When the model has been completely loaded, then any files or directories within its sub-directory can be added, removed, or modified; except for shared libraries implementing the model's backend.
140156

141-
* If the model has been completely loaded then any files or directories within that sub-directory can be added, removed or modified; except for shared libraries implementing the model's backend.
142-
Triton uses the backend shared libraries while the model is loading so removing or modifying them will likely cause Triton to crash.
143-
To update a model's backend you must first unload the model completely, modify the backend shared libraries, and then reload the model.
144-
With some operating systems, it may also be possible to simply move the existing shared-libraries to another location outside of the model repository, copy in the new shared libraries, and then reload the model.
157+
- Triton uses the backend shared libraries while the model is loading, removing or modifying them is recommended against as it can destabilize the Triton process.
145158

146-
* If only the model instance configuration in the [model configuration file](model_configuration.md) (config.pbtxt) is modified (i.e. increasing/decreasing the instance count), will Triton update the model rather then reloading it, when either a load request is received under [Model Control Mode EXPLICIT](#model-control-mode-explicit) or change to the [model configuration file](model_configuration.md) (config.pbtxt) is detected under [Model Control Mode POLL](#model-control-mode-poll).
159+
- To update a model's backend, the model cannot be loaded by Triton:
147160

148-
* The new model configuration may also be passed to Triton via the
149-
[load API](../protocol/extension_model_repository.md#load).
161+
1. Unload any loaded models relying on the backend-to-be-updated.
162+
2. Modify the backend's shared libraries.
163+
3. Load the previously unloaded models.
150164

151-
* Some text editors create a swap file in the model directory when the [model configuration file](model_configuration.md) (config.pbtxt) is modified in place.
165+
> [!TIP]
166+
> With some operating systems, it may also be possible to simply move the existing shared-libraries to another location outside of the model repository, copy in the new shared libraries, and then reload the model.
167+
168+
- When only the model instance configuration in the [model configuration file](model_configuration.md) (config.pbtxt) is modified (i.e. increasing/decreasing the instance count), will Triton update the model rather then reloading it.
169+
170+
- When either a load request is received under [Model Control Mode EXPLICIT](#model-control-mode-explicit) or change to the [model configuration file](model_configuration.md) (config.pbtxt) is detected under [Model Control Mode POLL](#model-control-mode-poll).
171+
172+
- The new model configuration may also be passed to Triton via the [load API](../protocol/extension_model_repository.md#load).
173+
174+
- Some text editors create a swap file in the model directory when the [model configuration file](model_configuration.md) (config.pbtxt) is modified in place.
152175
The swap file is not part of the model configuration, so its presence in the model directory may be detected as a new file and cause the model to fully reload when only an update is expected.
153176

154-
* If a sequence model is *updated* (i.e. decreasing the instance count), Triton will wait until the in-flight sequence is completed (or timed-out) before the instance behind the sequence is removed.
177+
- When a sequence model is *updated* (i.e. decreasing the instance count), Triton will wait until in-flight sequences are completed, or otherwise cleared, before the instance behind the sequence is removed.
178+
179+
- When the instance count is decreased, arbitrary instance(s) are selected among idle instances and instances with in-flight sequence(s) for removal.
155180

156-
* If the instance count is decreased, arbitrary instance(s) are selected among idle instances and instances with in-flight sequence(s) for removal.
181+
- When a sequence model is *reloaded* with in-flight sequence(s) (i.e. changes to the model file), Triton does not guarantee any remaining request(s) from the in-flight sequence(s) will be routed to the same model instance for processing.
157182

158-
* If a sequence model is *reloaded* with in-flight sequence(s) (i.e. changes to the model file), Triton does not guarantee any remaining request(s) from the in-flight sequence(s) will be routed to the same model instance for processing.
159-
It is currently the responsibility of the user to ensure any in-flight sequence(s) are completed before reloading a sequence model.
183+
> [!IMPORTANT]
184+
> It is currently the responsibility of the user to ensure any in-flight sequence(s) are completed before reloading a sequence model.
160185
161186
## Concurrently Loading Models
162187

163188
To reduce service downtime, Triton loads new models in the background while continuing to serve inferences on existing models.
164189
Based on use case and performance requirements, the optimal amount of resources dedicated to loading models may differ.
190+
165191
Triton exposes a `--model-load-thread-count` option to configure the number of threads dedicated to loading models, which defaults to `4`.
166192

167-
To set this parameter with the C API, refer to `TRITONSERVER_ServerOptionsSetModelLoadThreadCount` in
168-
[tritonserver.h](https://github.com/triton-inference-server/core/blob/main/include/triton/core/tritonserver.h).
193+
To set this parameter with the C API, refer to `TRITONSERVER_ServerOptionsSetModelLoadThreadCount` in [tritonserver.h](https://github.com/triton-inference-server/core/blob/main/include/triton/core/tritonserver.h).

0 commit comments

Comments
 (0)