ci: Warm up CUDA cache before tests in L0_batcher for GB300 runners - #8702
Conversation
mudit-eng
left a comment
mudit-eng
left a comment
There was a problem hiding this comment.
Please put a description of your changes.
|
You might have added files from Matthew's PR for SageMaker endpoint |
Fixed. Merged to wrong branch |
yinggeh
changed the title
| _ragged_batch_supported_trials.append("libtorch") | ||
|
|
||
| _max_queue_delay_ms = 10000 | ||
| _max_expected_response_ms = 3000 |
There was a problem hiding this comment.
should we make this an environment variable with a default of 3000 while we're updating this?
_max_expected_response_ms = os.getenv("MAX_EXPECTED_RESPONSE_MS", 3000)
There was a problem hiding this comment.
I haven't seen cases where we want to make it a dynamic value.
There was a problem hiding this comment.
then, if the value is considered a constant, it should be named MAX_EXPECTED_RESPONSE_TIME per the Python style guidelines. No?
There was a problem hiding this comment.
Maybe. I am just following the existing pattern here. It will be a lot more changes if I normalize the naming convention in this file. I can do it in a separate PR though.
| kill_server | ||
| } | ||
|
|
||
| # [TRI-830] Send a simpple request to warmup CUDA_CACHE for GB300 before testing. |
4 participants
Warm up CUDA cache before tests in L0_batcher for GB300 runners