Optimize RF-DETR object detection pre/post-processing#2667
Optimize RF-DETR object detection pre/post-processing#2667dkosowski87 wants to merge 27 commits into
Conversation
…preprocessor implementation. This update includes a new `resolve_rfdetr_preprocessor` function for selecting preprocessing strategies, and integrates preprocessor configuration into the `pre_process_network_input` function. Additionally, the `RFDetrForObjectDetectionTRT` class now supports customizable preprocessor settings, improving flexibility for users. Updated error handling for unsupported input types and added logging for selected preprocessors.
…ging to better reflect the significance of selected implementations.
This commit introduces a new preprocessor implementation, `RFDETR_PREPROCESSOR_TRITON_UNIVERSAL_V1`, enhancing the RF-DETR model's preprocessing capabilities. The implementation supports various input types and optimizes CUDA processing. Additionally, it integrates with the `RFDetrForObjectDetectionTRT` class, allowing for streamlined preprocessing during inference. A new runtime class, `UniversalFastPreprocessRuntime`, is added to handle the preprocessing logic, and corresponding unit tests are included to ensure functionality and compatibility.
This commit introduces a new postprocessing module for RF-DETR, including the `FusedObjectDetectionPostprocessor` that optimizes object detection results using Triton. The implementation allows for explicit selection of postprocessor strategies, enhancing flexibility during inference. Additionally, the `RFDetrForObjectDetectionTRT` class is updated to support the new postprocessor, and corresponding unit tests are added to validate functionality and performance. This update aims to improve efficiency and streamline the postprocessing workflow for RF-DETR models.
…iable support This commit introduces support for environment variables in the RF-DETR preprocessing and postprocessing modules. New environment variables, `INFERENCE_MODELS_RFDETR_PREPROCESSOR` and `INFERENCE_MODELS_RFDETR_POSTPROCESSOR`, allow users to specify preprocessing and postprocessing strategies without modifying code. The `resolve_rfdetr_preprocessor` and `resolve_rfdetr_postprocessor` functions are updated to utilize these environment variables, enhancing flexibility. Additionally, new unit tests validate the correct behavior of these features, ensuring robust functionality across different configurations.
This commit introduces a new `RFDetrExecutionPlan` class that allows users to define a composed execution plan for RF-DETR inference, enabling independent selection of preprocessing, postprocessing, buffer strategy, scheduler, and engine plugin stages. The implementation ensures that explicit plans cannot be combined with legacy arguments, enhancing clarity and usability. Additionally, the `RFDetrForObjectDetectionTRT` class is updated to support this new execution plan, improving the flexibility of inference configurations. Corresponding updates are made to the preprocessing and postprocessing modules to integrate with the new execution plan structure.
…d integration This commit introduces a new documentation file detailing the Inference-Path Optimization Architecture, explaining the selection and execution of inference-path implementations. It includes a flowchart visualized with Mermaid to illustrate the architecture's components and their interactions. Additionally, the mkdocs configuration is updated to include the new documentation and enable Mermaid support for enhanced visual representation. References to this architecture are added in relevant sections of the environment variables and RF-DETR object detection documentation, improving clarity and accessibility for users.
This commit introduces a comprehensive refactor of the RF-DETR optimization architecture, creating reusable components for inference-path implementations. New modules for contracts, execution plans, and a context-aware registry are added to enhance flexibility and maintainability. The `RFDetrExecutionPlan` class is updated to support independent selection of preprocessing, postprocessing, and other stages, while ensuring compatibility with environment variables. Documentation is also improved to clarify the architecture and its components, facilitating better user understanding and integration.
…d postprocessor implementation selections to better reflect their significance in the inference process.
…and fallback mechanisms This commit introduces a new `CompatibilityResult` class to encapsulate the results of compatibility checks for preprocessing implementations. It adds methods for checking model and request compatibility across various preprocessors, ensuring that incompatible configurations are handled gracefully with declared fallbacks. The `resolve_preprocessor_for_model` and `resolve_preprocessor_for_request` functions are updated to utilize these compatibility checks, logging fallback reasons when applicable. Additionally, the documentation is updated to reflect these changes, improving clarity on how preprocessing selections are made and their implications for inference execution.
|
👋 Thanks for the pull request! Here is how automated Claude review works here, so you spend credits (and reviewer time) wisely. 🚦 This PR is marked Ready for review, so automated Claude review will run — and every pass spends real credits. Warning 💸 The Claude reviewer bills in credits, not vibesAutomated review spins up a real agent that reads real code and spends real credits on every pass. It is glad to help — but it is not a rubber duck, a linter you poke in a loop, or a substitute for reading the contributing guide. Treat it like an expensive senior reviewer whose time you booked, and show up prepared. Draft when unsure, Ready when you mean it:
However you get there, arrive prepared:
Reviews are not free. A draft costs nothing to review; a Ready PR is a promise that it is worth reviewing.
|
|
🤖 Claude review started at commit New commits are not auto-reviewed. Add the |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| See [Inference-Path Optimization Architecture](../contributors/inference-path-optimization-architecture.md) | ||
| for the selection model and the complete RF-DETR execution flow. | ||
|
|
||
| **`INFERENCE_MODELS_RFDETR_PREPROCESSOR`** |
There was a problem hiding this comment.
Medium — missing changelog entry (and version bump if 0.31.0 is already released).
This PR adds user-visible surface — three new env vars (INFERENCE_MODELS_RFDETR_PREPROCESSOR, ..._PREPROCESSOR_MAX_WORKERS, ..._POSTPROCESSOR), a new public API (RFDetrExecutionPlan), and new from_pretrained kwargs (rfdetr_preprocessor_max_workers, rfdetr_execution_plan) — but inference_models/docs/changelog.md has no entry describing it, and inference_models/pyproject.toml is unchanged (still 0.31.0, same as the base SHA).
Per the inference_models contribution convention, a functional/user-visible change requires a docs/changelog.md entry (under the current version heading) and, if 0.31.0 has already been published to PyPI, a version bump. Please add a changelog entry for the RF-DETR pre/post-processing optimization and confirm whether 0.31.0 still needs a bump.
There was a problem hiding this comment.
Changelog entry added.
|
⏳ This PR is on hold pending your answers — the review will not advance to sign-off and the PR should not proceed to merge until the IMPORTANT questions below are answered. Thanks for the thorough PR — the design is clean, the default ( IMPORTANT — please answer
Optional
Documentation (see inline)
Reviewed at HEAD: 748ad91 |
This commit introduces the `allow_compatibility_fallback` parameter to the `RFDetrExecutionPlan`, enabling users to specify whether to allow fallback to a base implementation when an explicitly requested implementation is incompatible. The documentation is updated to clarify the implications of this setting, and the relevant selection functions are modified to respect this new parameter. Additionally, unit tests are added to validate the behavior of the fallback mechanism across preprocessing and postprocessing stages, ensuring robust handling of compatibility checks during inference execution.
Issue should be fixed now.
Confirmed: 931 detection, 4 of them had a 1 px box difference between runs, this is on a |
Enhance the RFDetrForObjectDetectionTRT class to support an independent stage execution mode. This allows public preprocessing to return a ready tensor without relying on the model's readiness state during the forward pass. Update documentation to reflect this new feature and provide usage examples. Add integration tests to ensure the new functionality works as expected, matching outputs with composed inference. The default behavior remains asynchronous to preserve existing optimizations.
…ty checks - Change default preprocessor to `triton-universal-v1` and postprocessor to `triton-fused-v1` in the RF-DETR execution plan. - Update documentation to clarify the behavior when no explicit plan or environment overrides are provided. - Add compatibility checks in postprocessing and preprocessing classes to report Triton availability issues. - Enhance unit tests to validate the new default behavior and Triton dependency checks.
…ent stage execution - Updated RF-DETR TensorRT to default to `triton-universal-v1` preprocessing and `triton-fused-v1` postprocessing. - Introduced breaking changes for independent stage execution, requiring `independent_stage_execution=True` for callers using `pre_process()` and `forward()` separately. - Added documentation for new execution plan features, including environment variable controls and usage examples for independent stage execution. - Improved compatibility checks and metadata reporting for preprocessing and postprocessing implementations.
|
|
🤖 Claude review started at commit New commits are not auto-reviewed. Add the |
| if pre_processing_overrides is not None: | ||
| unsupported.append("pre-processing overrides") |
There was a problem hiding this comment.
Medium — the new default triton-universal-v1 preprocessor always falls back to base (and warns per request) on the standard inference-server path.
This check marks any non-None pre_processing_overrides as incompatible. But the core adapter that drives RF-DETR through the server always constructs one:
# inference/core/models/inference_models_adapters.py (InferenceModelsObjectDetectionAdapter)
def map_inference_kwargs(self, kwargs: dict) -> dict:
kwargs["input_color_format"] = "bgr"
pre_processing_overrides = PreProcessingOverrides( # never None
disable_contrast_enhancement=kwargs.get("disable_preproc_contrast", False),
disable_grayscale=kwargs.get("disable_preproc_grayscale", False),
disable_static_crop=kwargs.get("disable_preproc_static_crop", False),
)
kwargs["pre_processing_overrides"] = pre_processing_overrides
return kwargsPreProcessingOverrides defaults every field to False (a no-op), and check_model_compatibility above already rejects models whose config has static_crop/contrast/grayscale enabled — so for any model that reaches the Triton preprocessor these disable-flags are meaningless. Yet because the object is non-None, resolve_preprocessor_for_request (with the default allow_compatibility_fallback=True) falls back to base on every request, and pre_process() emits LOGGER.warning(...) on every call.
Net effect through the primary runtime:
- The advertised default preprocessing optimization (
triton-universal-v1) is never active for RF-DETR served viainference_models— it silently runsbase. (This is consistent with the PR's own snapshot note that "Preprocessing is identical — input tensor is bit-exact"; base vs. base is trivially bit-exact.) - A WARNING line is logged per inference — log spam under load.
Suggest inspecting the override fields (treat an all-False override as compatible) rather than rejecting on is not None, and/or de-duplicating the per-request fallback warning (warn-once). Please confirm the intended behavior.
There was a problem hiding this comment.
This is consistent with the PR's own snapshot note that "Preprocessing is identical — input tensor is bit-exact"; base vs. base is trivially bit-exact.
The snapshot is checked on inference_models model infer method thus doesn't go through the inference/core/models/inference_models_adapters.py. It doesn' resolve to the trivial solution.
There was a problem hiding this comment.
PreProcessingOverrides with default False values now handled correctly
There was a problem hiding this comment.
Warning only logged once per (stage, requested_id, effective_id, reason)
|
⏳ This PR is on hold pending your answer — the review will not advance to sign-off and the PR should not proceed to merge until the IMPORTANT question below is resolved. Thanks for the follow-ups. I re-reviewed the delta since my last pass (748ad91): the fallback logic is now symmetric across pre/post (_apply_declared_fallback + allow_compatibility_fallback), the defaults were flipped to the Triton path, independent_stage_execution was added, and a changelog entry now covers all of it. I verified the async default is safe through the standard infer() path (the exact preprocessed tensor flows through Model.infer to predict on the same instance, so PreprocessReadinessTracker keys correctly). The prior two IMPORTANT questions (fallback asymmetry, parity re-verification) are addressed. Flipping the default to Triton, however, surfaced one new concern. IMPORTANT — please answer
Optional / carry-forward
Reviewed at HEAD: 93e1b9c |
|
Release coordination (maintainers) — This PR makes a functional, user-visible change to inference-models requires a version bump for release. At release time, maintainers choose the version, update Reviewed at HEAD: 93e1b9c |
PawelPeczek-Roboflow
left a comment
PawelPeczek-Roboflow
left a comment
There was a problem hiding this comment.
I believe we need bump in version of inference models
…patibility checks - Introduced `FallbackWarningTracker` to ensure each distinct fallback warning is logged only once per model instance, improving logging efficiency. - Updated RF-DETR model to utilize the new tracker for preprocessing and postprocessing fallback warnings. - Enhanced documentation to clarify the behavior of active preprocessing overrides and their impact on compatibility checks. - Added unit tests to validate the functionality of the fallback warning system and its thread safety.
- Bump project version in `pyproject.toml` to 0.32.0. - Update `uv.lock` revision to 3. - Document changes in `changelog.md`, including updates to RF-DETR TensorRT object detection and other enhancements.
|
Inference models version bumped to |
dkosowski87
added
claude-review
|
🤖 Claude review started at commit New commits are not auto-reviewed. Add the |
|
Release coordination (maintainers) — refresh of the prior notice. The contributor has now bumped inference-models One cross-package heads-up for the eventual inference server release (not a blocker for this PR, and not owned by it):
Commands that informed this review: Reviewed at HEAD: 84b064b |
|
😎 PR passes the vibe-check and trust-me-bro verification. |
- Bumped the version of `inference-models` to 0.32.0 in `requirements.cpu.txt`, `requirements.gpu.txt`, `requirements.jetson.txt`, and `requirements.vino.txt` to ensure consistency across environments.
|
Inference requirements files updated for new |
…files for consistency
…tage handling - Update changelog to clarify the backward compatibility of direct RF-DETR TensorRT stage calls, ensuring public `pre_process()` synchronizes by default. - Revise documentation to reflect the new default behavior of public preprocessing and its interaction with composed inference calls. - Enhance the `RFDetrForObjectDetectionTRT` class to streamline the handling of independent stage execution, removing the need for the `independent_stage_execution` parameter in the constructor. - Improve integration tests to validate the new behavior and ensure consistency across public stage calls.
- Bump version in `pyproject.toml` and `uv.lock` to 0.32.0rc3 for consistency across the project.
PawelPeczek-Roboflow
left a comment
PawelPeczek-Roboflow
left a comment
There was a problem hiding this comment.
LGTM
when CI passes, 0.32.0 release to be done + requirements update
…files for consistency
2 participants
What does this PR do?
Linked issue: None.
This PR optimizes the RF-DETR TensorRT inference path while introducing a reusable architecture for selecting and managing alternative inference-stage implementations. It preserves the TensorRT semantic forward boundary while allowing preprocessing and postprocessing strategies to evolve independently.
The implementation introduces immutable execution plans, typed compatibility metadata, runtime execution contexts, and a registry responsible for strict implementation selection. RF-DETR clients can select implementations through an explicit execution plan or environment variables, including clients that cannot pass backend-specific arguments. Non-supported paths currently fallback to
baseprocessors. Reusable control-plane components live undermodels/optimization, while RF-DETR contracts, catalogs, configuration, and concrete implementations remain model-local. CUDA-event readiness tracking preserves asynchronous dependencies between preprocessing and TensorRT without attaching dynamic state to tensors. Runtime metadata exposes the resolved plan and selected implementations for profiling and validation.Main elements:
autoselectionType of Change
How to enable?
Latency Profile
Run ID:
profile-latency-20260716-191752Host:
orin-agx-jp62(Orin, compute capability 8.7)Date: 2026-07-16
Docker image:
roboflow/roboflow-inference-server-jetson-6.2.0:1.3.5Runtime ref:
optimize/rf-detr-object-detection(d00376b3)Profiler ref:
main(edd68971)Preprocessor / postprocessor:
triton-universal-v1/triton-fused-v1(env overrides)Iterations: 20 warmup / 100 measured (5 repetitions, convergence threshold 0.05)
Package: trt, fp16
Scenario:
camera_3840x2160_batch_4_high(3840×2160, batch 4)Scenario:
camera_640x480_batch_1_base(640×480, batch 4)Memory Profile
Run ID:
profile-memory-20260716-201948Host:
orin-agx-jp62(Orin, compute capability 8.7, integrated memory)Date: 2026-07-16
Docker image:
roboflow/roboflow-inference-server-jetson-6.2.0:1.3.5Iterations: 5 warmup / 20 measured
Scenario:
camera_3840x2160_batch_4_highRuntime ref:
optimize/rf-detr-object-detection(d00376b3)Package: trt, fp16
GPU memory used whole-device CUDA accounting (
cuda_device) with the pre-load baseline subtracted. Competing GPU workloads can still inflate whole-device readings.GPU memory
Snapshot parity validation
Compared end-to-end inference snapshots on Orin AGX (JP 6.2) to verify that the Triton fused pre/post path produces the same results as the baseline runtime.
Method
Used the
snapshot-model-outputremote-runner recipe to capture exact invocation inputs and postprocessed outputs (typed JSON + lossless NPY sidecars) for a single deterministic workload:orin-agx-jp62rfdetr-smallbbf73c3014b86386434c8051f132cdf2(TRT fp16)camera_640x480_batch_1_base000000000139.jpg→ 640×480roboflow/roboflow-inference-server-jetson-6.2.0:1.3.5inference-profiler@main(edd6897)Runs compared
snapshot-model-output-20260716-155717snapshot-model-output-20260716-132958main(e3d91414)codex/rfdetr-triton-fused-postprocess-v1(c9c6fc54)triton-universal-v1triton-fused-v1Array comparison
Compared all NPY artifacts under
model-output-snapshots/bbf73c3014b86386434c8051f132cdf2/default/camera_640x480_batch_1_base/arrays/:input-0000.npyoutput-0002.npyoutput-0000.npyoutput-0001.npyTie-breaking note
14/16 detections match at the same array index. The only positional difference is indices 11 and 12, where two detections with identical confidence (
0.58130306) are swapped:main)[490, 193, 514, 320][166, 263, 186, 301][166, 263, 186, 301][490, 193, 514, 320]When sorted by
(confidence, class_id, xyxy), all 16 detections match exactly between runs. This is consistent with non-deterministic tie-breaking in NMS/postprocess ordering, not a semantic output difference.Conclusion
For this workload on Orin AGX:
This snapshot comparison demonstrates functional parity between the baseline and Triton fused pre/post implementations for
rfdetr-smallTRT fp16 on Jetson Orin.Testing
Unit tests
Checklist