diff --git a/inference_models/docs/changelog.md b/inference_models/docs/changelog.md index e9a1562cb3..325905365a 100644 --- a/inference_models/docs/changelog.md +++ b/inference_models/docs/changelog.md @@ -6,6 +6,25 @@ Add user-facing changes below using `### Added`, `### Changed`, `### Fixed`, or `### Removed` subsections as appropriate. --- + +## `0.32.0` + +### Changed + +- RF-DETR TensorRT object detection now selects `triton-universal-v1` + preprocessing and `triton-fused-v1` postprocessing by default. Incompatible requests + use the declared `base` implementation unless strict selection is requested through + an explicit execution plan. The selected implementations can be controlled with an + `RFDetrExecutionPlan` or the `INFERENCE_MODELS_RFDETR_PREPROCESSOR` and + `INFERENCE_MODELS_RFDETR_POSTPROCESSOR` environment variables. No-op preprocessing + override containers used by the inference server remain on the optimized path; + active overrides use the declared fallback. Repeated occurrences of the same + request-level fallback warning are logged only once per model instance. +- Direct RF-DETR TensorRT stage calls remain backward compatible: public + `pre_process()` synchronizes before returning by default, so its output is ready for + an independent `forward()` call. Composed `model(...)` and `infer()` calls explicitly + use the asynchronous exact-tensor readiness handoff to avoid a host synchronization. + ### Fixed - SAM3 concept-segmentation postprocessing no longer scales its memory working set with @@ -17,10 +36,15 @@ Add user-facing changes below using `### Added`, `### Changed`, `### Fixed`, or ### Added +- Composable RF-DETR TensorRT execution plans, implementation contracts and registries, + compatibility-aware implementation selection, and runtime metadata reporting the + requested and effective preprocessing and postprocessing implementations. - `segment_with_text_prompts` accepts `max_detections` (top-k by score, applied before mask interpolation; default `-1` = uncapped) and `mask_format` (`"dense"` default, or `"rle"` for COCO RLE at original resolution). +--- + ## `0.31.0` ### Fixed diff --git a/inference_models/docs/contributors/inference-path-optimization-architecture.md b/inference_models/docs/contributors/inference-path-optimization-architecture.md new file mode 100644 index 0000000000..2196e7152a --- /dev/null +++ b/inference_models/docs/contributors/inference-path-optimization-architecture.md @@ -0,0 +1,299 @@ +# Inference-Path Optimization Architecture + +This guide explains how independently selectable inference-path implementations are +described, selected, and executed. It first describes the reusable architecture and +then maps it to the current RF-DETR TensorRT object-detection implementation. + +The design keeps two concerns separate: + +- **selection** decides which implementation is valid for a runtime; +- **execution** runs the selected implementation without changing the model's semantic + forward pass. + +## Component overview + +```mermaid +flowchart TD + selection["Selection inputs
explicit plan / environment"] + plan["ExecutionPlan
stage IDs only"] + catalog["Catalog
metadata + constructors"] + registry["ImplementationRegistry
registered instances"] + context["ExecutionContext
device + scenario + resolved axes + stream"] + contracts["Contracts
metadata + compatibility + requests + results + protocols"] + implementations["Implementations
base and optimized choices in separate modules"] + model["Model
preprocess → protected forward → postprocess"] + observable["Runtime observability
selected IDs + resolved plan + metadata"] + + selection --> plan + plan -->|requested stage IDs| registry + implementations -->|available classes| catalog + catalog -->|constructs and registers| registry + context -->|runtime constraints| registry + registry -->|selected stage objects| model + model --> observable + + contracts -.->|defines plan shape| plan + contracts -.->|defines resolution inputs| registry + contracts -.->|defines stage interfaces| implementations +``` + +### Contracts + +Shared contracts define the stable language used by every optimized inference path. +They include: + +- `OptimizationMetadata`, including the stable implementation ID, stage, version, + target, input constraints, dependencies, numerical behavior, stream behavior, + output contract, fallback ID, and validation history; +- `ExecutionContext`, which describes the actual device, scenario, resolved input + axes, compute capability, and current stream; +- the common `InferenceStage` compatibility protocol. + +Stage-specific requests, results, and protocols stay in the model namespace because +their signatures depend on that model's inputs and outputs. + +Metadata is immutable and can be serialized with `to_dict()`. An implementation can +therefore be inspected without executing it, and the resolved runtime configuration +can be attached to profiling results. + +### Catalog + +The catalog is the inventory of available choices. It exposes read-only metadata maps +for introspection and constructs a registry containing the corresponding runtime +objects. It does not decide which choice should run. + +Keeping construction in the catalog avoids importing one concrete implementation from +another and gives the model a single place to assemble all available stages. + +### Execution plan + +An execution plan is an immutable collection of implementation IDs, one per selectable +stage: + +```python +RFDetrExecutionPlan( + preprocessor_id="triton-universal-v1", + buffer_strategy_id="base", + scheduler_id="base", + postprocessor_id="triton-fused-v1", + engine_plugin_id="base", + allow_compatibility_fallback=True, +) +``` + +The plan contains choices, not implementation objects or mutable runtime state. This +makes it suitable for configuration, logging, and comparison between profiling runs. + +### Implementation registry + +The registry owns the constructed stage objects and resolves a requested ID against an +`ExecutionContext`. Resolution follows these rules: + +1. `base` selects the preserved reference implementation. +2. An explicit implementation ID selects that implementation when compatible. A + declared compatibility miss may follow its observable `fallback_id`. +3. `auto` selects a compatible implementation only when it has a matching validated + environment; otherwise it selects `base`. +4. Unknown IDs and failures during implementation execution never fall back. + +Compatibility fallback is decided before execution and records the requested ID, +effective ID, and reason. It does not catch compilation, CUDA, allocation, or other +unexpected runtime failures. + +The same policy applies to every selectable stage. Set +`allow_compatibility_fallback=False` when an explicitly requested implementation must +either run or raise. The default is `True`, which preserves the base inference path for +contracts that an optimized implementation declares unsupported. + +The catalog answers **what exists**. The registry answers **what may run here**. + +### Implementations + +Each implementation lives in its own module and depends on the shared contracts. Base +implementations preserve the original behavior, while optimized implementations make +their own compatibility checks before performing work. + +Implementation-local state may include streams, events, reusable buffers, or bounded +caches. That state belongs to the implementation object rather than the plan or +metadata. + +### Runtime observability + +The resolved plan is the model-level plan that actually runs. Models expose both +selected IDs and JSON-compatible selection metadata. The metadata records model-level +and latest per-request requested/effective IDs plus any fallback reason, so profiling +and validation output does not need to infer selection from environment variables or +log messages. + +## Current RF-DETR integration + +The current implementation applies this architecture to RF-DETR TensorRT object +detection. The TensorRT semantic forward pass remains protected and is not a selectable +implementation. + +### Model initialization and selection + +```mermaid +flowchart TD + load["AutoModel.from_pretrained(...)"] --> init["RFDetrForObjectDetectionTRT.__init__"] + init --> explicit{"Explicit RFDetrExecutionPlan?"} + explicit -->|yes| use_plan["Use plan
do not read selection environment"] + explicit -->|no| precedence["Resolve each stage
environment → optimized default"] + use_plan --> requested["Requested RFDetrExecutionPlan"] + precedence --> requested + + requested --> build["build_rfdetr_implementation_registry
(device, max_workers)"] + + subgraph preprocessors["Preprocessor implementations"] + pre_base["base"] + pre_threaded["threaded-exact-v1"] + pre_triton["triton-universal-v1"] + end + + subgraph postprocessors["Postprocessor implementations"] + post_base["base"] + post_triton["triton-fused-v1"] + end + + pre_base --> build + pre_threaded --> build + pre_triton --> build + post_base --> build + post_triton --> build + + build --> resolve["Resolve preprocessor and postprocessor
with ExecutionContext"] + resolve --> store["Store resolved plan"] + store --> metadata["Expose optimization_runtime_metadata"] +``` + +Environment management remains available for clients that cannot pass new model +arguments: + +| Variable | Stage | Example value | +|---|---|---| +| `INFERENCE_MODELS_RFDETR_PREPROCESSOR` | preprocessing | `triton-universal-v1` | +| `INFERENCE_MODELS_RFDETR_PREPROCESSOR_MAX_WORKERS` | threaded preprocessing | `4` | +| `INFERENCE_MODELS_RFDETR_POSTPROCESSOR` | postprocessing | `triton-fused-v1` | + +An explicit plan has the clearest provenance and takes precedence over environment +variables. Environment values are read only when no plan is supplied. When neither an +explicit plan nor environment overrides are present, RF-DETR selects +`triton-universal-v1` preprocessing and `triton-fused-v1` postprocessing. A declared +contract mismatch or unavailable Triton dependency follows the implementation's +`base` fallback before execution. + +### Per-request execution + +```mermaid +flowchart TD + inputs["Image inputs
NumPy or torch.Tensor
CPU or CUDA"] + invocation{"Invocation"} + pre_request["PreprocessRequest + ExecutionContext"] + pre_selected{"Selected Preprocessor
compatible with request?"} + pre_base["base / threaded exact
synchronize before return"] + pre_triton["Triton universal
record CUDA ready event"] + boundary{"independent_stage_execution?
default: true"} + independent["Synchronize producer
do not register readiness"] + readiness["PreprocessReadinessTracker
associate readiness with exact tensor"] + wait["TensorRT inference stream
wait on ready event when present"] + forward["Protected TensorRT forward"] + outputs["Boxes + logits on CUDA"] + post_request["PostprocessRequest + ExecutionContext"] + post_selected{"Selected Postprocessor"} + post_base["Base PyTorch path"] + post_triton["Fused Triton path"] + detections["list[Detections]"] + + inputs --> invocation --> pre_request --> pre_selected + pre_selected -->|base or declared fallback| pre_base + pre_selected -->|compatible triton-universal-v1| pre_triton + pre_base --> boundary + pre_triton --> boundary + invocation -.->|public pre_process: true| boundary + invocation -.->|composed infer: false| boundary + boundary -->|true| independent --> forward + boundary -->|false| readiness + readiness --> wait --> forward --> outputs --> post_request --> post_selected + post_selected -->|base| post_base + post_selected -->|triton-fused-v1| post_triton + post_base --> detections + post_triton --> detections +``` + +`PreprocessReadinessTracker` is deliberately separate from tensors. It uses the exact +tensor identity and a weak reference to transfer an optional CUDA event from +preprocessing to the TensorRT consumer without adding dynamic attributes to framework +tensors. + +Public `pre_process()` calls default to `independent_stage_execution=True`: they +synchronize their producer before returning and do not add a tracker entry. Composed +`model(...)` and `infer()` calls explicitly pass `False`, allowing preprocessing to +return asynchronously after associating its CUDA event with the exact output tensor. +`forward()` always checks for such an entry and waits on its event when present; an +independently prepared ready tensor has no entry and proceeds normally. + +```python +model = AutoModel.from_pretrained( + "rfdetr-small", + backend="trt", +) +preprocessed, metadata = model.pre_process(image) +raw_predictions = model.forward(preprocessed) +detections = model.post_process(raw_predictions, metadata) +``` + +This is an invocation-boundary policy rather than an implementation choice, so it is +not part of `InferenceExecutionPlan`. Safe standalone behavior is the public default; +the composed inference path opts into the optimized asynchronous handoff internally. + +The preprocessing, inference, and postprocessing streams are reused. Events express +the GPU dependency at the consumer boundary; the protected TensorRT forward does not +need to know which preprocessor produced its input. + +### RF-DETR files and responsibilities + +| Path | Responsibility | +|---|---| +| `models/optimization/contracts.py` | Reusable metadata, compatibility, runtime context, and base stage protocol | +| `models/optimization/execution_plan.py` | Reusable immutable execution-plan representation | +| `models/optimization/fallback_warnings.py` | Thread-safe per-model de-duplication of request fallback warnings | +| `models/optimization/ids.py` | Conventional `base` and `auto` implementation IDs | +| `models/optimization/registry.py` | Strict explicit and conservative automatic resolution | +| `models/optimization/torch_readiness.py` | Generic one-shot state handoff tied to exact tensor identity | +| `models/rfdetr/optimization/contracts.py` | RF-DETR requests, results, and stage-specific protocols | +| `models/rfdetr/optimization/ids.py` | Stable implementation IDs and environment-variable names | +| `models/rfdetr/optimization/execution_plan.py` | RF-DETR environment resolution and supported-stage validation | +| `models/rfdetr/optimization/catalog.py` | Read-only metadata catalogs and registry construction | +| `models/rfdetr/optimization/readiness.py` | RF-DETR readiness payload and shared-tracker adapter | +| `models/rfdetr/optimization/preprocessors/` | One module per preprocessing choice | +| `models/rfdetr/optimization/postprocessors/` | One module per postprocessing choice | +| `models/rfdetr/rfdetr_object_detection_trt.py` | Plan integration and request-stage orchestration | + +### Current boundaries + +- Preprocessing and postprocessing have selectable implementations. +- Buffer strategy, scheduler, and engine-plugin slots exist in the plan but currently + accept only `base`. Selecting an unimplemented value raises an error. +- `auto` remains on `base` until machine-readable validation records are added for a + matching runtime environment. +- Static model incompatibilities resolve the stored plan through the implementation's + declared fallback. Request-only incompatibilities use the fallback for that request. +- Fallback decisions are logged and carried with preprocessing readiness metadata; + execution failures still propagate. +- Target-device profiling and output-snapshot parity checks remain required before an + optimized choice is promoted for automatic selection. + +## Adding another implementation + +1. Give the implementation a stable ID in `ids.py`. +2. Add a separate implementation module that satisfies the stage protocol. +3. Declare immutable compatibility and behavioral metadata. +4. Reject unsupported explicit inputs with an actionable error. +5. Register the implementation in `catalog.py`. +6. Add contract, compatibility, numerical-parity, and selection tests. +7. Profile and compare snapshots on the target device. +8. Add validated environment records only after the target results justify automatic + selection. + +This sequence keeps a new optimization independently selectable, attributable in +profiling, and removable without changing the semantic model forward pass. diff --git a/inference_models/docs/how-to/environment-variables.md b/inference_models/docs/how-to/environment-variables.md index 3d46de5bc1..4603f2e7e9 100644 --- a/inference_models/docs/how-to/environment-variables.md +++ b/inference_models/docs/how-to/environment-variables.md @@ -422,6 +422,102 @@ Default: Inherits from `INFERENCE_MODELS_DEFAULT_CONFIDENCE` export INFERENCE_MODELS_RFDETR_DEFAULT_CONFIDENCE="0.5" ``` +The following variables select RF-DETR TensorRT pipeline implementations when a client +cannot pass backend-specific `from_pretrained` arguments. Explicit arguments take +precedence over these environment variables. + +See [Inference-Path Optimization Architecture](../contributors/inference-path-optimization-architecture.md) +for the selection model and the complete RF-DETR execution flow. + +**`INFERENCE_MODELS_RFDETR_PREPROCESSOR`** +Default: `triton-universal-v1` + +Supported values: `base`, `auto`, `threaded-exact-v1`, `triton-universal-v1`. + +```bash +export INFERENCE_MODELS_RFDETR_PREPROCESSOR="triton-universal-v1" +``` + +**`INFERENCE_MODELS_RFDETR_PREPROCESSOR_MAX_WORKERS`** +Default: `4` + +Controls the bounded worker count used by `threaded-exact-v1`. + +```bash +export INFERENCE_MODELS_RFDETR_PREPROCESSOR_MAX_WORKERS="4" +``` + +**`INFERENCE_MODELS_RFDETR_POSTPROCESSOR`** +Default: `triton-fused-v1` + +Supported values: `base`, `auto`, `triton-fused-v1`. + +```bash +export INFERENCE_MODELS_RFDETR_POSTPROCESSOR="triton-fused-v1" +``` + +Code that can pass backend-specific arguments may instead provide a composed, immutable +execution plan: + +```python +from inference_models import AutoModel +from inference_models.models.rfdetr.optimization.execution_plan import ( + RFDetrExecutionPlan, +) + +plan = RFDetrExecutionPlan( + preprocessor_id="triton-universal-v1", + postprocessor_id="triton-fused-v1", + allow_compatibility_fallback=True, +) +model = AutoModel.from_pretrained( + "rfdetr-small", + backend="trt", + rfdetr_execution_plan=plan, +) +``` + +Public preprocessing synchronizes by default, so its result can be consumed by an +independent `forward()` call without relying on model-owned readiness state: + +```python +model = AutoModel.from_pretrained( + "rfdetr-small", + backend="trt", + rfdetr_execution_plan=plan, +) +preprocessed, metadata = model.pre_process(image) +raw_predictions = model.forward(preprocessed) +``` + +This invocation-boundary policy is intentionally separate from the execution plan. +Composed `model(...)` and `infer()` calls pass +`independent_stage_execution=False` to preprocessing internally, record a CUDA event, +and let `forward()` wait on that event without a host synchronization. + +The plan also reserves independently selectable buffer-strategy, scheduler, and engine +plugin stages. Those stages currently accept only `base`. When supplied, an explicit +plan takes precedence and the implementation-selection environment variables are not +read. + +When a selected optimized stage declares that it cannot preserve a model or request +contract, RF-DETR uses its declared `base` fallback and records the requested +implementation, effective implementation, and reason in logs and runtime metadata. +This policy applies consistently to preprocessing and postprocessing. Set +`allow_compatibility_fallback=False` in an explicit plan to require the selected +implementation or an error. Compilation, CUDA, allocation, and other execution +failures are never converted into fallbacks. + +An all-`False` `PreProcessingOverrides` object is a no-op and remains compatible with +`triton-universal-v1`. Requests with any active preprocessing override use the declared +`base` fallback. A distinct request-level fallback reason is warned once per model +instance rather than once per inference. + +When Triton is unavailable, uint8 universal preprocessing and fused postprocessing +declare a compatibility miss and use their `base` fallback. Floating-point tensor +preprocessing remains eligible for `triton-universal-v1` because that input path uses +Torch operations and does not require Triton kernels. + #### Roboflow Instant **`INFERENCE_MODELS_ROBOFLOW_INSTANT_DEFAULT_CONFIDENCE`** diff --git a/inference_models/docs/models/rfdetr-object-detection.md b/inference_models/docs/models/rfdetr-object-detection.md index f02c1b57c1..5a92621b53 100644 --- a/inference_models/docs/models/rfdetr-object-detection.md +++ b/inference_models/docs/models/rfdetr-object-detection.md @@ -37,6 +37,9 @@ All pre-trained RF-DETR object detection models are trained on the COCO dataset ## Supported Backends +For the selectable TensorRT preprocessing and postprocessing architecture, see +[Inference-Path Optimization Architecture](../contributors/inference-path-optimization-architecture.md#current-rf-detr-integration). + | Backend | Extras Required | |---------|----------------| | `torch` | `torch-cpu`, `torch-cu118`, `torch-cu124`, `torch-cu126`, `torch-cu128`, `torch-jp6-cu126` | @@ -89,6 +92,33 @@ annotated_image = bounding_box_annotator.annotate(image, detections) cv2.imwrite("annotated.jpg", annotated_image) ``` +### Calling TensorRT stages independently + +The TensorRT backend normally keeps optimized preprocessing asynchronous. Its +`pre_process()` and `forward()` methods coordinate through readiness state owned by the +model instance, avoiding a host synchronization in the normal `model(...)` path. + +If your application calls `pre_process()`, `forward()`, and `post_process()` separately, +the default public-stage behavior is safe without additional model configuration: + +```python +from inference_models import AutoModel + +model = AutoModel.from_pretrained( + "rfdetr-base", + backend="trt", +) + +preprocessed, metadata = model.pre_process(image) +raw_predictions = model.forward(preprocessed) +predictions = model.post_process(raw_predictions, metadata) +``` + +Public `pre_process()` synchronizes its CUDA producer before returning, so its tensor can +be passed to `forward()` independently and does not depend on readiness state retained +by the model. Composed `model(...)` and `infer()` calls use an internal event-based +handoff instead, avoiding this host synchronization on the optimized inference path. + ## Trained RF-DETR Outside Roboflow? Use with `inference-models` RF-DETR offers a **seamless training-to-deployment workflow** that makes it incredibly easy to go from training to production. @@ -158,4 +188,3 @@ cv2.imwrite("annotated.jpg", annotated_image) - ✅ **Production-ready** - Leverage all `inference-models` features (multi-backend, caching, optimization) This seamless workflow eliminates the traditional friction between training and deployment, letting you iterate faster and deploy with confidence. - diff --git a/inference_models/inference_models/models/optimization/__init__.py b/inference_models/inference_models/models/optimization/__init__.py new file mode 100644 index 0000000000..5d5b43586f --- /dev/null +++ b/inference_models/inference_models/models/optimization/__init__.py @@ -0,0 +1 @@ +"""Reusable contracts and selection utilities for optimized inference paths.""" diff --git a/inference_models/inference_models/models/optimization/contracts.py b/inference_models/inference_models/models/optimization/contracts.py new file mode 100644 index 0000000000..c8a872c016 --- /dev/null +++ b/inference_models/inference_models/models/optimization/contracts.py @@ -0,0 +1,284 @@ +"""Reusable metadata and runtime contracts for inference-path implementations.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from enum import Enum +from types import MappingProxyType +from typing import Any, Dict, Literal, Mapping, Optional, Protocol, Tuple + + +def immutable_mapping(values: Optional[Mapping[str, Any]] = None) -> Mapping[str, Any]: + """Create an immutable shallow copy of a metadata mapping. + + Args: + values: Optional source mapping. + + Returns: + Read-only mapping detached from the source. + """ + immutable = MappingProxyType(dict(values or {})) + + return immutable + + +@dataclass(frozen=True) +class CompatibilityResult: + """Result of checking an implementation against a concrete contract.""" + + supported: bool + reasons: Tuple[str, ...] = () + + @classmethod + def compatible(cls) -> "CompatibilityResult": + """Create a successful compatibility result. + + Returns: + Result indicating that the implementation may execute. + """ + result = cls(supported=True) + + return result + + @classmethod + def incompatible(cls, *reasons: str) -> "CompatibilityResult": + """Create an unsupported compatibility result. + + Args: + *reasons: Human-readable incompatibility reasons. + + Returns: + Result indicating that a fallback or error is required. + """ + result = cls(supported=False, reasons=tuple(reasons)) + + return result + + @property + def reason(self) -> str: + """Return all incompatibility reasons as one readable value. + + Returns: + Comma-separated reasons, or an empty string when compatible. + """ + reason = ", ".join(self.reasons) + + return reason + + +class OptimizationStage(str, Enum): + """Common selectable stages in an inference path.""" + + PREPROCESS = "preprocess" + BUFFER_STRATEGY = "buffer_strategy" + SCHEDULER = "scheduler" + POSTPROCESS = "postprocess" + ENGINE_PLUGIN = "engine_plugin" + + +@dataclass(frozen=True) +class DeviceCompatibility: + """Hardware compatibility declared by one implementation.""" + + device_kind: Literal["cpu", "gpu"] + device_families: Tuple[str, ...] = () + minimum_compute_capability: Optional[Tuple[int, int]] = None + + +@dataclass(frozen=True) +class InputCompatibility: + """Input constraints declared by one implementation.""" + + scenarios: Tuple[str, ...] + axis_constraints: Mapping[str, Any] = field(default_factory=immutable_mapping) + dtypes: Tuple[str, ...] = () + layouts: Tuple[str, ...] = () + + def __post_init__(self) -> None: + object.__setattr__( + self, + "axis_constraints", + immutable_mapping(self.axis_constraints), + ) + + +@dataclass(frozen=True) +class ValidationEnvironment: + """One measured environment for an implementation.""" + + machine_type: str + device_kind: Literal["cpu", "gpu"] + device_name: str + scenario: str + resolved_axes: Mapping[str, Any] + runtime_versions: Mapping[str, str] + source_commit: str + profiling_bundle: str + status: Literal["validated", "failed", "inconclusive"] + + def __post_init__(self) -> None: + object.__setattr__(self, "resolved_axes", immutable_mapping(self.resolved_axes)) + object.__setattr__( + self, + "runtime_versions", + immutable_mapping(self.runtime_versions), + ) + + def matches(self, context: "ExecutionContext") -> bool: + """Return whether this validation record matches a runtime context. + + Args: + context: Runtime context being resolved. + + Returns: + Whether validated target and scenario identity match. + """ + target_matches = ( + self.status == "validated" + and self.device_kind == context.device_kind + and self.device_name == context.device_name + ) + scenario_matches = self.scenario in {context.scenario, "*"} + + return target_matches and scenario_matches + + +@dataclass(frozen=True) +class OptimizationMetadata: + """Stable metadata for one inference-stage implementation.""" + + implementation_id: str + stage: OptimizationStage + version: str + target: DeviceCompatibility + inputs: InputCompatibility + dependencies: Tuple[str, ...] + fallback_id: str + changes_numerics: bool + supports_concurrency: bool + supports_cuda_graphs: bool + output_contract: Mapping[str, Any] = field(default_factory=immutable_mapping) + numerical_behavior: str = "" + stream_behavior: str = "" + validated_environments: Tuple[ValidationEnvironment, ...] = () + + def __post_init__(self) -> None: + object.__setattr__( + self, + "output_contract", + immutable_mapping(self.output_contract), + ) + + def to_dict(self) -> Dict[str, Any]: + """Serialize metadata for logs and runtime-profile records. + + Returns: + JSON-compatible metadata dictionary. + """ + environments = [ + { + "machine_type": environment.machine_type, + "device_kind": environment.device_kind, + "device_name": environment.device_name, + "scenario": environment.scenario, + "resolved_axes": dict(environment.resolved_axes), + "runtime_versions": dict(environment.runtime_versions), + "source_commit": environment.source_commit, + "profiling_bundle": environment.profiling_bundle, + "status": environment.status, + } + for environment in self.validated_environments + ] + serialized = { + "implementation_id": self.implementation_id, + "stage": self.stage.value, + "version": self.version, + "target": { + "device_kind": self.target.device_kind, + "device_families": list(self.target.device_families), + "minimum_compute_capability": self.target.minimum_compute_capability, + }, + "inputs": { + "scenarios": list(self.inputs.scenarios), + "axis_constraints": dict(self.inputs.axis_constraints), + "dtypes": list(self.inputs.dtypes), + "layouts": list(self.inputs.layouts), + }, + "dependencies": list(self.dependencies), + "fallback_id": self.fallback_id, + "changes_numerics": self.changes_numerics, + "supports_concurrency": self.supports_concurrency, + "supports_cuda_graphs": self.supports_cuda_graphs, + "output_contract": dict(self.output_contract), + "numerical_behavior": self.numerical_behavior, + "stream_behavior": self.stream_behavior, + "validated_environments": environments, + } + + return serialized + + +@dataclass(frozen=True) +class ExecutionContext: + """Runtime target and request context used for stage resolution.""" + + device_kind: Literal["cpu", "gpu"] + device: str + device_name: str + machine_type: str + scenario: str + resolved_axes: Mapping[str, Any] = field(default_factory=immutable_mapping) + current_stream: Optional[Any] = None + device_family: Optional[str] = None + compute_capability: Optional[Tuple[int, int]] = None + + def __post_init__(self) -> None: + object.__setattr__(self, "resolved_axes", immutable_mapping(self.resolved_axes)) + + +def metadata_supports_context( + metadata: OptimizationMetadata, + context: ExecutionContext, +) -> bool: + """Return whether declared target constraints support a runtime context. + + Args: + metadata: Implementation compatibility metadata. + context: Runtime target and request context. + + Returns: + Whether the target constraints match. + """ + target = metadata.target + if target.device_kind != context.device_kind: + return False + if ( + target.device_families + and context.device_family is not None + and context.device_family not in target.device_families + ): + return False + if ( + target.minimum_compute_capability is not None + and context.compute_capability is not None + and context.compute_capability < target.minimum_compute_capability + ): + return False + + return True + + +class InferenceStage(Protocol): + """Common interface implemented by every selectable stage.""" + + metadata: OptimizationMetadata + + def is_compatible(self, context: ExecutionContext) -> bool: + """Return whether the stage supports a runtime context. + + Args: + context: Runtime target and request context. + + Returns: + Whether the stage is compatible. + """ diff --git a/inference_models/inference_models/models/optimization/execution_plan.py b/inference_models/inference_models/models/optimization/execution_plan.py new file mode 100644 index 0000000000..8de7cd62a3 --- /dev/null +++ b/inference_models/inference_models/models/optimization/execution_plan.py @@ -0,0 +1,35 @@ +"""Reusable composed inference execution-plan representation.""" + +from dataclasses import dataclass +from typing import Any, Dict + +from inference_models.models.optimization.ids import BASE_IMPLEMENTATION_ID + + +@dataclass(frozen=True) +class InferenceExecutionPlan: + """Independent implementation selections for an inference path.""" + + preprocessor_id: str = BASE_IMPLEMENTATION_ID + buffer_strategy_id: str = BASE_IMPLEMENTATION_ID + scheduler_id: str = BASE_IMPLEMENTATION_ID + postprocessor_id: str = BASE_IMPLEMENTATION_ID + engine_plugin_id: str = BASE_IMPLEMENTATION_ID + allow_compatibility_fallback: bool = True + + def to_dict(self) -> Dict[str, Any]: + """Serialize the composed execution plan. + + Returns: + Stage names mapped to selected implementation IDs. + """ + serialized = { + "preprocessor": self.preprocessor_id, + "buffer_strategy": self.buffer_strategy_id, + "scheduler": self.scheduler_id, + "postprocessor": self.postprocessor_id, + "engine_plugin": self.engine_plugin_id, + "allow_compatibility_fallback": self.allow_compatibility_fallback, + } + + return serialized diff --git a/inference_models/inference_models/models/optimization/fallback_warnings.py b/inference_models/inference_models/models/optimization/fallback_warnings.py new file mode 100644 index 0000000000..cb5a2b3bad --- /dev/null +++ b/inference_models/inference_models/models/optimization/fallback_warnings.py @@ -0,0 +1,44 @@ +"""Thread-safe warning de-duplication for compatibility fallbacks.""" + +import threading +from typing import Optional, Set, Tuple + +from inference_models.models.optimization.contracts import OptimizationStage + +FallbackWarningKey = Tuple[OptimizationStage, str, str, Optional[str]] + + +class FallbackWarningTracker: + """Claim each distinct fallback warning once per tracker instance.""" + + def __init__(self) -> None: + self._claimed: Set[FallbackWarningKey] = set() + self._lock = threading.Lock() + + def claim( + self, + *, + stage: OptimizationStage, + requested_id: str, + effective_id: str, + reason: Optional[str], + ) -> bool: + """Return whether the caller should emit this fallback warning. + + Args: + stage: Inference stage that followed a fallback. + requested_id: Originally selected implementation ID. + effective_id: Implementation ID used after fallback. + reason: Compatibility reason that caused the fallback. + + Returns: + ``True`` for the first claim of a distinct warning key and + ``False`` for subsequent claims. + """ + key = (stage, requested_id, effective_id, reason) + with self._lock: + if key in self._claimed: + return False + self._claimed.add(key) + + return True diff --git a/inference_models/inference_models/models/optimization/ids.py b/inference_models/inference_models/models/optimization/ids.py new file mode 100644 index 0000000000..fe0cc2d4e0 --- /dev/null +++ b/inference_models/inference_models/models/optimization/ids.py @@ -0,0 +1,4 @@ +"""Stable conventional IDs shared by inference-path optimizations.""" + +BASE_IMPLEMENTATION_ID = "base" +AUTO_IMPLEMENTATION_ID = "auto" diff --git a/inference_models/inference_models/models/optimization/registry.py b/inference_models/inference_models/models/optimization/registry.py new file mode 100644 index 0000000000..25d6900856 --- /dev/null +++ b/inference_models/inference_models/models/optimization/registry.py @@ -0,0 +1,170 @@ +"""Context-aware registry for selectable inference-stage implementations.""" + +from __future__ import annotations + +from collections import defaultdict +from typing import DefaultDict, Dict, Iterable, Tuple + +from inference_models.errors import ModelRuntimeError +from inference_models.models.optimization.contracts import ( + ExecutionContext, + InferenceStage, + OptimizationStage, +) +from inference_models.models.optimization.ids import ( + AUTO_IMPLEMENTATION_ID, + BASE_IMPLEMENTATION_ID, +) + + +class ImplementationRegistry: + """Register and resolve typed inference-stage implementations.""" + + def __init__( + self, + *, + scope_name: str, + base_id: str = BASE_IMPLEMENTATION_ID, + auto_id: str = AUTO_IMPLEMENTATION_ID, + ) -> None: + self._scope_name = scope_name + self._base_id = base_id + self._auto_id = auto_id + self._implementations: DefaultDict[ + OptimizationStage, Dict[str, InferenceStage] + ] = defaultdict(dict) + + def register(self, implementation: InferenceStage) -> None: + """Register one implementation by stage and stable ID. + + Args: + implementation: Typed stage implementation. + + Raises: + ValueError: If the stage and ID are already registered. + """ + metadata = implementation.metadata + stage_implementations = self._implementations[metadata.stage] + if metadata.implementation_id in stage_implementations: + raise ValueError( + f"Duplicate {metadata.stage.value} implementation " + f"{metadata.implementation_id!r}." + ) + stage_implementations[metadata.implementation_id] = implementation + + def resolve( + self, + *, + stage: OptimizationStage, + requested_id: str, + context: ExecutionContext, + ) -> InferenceStage: + """Resolve one compatible explicit or automatic implementation. + + Args: + stage: Stage category being selected. + requested_id: Stable implementation ID or the automatic-selection ID. + context: Runtime target and request context. + + Returns: + Compatible implementation instance. + + Raises: + ModelRuntimeError: If the ID is unknown or incompatible. + """ + stage_implementations = self._implementations.get(stage, {}) + if requested_id == self._auto_id: + implementation = self._resolve_auto( + stage=stage, + implementations=stage_implementations.values(), + context=context, + ) + + return implementation + + implementation = stage_implementations.get(requested_id) + if implementation is None: + available = ", ".join(sorted([self._auto_id, *stage_implementations])) + raise ModelRuntimeError( + message=( + f"Unknown {self._scope_name} {stage.value} implementation " + f"{requested_id!r}. Available implementations: {available}." + ), + help_url=( + "https://inference-models.roboflow.com/errors/models-runtime/" + "#modelruntimeerror" + ), + ) + if not implementation.is_compatible(context): + raise ModelRuntimeError( + message=( + f"{self._scope_name} {stage.value} implementation " + f"{requested_id!r} is not compatible with " + f"device={context.device!r}, " + f"device_kind={context.device_kind!r}, " + f"device_family={context.device_family!r}." + ), + help_url=( + "https://inference-models.roboflow.com/errors/models-runtime/" + "#modelruntimeerror" + ), + ) + + return implementation + + def implementations( + self, + stage: OptimizationStage, + ) -> Tuple[InferenceStage, ...]: + """Return registered implementations for one stage. + + Args: + stage: Stage category to inspect. + + Returns: + Implementations in registration order. + """ + registered = tuple(self._implementations.get(stage, {}).values()) + + return registered + + def _resolve_auto( + self, + *, + stage: OptimizationStage, + implementations: Iterable[InferenceStage], + context: ExecutionContext, + ) -> InferenceStage: + implementations = tuple(implementations) + for implementation in implementations: + metadata = implementation.metadata + if metadata.implementation_id == self._base_id: + continue + validated = any( + environment.matches(context) + for environment in metadata.validated_environments + ) + if validated and implementation.is_compatible(context): + return implementation + + base = next( + ( + implementation + for implementation in implementations + if implementation.metadata.implementation_id == self._base_id + ), + None, + ) + if base is None: + raise ModelRuntimeError( + message=( + f"{self._scope_name} {stage.value} registry has no " + f"{self._base_id!r} implementation." + ), + help_url=( + "https://inference-models.roboflow.com/errors/models-runtime/" + "#modelruntimeerror" + ), + ) + + return base diff --git a/inference_models/inference_models/models/optimization/torch_readiness.py b/inference_models/inference_models/models/optimization/torch_readiness.py new file mode 100644 index 0000000000..67d010bca6 --- /dev/null +++ b/inference_models/inference_models/models/optimization/torch_readiness.py @@ -0,0 +1,55 @@ +"""Weakly referenced state handoff for exact torch tensor instances.""" + +from __future__ import annotations + +import threading +import weakref +from typing import Dict, Generic, Optional, Tuple, TypeVar + +import torch + +StateT = TypeVar("StateT") + + +class TensorReadinessTracker(Generic[StateT]): + """Associate one-shot readiness state with exact tensor instances.""" + + def __init__(self) -> None: + self._lock = threading.Lock() + self._states: Dict[int, Tuple[weakref.ReferenceType[torch.Tensor], StateT]] = {} + + def record(self, tensor: torch.Tensor, *, state: StateT) -> None: + """Record state for a tensor without mutating the tensor. + + Args: + tensor: Tensor passed between asynchronous inference stages. + state: Model-specific readiness state associated with the tensor. + """ + key = id(tensor) + + def discard(reference: weakref.ReferenceType[torch.Tensor]) -> None: + with self._lock: + current = self._states.get(key) + if current is not None and current[0] is reference: + self._states.pop(key, None) + + reference = weakref.ref(tensor, discard) + with self._lock: + self._states[key] = (reference, state) + + def consume(self, tensor: torch.Tensor) -> Optional[StateT]: + """Consume state recorded for the exact tensor instance. + + Args: + tensor: Tensor entering the consuming stage. + + Returns: + Recorded state, or ``None`` when the tensor has no associated state. + """ + key = id(tensor) + with self._lock: + current = self._states.pop(key, None) + if current is None or current[0]() is not tensor: + return None + + return current[1] diff --git a/inference_models/inference_models/models/rfdetr/optimization/__init__.py b/inference_models/inference_models/models/rfdetr/optimization/__init__.py new file mode 100644 index 0000000000..21efdf23fb --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/optimization/__init__.py @@ -0,0 +1 @@ +"""Typed RF-DETR inference-path implementation selection.""" diff --git a/inference_models/inference_models/models/rfdetr/optimization/catalog.py b/inference_models/inference_models/models/rfdetr/optimization/catalog.py new file mode 100644 index 0000000000..0ee5110abd --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/optimization/catalog.py @@ -0,0 +1,64 @@ +"""RF-DETR implementation catalog and registry construction.""" + +from types import MappingProxyType +from typing import Mapping + +import torch + +from inference_models.models.optimization.contracts import OptimizationMetadata +from inference_models.models.optimization.registry import ImplementationRegistry +from inference_models.models.rfdetr.optimization.postprocessors import ( + BasePostprocessor, + TritonFusedPostprocessor, +) +from inference_models.models.rfdetr.optimization.preprocessors import ( + BasePreprocessor, + ThreadedExactPreprocessor, + TritonUniversalPreprocessor, +) + +RFDETR_PREPROCESSOR_IMPLEMENTATIONS: Mapping[str, OptimizationMetadata] = ( + MappingProxyType( + { + implementation.metadata.implementation_id: implementation.metadata + for implementation in ( + BasePreprocessor, + ThreadedExactPreprocessor, + TritonUniversalPreprocessor, + ) + } + ) +) + +RFDETR_POSTPROCESSOR_IMPLEMENTATIONS: Mapping[str, OptimizationMetadata] = ( + MappingProxyType( + { + implementation.metadata.implementation_id: implementation.metadata + for implementation in (BasePostprocessor, TritonFusedPostprocessor) + } + ) +) + + +def build_rfdetr_implementation_registry( + *, + device: torch.device, + preprocessor_max_workers: int, +) -> ImplementationRegistry: + """Build the complete RF-DETR stage implementation registry. + + Args: + device: CUDA target selected for the TensorRT model. + preprocessor_max_workers: Bounded threaded preprocessing worker limit. + + Returns: + Registry containing every available preprocessing and postprocessing choice. + """ + registry = ImplementationRegistry(scope_name="RF-DETR") + registry.register(BasePreprocessor(max_workers=preprocessor_max_workers)) + registry.register(ThreadedExactPreprocessor(max_workers=preprocessor_max_workers)) + registry.register(TritonUniversalPreprocessor(device=device)) + registry.register(BasePostprocessor()) + registry.register(TritonFusedPostprocessor(device=device)) + + return registry diff --git a/inference_models/inference_models/models/rfdetr/optimization/contracts.py b/inference_models/inference_models/models/rfdetr/optimization/contracts.py new file mode 100644 index 0000000000..bf64b702d5 --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/optimization/contracts.py @@ -0,0 +1,169 @@ +"""RF-DETR-specific requests, results, and stage protocols.""" + +from dataclasses import dataclass +from typing import List, Optional, Protocol, Union + +import numpy as np +import torch + +from inference_models import Detections, PreProcessingOverrides +from inference_models.entities import ColorFormat +from inference_models.models.common.roboflow.model_packages import ( + ImagePreProcessing, + NetworkInputDefinition, + PreProcessingMetadata, +) +from inference_models.models.optimization.contracts import ( + CompatibilityResult, + ExecutionContext, + InferenceStage, + OptimizationMetadata, +) +from inference_models.models.rfdetr.class_remapping import ClassesReMapping + +ImageInput = Union[np.ndarray, torch.Tensor] + + +@dataclass(frozen=True) +class PreprocessRequest: + """Inputs required by an RF-DETR preprocessing implementation.""" + + images: Union[ImageInput, List[ImageInput]] + input_color_format: Optional[ColorFormat] + image_pre_processing: ImagePreProcessing + network_input: NetworkInputDefinition + pre_processing_overrides: Optional[PreProcessingOverrides] + + +@dataclass(frozen=True) +class PreprocessResult: + """Typed preprocessing output and asynchronous readiness state.""" + + tensor: torch.Tensor + metadata: List[PreProcessingMetadata] + implementation_id: str + ready_event: Optional[torch.cuda.Event] = None + input_kind: str = "reference" + fallback_reason: Optional[str] = None + + +@dataclass(frozen=True) +class PostprocessRequest: + """Inputs required by an RF-DETR postprocessing implementation.""" + + bboxes: torch.Tensor + logits: torch.Tensor + pre_processing_meta: List[PreProcessingMetadata] + threshold: Union[float, torch.Tensor] + num_classes: int + classes_re_mapping: Optional[ClassesReMapping] + + +class Preprocessor(InferenceStage, Protocol): + """RF-DETR preprocessing stage interface.""" + + metadata: OptimizationMetadata + + def is_compatible(self, context: ExecutionContext) -> bool: + """Return whether the preprocessor supports a runtime context. + + Args: + context: Runtime target and request context. + + Returns: + Whether the preprocessor is compatible. + """ + + def check_model_compatibility( + self, + *, + image_pre_processing: ImagePreProcessing, + network_input: NetworkInputDefinition, + ) -> CompatibilityResult: + """Check compatibility with static model preprocessing configuration. + + Args: + image_pre_processing: Model-package image transformations. + network_input: Model-package network input definition. + + Returns: + Compatibility result with actionable reasons. + """ + + def check_request_compatibility( + self, + *, + request: PreprocessRequest, + context: ExecutionContext, + ) -> CompatibilityResult: + """Check compatibility with one concrete preprocessing request. + + Args: + request: Typed preprocessing request. + context: Runtime target and request context. + + Returns: + Compatibility result with actionable reasons. + """ + + def preprocess( + self, + request: PreprocessRequest, + context: ExecutionContext, + ) -> PreprocessResult: + """Preprocess one request on the context stream. + + Args: + request: Typed preprocessing request. + context: Runtime context containing the selected stream. + + Returns: + Typed preprocessing result. + """ + + +class Postprocessor(InferenceStage, Protocol): + """RF-DETR postprocessing stage interface.""" + + metadata: OptimizationMetadata + + def is_compatible(self, context: ExecutionContext) -> bool: + """Return whether the postprocessor supports a runtime context. + + Args: + context: Runtime target and request context. + + Returns: + Whether the postprocessor is compatible. + """ + + def check_request_compatibility( + self, + *, + request: PostprocessRequest, + context: ExecutionContext, + ) -> CompatibilityResult: + """Check compatibility with one concrete postprocessing request. + + Args: + request: Typed postprocessing request. + context: Runtime target and request context. + + Returns: + Compatibility result with actionable reasons. + """ + + def postprocess( + self, + request: PostprocessRequest, + context: ExecutionContext, + ) -> List[Detections]: + """Postprocess one request on the context stream. + + Args: + request: Typed postprocessing request. + context: Runtime context containing the selected stream. + + Returns: + Per-image detections. + """ diff --git a/inference_models/inference_models/models/rfdetr/optimization/execution_plan.py b/inference_models/inference_models/models/rfdetr/optimization/execution_plan.py new file mode 100644 index 0000000000..b6568c2c98 --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/optimization/execution_plan.py @@ -0,0 +1,84 @@ +"""Composed RF-DETR execution-plan selection.""" + +from __future__ import annotations + +import os +from dataclasses import dataclass +from typing import Optional + +from inference_models.errors import ModelRuntimeError +from inference_models.models.optimization.execution_plan import InferenceExecutionPlan +from inference_models.models.optimization.ids import BASE_IMPLEMENTATION_ID +from inference_models.models.rfdetr.optimization.ids import ( + RFDETR_POSTPROCESSOR_ENV_NAME, + RFDETR_POSTPROCESSOR_TRITON_FUSED_V1, + RFDETR_PREPROCESSOR_ENV_NAME, + RFDETR_PREPROCESSOR_TRITON_UNIVERSAL_V1, +) + + +@dataclass(frozen=True) +class RFDetrExecutionPlan(InferenceExecutionPlan): + """Independent implementation selections for the RF-DETR inference path.""" + + preprocessor_id: str = RFDETR_PREPROCESSOR_TRITON_UNIVERSAL_V1 + postprocessor_id: str = RFDETR_POSTPROCESSOR_TRITON_FUSED_V1 + + @classmethod + def resolve( + cls, + *, + execution_plan: Optional["RFDetrExecutionPlan"] = None, + ) -> "RFDetrExecutionPlan": + """Resolve a plan from an explicit plan or RF-DETR environment values. + + Args: + execution_plan: Explicit composed plan. When omitted, stage IDs are read + from the RF-DETR environment variables. + + Returns: + Immutable requested execution plan. + + Raises: + ModelRuntimeError: If the plan requests an unsupported stage category. + """ + if execution_plan is not None: + plan = execution_plan + else: + plan = cls( + preprocessor_id=os.getenv( + RFDETR_PREPROCESSOR_ENV_NAME, + RFDETR_PREPROCESSOR_TRITON_UNIVERSAL_V1, + ), + postprocessor_id=os.getenv( + RFDETR_POSTPROCESSOR_ENV_NAME, + RFDETR_POSTPROCESSOR_TRITON_FUSED_V1, + ), + ) + + plan._validate_supported_stage_categories() + + return plan + + def _validate_supported_stage_categories(self) -> None: + unsupported = { + "buffer_strategy_id": self.buffer_strategy_id, + "scheduler_id": self.scheduler_id, + "engine_plugin_id": self.engine_plugin_id, + } + selected = [ + f"{name}={value!r}" + for name, value in unsupported.items() + if value != BASE_IMPLEMENTATION_ID + ] + if selected: + raise ModelRuntimeError( + message=( + "RF-DETR does not yet provide these execution-plan stages: " + + ", ".join(selected) + ), + help_url=( + "https://inference-models.roboflow.com/errors/models-runtime/" + "#modelruntimeerror" + ), + ) diff --git a/inference_models/inference_models/models/rfdetr/optimization/ids.py b/inference_models/inference_models/models/rfdetr/optimization/ids.py new file mode 100644 index 0000000000..7df25066f2 --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/optimization/ids.py @@ -0,0 +1,15 @@ +"""Stable IDs for RF-DETR inference-path implementations.""" + +from inference_models.models.optimization.ids import BASE_IMPLEMENTATION_ID + +RFDETR_PREPROCESSOR_BASE = BASE_IMPLEMENTATION_ID +RFDETR_PREPROCESSOR_THREADED_EXACT_V1 = "threaded-exact-v1" +RFDETR_PREPROCESSOR_TRITON_UNIVERSAL_V1 = "triton-universal-v1" +RFDETR_PREPROCESSOR_ENV_NAME = "INFERENCE_MODELS_RFDETR_PREPROCESSOR" +RFDETR_PREPROCESSOR_MAX_WORKERS_ENV_NAME = ( + "INFERENCE_MODELS_RFDETR_PREPROCESSOR_MAX_WORKERS" +) +RFDETR_PREPROCESSOR_DEFAULT_MAX_WORKERS = 4 +RFDETR_POSTPROCESSOR_BASE = BASE_IMPLEMENTATION_ID +RFDETR_POSTPROCESSOR_TRITON_FUSED_V1 = "triton-fused-v1" +RFDETR_POSTPROCESSOR_ENV_NAME = "INFERENCE_MODELS_RFDETR_POSTPROCESSOR" diff --git a/inference_models/inference_models/models/rfdetr/optimization/postprocessors/__init__.py b/inference_models/inference_models/models/rfdetr/optimization/postprocessors/__init__.py new file mode 100644 index 0000000000..b87d907a19 --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/optimization/postprocessors/__init__.py @@ -0,0 +1,10 @@ +"""RF-DETR postprocessing implementation choices.""" + +from inference_models.models.rfdetr.optimization.postprocessors.base import ( + BasePostprocessor, +) +from inference_models.models.rfdetr.optimization.postprocessors.triton_fused import ( + TritonFusedPostprocessor, +) + +__all__ = ["BasePostprocessor", "TritonFusedPostprocessor"] diff --git a/inference_models/inference_models/models/rfdetr/optimization/postprocessors/base.py b/inference_models/inference_models/models/rfdetr/optimization/postprocessors/base.py new file mode 100644 index 0000000000..3737f12e43 --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/optimization/postprocessors/base.py @@ -0,0 +1,105 @@ +"""Preserved base RF-DETR object-detection postprocessing choice.""" + +from typing import List + +from inference_models import Detections +from inference_models.models.optimization.contracts import ( + CompatibilityResult, + DeviceCompatibility, + ExecutionContext, + InputCompatibility, + OptimizationMetadata, + OptimizationStage, + immutable_mapping, + metadata_supports_context, +) +from inference_models.models.rfdetr.common import post_process_object_detection_results +from inference_models.models.rfdetr.optimization.contracts import PostprocessRequest +from inference_models.models.rfdetr.optimization.ids import RFDETR_POSTPROCESSOR_BASE + + +class BasePostprocessor: + """Preserve the original RF-DETR PyTorch postprocessing path.""" + + metadata = OptimizationMetadata( + implementation_id=RFDETR_POSTPROCESSOR_BASE, + stage=OptimizationStage.POSTPROCESS, + version="1", + target=DeviceCompatibility(device_kind="gpu"), + inputs=InputCompatibility( + scenarios=("*",), + axis_constraints=immutable_mapping({"batch": ">=1", "queries": ">=1"}), + dtypes=("float32",), + layouts=("contiguous BQ4", "contiguous BQC"), + ), + dependencies=("torch",), + fallback_id=RFDETR_POSTPROCESSOR_BASE, + changes_numerics=False, + supports_concurrency=True, + supports_cuda_graphs=False, + output_contract=immutable_mapping( + { + "type": "list[Detections]", + "ownership": "per-call tensors owned by returned Detections", + } + ), + numerical_behavior="reference RF-DETR PyTorch pipeline", + stream_behavior="runs on the caller postprocessing stream", + ) + + def is_compatible(self, context: ExecutionContext) -> bool: + """Return whether the base path supports the runtime context. + + Args: + context: Runtime target and request context. + + Returns: + Whether the target is compatible. + """ + return metadata_supports_context(self.metadata, context) + + def check_request_compatibility( + self, + *, + request: PostprocessRequest, + context: ExecutionContext, + ) -> CompatibilityResult: + """Accept requests handled by the preserved postprocessing path. + + Args: + request: Typed postprocessing request. + context: Runtime target and request context. + + Returns: + Compatible result; detailed validation remains in the base path. + """ + del request, context + result = CompatibilityResult.compatible() + + return result + + def postprocess( + self, + request: PostprocessRequest, + context: ExecutionContext, + ) -> List[Detections]: + """Run preserved RF-DETR postprocessing. + + Args: + request: Typed postprocessing request. + context: Runtime context containing the postprocessing stream. + + Returns: + Per-image detections. + """ + results = post_process_object_detection_results( + bboxes=request.bboxes, + logits=request.logits, + pre_processing_meta=request.pre_processing_meta, + threshold=request.threshold, + num_classes=request.num_classes, + classes_re_mapping=request.classes_re_mapping, + device=request.bboxes.device, + ) + + return results diff --git a/inference_models/inference_models/models/rfdetr/optimization/postprocessors/triton_fused.py b/inference_models/inference_models/models/rfdetr/optimization/postprocessors/triton_fused.py new file mode 100644 index 0000000000..7ab4c8cde1 --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/optimization/postprocessors/triton_fused.py @@ -0,0 +1,146 @@ +"""Fused Triton RF-DETR object-detection postprocessing choice.""" + +from typing import List + +import torch + +from inference_models import Detections +from inference_models.errors import ModelRuntimeError +from inference_models.models.optimization.contracts import ( + CompatibilityResult, + DeviceCompatibility, + ExecutionContext, + InputCompatibility, + OptimizationMetadata, + OptimizationStage, + immutable_mapping, + metadata_supports_context, +) +from inference_models.models.rfdetr.optimization.contracts import PostprocessRequest +from inference_models.models.rfdetr.optimization.ids import ( + RFDETR_POSTPROCESSOR_BASE, + RFDETR_POSTPROCESSOR_TRITON_FUSED_V1, +) +from inference_models.models.rfdetr.triton_object_detection_postprocess import ( + FusedObjectDetectionPostprocessor, +) + + +class TritonFusedPostprocessor: + """Run the explicit fused Triton object-detection postprocessor.""" + + metadata = OptimizationMetadata( + implementation_id=RFDETR_POSTPROCESSOR_TRITON_FUSED_V1, + stage=OptimizationStage.POSTPROCESS, + version="1", + target=DeviceCompatibility( + device_kind="gpu", + device_families=("nvidia_jetson", "nvidia_discrete_gpu"), + ), + inputs=InputCompatibility( + scenarios=("*",), + axis_constraints=immutable_mapping({"batch": ">=1", "queries": "1..1024"}), + dtypes=("float32",), + layouts=("contiguous BQ4", "contiguous BQC"), + ), + dependencies=("torch", "triton"), + fallback_id=RFDETR_POSTPROCESSOR_BASE, + changes_numerics=False, + supports_concurrency=True, + supports_cuda_graphs=False, + output_contract=immutable_mapping( + { + "type": "list[Detections]", + "boxes_dtype": "int32", + "confidence_dtype": "float32", + "class_dtype": "int32", + "ownership": "per-call tensors owned by returned Detections", + } + ), + numerical_behavior=( + "same sigmoid, sorted flat top-k, strict threshold, metadata rescaling, " + "clipping, and round-to-int semantics as base" + ), + stream_behavior=( + "runs on the caller stream with one batched count synchronization" + ), + ) + + def __init__(self, *, device: torch.device) -> None: + self._runtime = FusedObjectDetectionPostprocessor(device=device) + + def is_compatible(self, context: ExecutionContext) -> bool: + """Return whether the Triton path supports the runtime context. + + Args: + context: Runtime target and request context. + + Returns: + Whether the target is compatible. + """ + return metadata_supports_context(self.metadata, context) + + def check_request_compatibility( + self, + *, + request: PostprocessRequest, + context: ExecutionContext, + ) -> CompatibilityResult: + """Check request constraints supported by fused Triton postprocessing. + + Args: + request: Typed postprocessing request. + context: Runtime target and request context. + + Returns: + Compatibility result with actionable reasons. + """ + del context + result = self._runtime.check_request_compatibility( + bboxes=request.bboxes, + logits=request.logits, + pre_processing_meta=request.pre_processing_meta, + threshold=request.threshold, + num_classes=request.num_classes, + classes_re_mapping=request.classes_re_mapping, + ) + + return result + + def postprocess( + self, + request: PostprocessRequest, + context: ExecutionContext, + ) -> List[Detections]: + """Run fused Triton postprocessing. + + Args: + request: Typed postprocessing request. + context: Runtime context containing the postprocessing stream. + + Returns: + Per-image detections. + + Raises: + ModelRuntimeError: If the execution context has no CUDA stream. + """ + stream = context.current_stream + if stream is None: + raise ModelRuntimeError( + message="triton-fused-v1 requires a postprocessing CUDA stream.", + help_url=( + "https://inference-models.roboflow.com/errors/models-runtime/" + "#modelruntimeerror" + ), + ) + results = self._runtime.postprocess( + bboxes=request.bboxes, + logits=request.logits, + pre_processing_meta=request.pre_processing_meta, + threshold=request.threshold, + num_classes=request.num_classes, + classes_re_mapping=request.classes_re_mapping, + stream=stream, + ) + + return results diff --git a/inference_models/inference_models/models/rfdetr/optimization/preprocessors/__init__.py b/inference_models/inference_models/models/rfdetr/optimization/preprocessors/__init__.py new file mode 100644 index 0000000000..d5c4b2a446 --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/optimization/preprocessors/__init__.py @@ -0,0 +1,17 @@ +"""RF-DETR preprocessing implementation choices.""" + +from inference_models.models.rfdetr.optimization.preprocessors.base import ( + BasePreprocessor, +) +from inference_models.models.rfdetr.optimization.preprocessors.threaded_exact import ( + ThreadedExactPreprocessor, +) +from inference_models.models.rfdetr.optimization.preprocessors.triton_universal import ( + TritonUniversalPreprocessor, +) + +__all__ = [ + "BasePreprocessor", + "ThreadedExactPreprocessor", + "TritonUniversalPreprocessor", +] diff --git a/inference_models/inference_models/models/rfdetr/optimization/preprocessors/base.py b/inference_models/inference_models/models/rfdetr/optimization/preprocessors/base.py new file mode 100644 index 0000000000..7a6097e444 --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/optimization/preprocessors/base.py @@ -0,0 +1,136 @@ +"""Preserved base RF-DETR preprocessing choice.""" + +from inference_models.models.common.roboflow.model_packages import ( + ImagePreProcessing, + NetworkInputDefinition, +) +from inference_models.models.optimization.contracts import ( + CompatibilityResult, + DeviceCompatibility, + ExecutionContext, + InputCompatibility, + OptimizationMetadata, + OptimizationStage, + immutable_mapping, + metadata_supports_context, +) +from inference_models.models.rfdetr.optimization.contracts import ( + PreprocessRequest, + PreprocessResult, +) +from inference_models.models.rfdetr.optimization.ids import RFDETR_PREPROCESSOR_BASE +from inference_models.models.rfdetr.optimization.preprocessors.common import ( + run_reference_preprocessor, +) +from inference_models.models.rfdetr.optimization.preprocessors.compatibility import ( + check_base_request_compatibility, +) + + +class BasePreprocessor: + """Preserve the original RF-DETR PIL/tensor preprocessing path.""" + + metadata = OptimizationMetadata( + implementation_id=RFDETR_PREPROCESSOR_BASE, + stage=OptimizationStage.PREPROCESS, + version="1", + target=DeviceCompatibility(device_kind="gpu"), + inputs=InputCompatibility( + scenarios=("*",), + axis_constraints=immutable_mapping({"batch": ">=1"}), + dtypes=("uint8", "floating"), + layouts=("HWC", "NHWC", "CHW", "NCHW"), + ), + dependencies=("Pillow", "torch", "torchvision"), + fallback_id=RFDETR_PREPROCESSOR_BASE, + changes_numerics=False, + supports_concurrency=True, + supports_cuda_graphs=False, + output_contract=immutable_mapping( + { + "device": "selected CUDA device", + "dtype": "float32", + "layout": "contiguous NCHW", + "ownership": "new tensor owned by caller", + } + ), + numerical_behavior="reference RF-DETR PIL/torch pipeline", + stream_behavior="submits H2D work to the caller preprocessing stream", + ) + + def __init__(self, *, max_workers: int) -> None: + self._max_workers = max_workers + + def is_compatible(self, context: ExecutionContext) -> bool: + """Return whether the base path supports the runtime context. + + Args: + context: Runtime target and request context. + + Returns: + Whether the target is compatible. + """ + return metadata_supports_context(self.metadata, context) + + def check_model_compatibility( + self, + *, + image_pre_processing: ImagePreProcessing, + network_input: NetworkInputDefinition, + ) -> CompatibilityResult: + """Accept model configurations handled by the reference path. + + Args: + image_pre_processing: Model-package image transformations. + network_input: Model-package network input definition. + + Returns: + Compatible result; detailed validation remains in the reference path. + """ + del image_pre_processing, network_input + result = CompatibilityResult.compatible() + + return result + + def check_request_compatibility( + self, + *, + request: PreprocessRequest, + context: ExecutionContext, + ) -> CompatibilityResult: + """Check broad request compatibility for the reference path. + + Args: + request: Typed preprocessing request. + context: Runtime target and request context. + + Returns: + Compatibility result with actionable reasons. + """ + del context + result = check_base_request_compatibility(request=request) + + return result + + def preprocess( + self, + request: PreprocessRequest, + context: ExecutionContext, + ) -> PreprocessResult: + """Run the preserved base preprocessor. + + Args: + request: Typed preprocessing request. + context: Runtime context containing the preprocessing stream. + + Returns: + Typed preprocessing result. + """ + result = run_reference_preprocessor( + request, + context, + implementation_id=self.metadata.implementation_id, + max_workers=self._max_workers, + ) + + return result diff --git a/inference_models/inference_models/models/rfdetr/optimization/preprocessors/common.py b/inference_models/inference_models/models/rfdetr/optimization/preprocessors/common.py new file mode 100644 index 0000000000..aca77511fc --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/optimization/preprocessors/common.py @@ -0,0 +1,64 @@ +"""Shared adapter for reference RF-DETR preprocessing implementations.""" + +import torch + +from inference_models.errors import ModelRuntimeError +from inference_models.models.optimization.contracts import ExecutionContext +from inference_models.models.rfdetr.optimization.contracts import ( + PreprocessRequest, + PreprocessResult, +) +from inference_models.models.rfdetr.pre_processing import pre_process_network_input + + +def run_reference_preprocessor( + request: PreprocessRequest, + context: ExecutionContext, + *, + implementation_id: str, + max_workers: int, +) -> PreprocessResult: + """Run the existing RF-DETR preprocessor on the context stream. + + Args: + request: Typed preprocessing request. + context: Runtime context containing the CUDA stream. + implementation_id: Base or threaded implementation ID. + max_workers: Bounded threaded worker limit. + + Returns: + Typed preprocessing result. + + Raises: + ModelRuntimeError: If the execution context has no CUDA stream. + """ + stream = context.current_stream + if stream is None: + raise ModelRuntimeError( + message=f"{implementation_id!r} requires a preprocessing CUDA stream.", + help_url=( + "https://inference-models.roboflow.com/errors/models-runtime/" + "#modelruntimeerror" + ), + ) + + with torch.cuda.stream(stream): + tensor, metadata = pre_process_network_input( + images=request.images, + image_pre_processing=request.image_pre_processing, + network_input=request.network_input, + target_device=torch.device(context.device), + input_color_format=request.input_color_format, + pre_processing_overrides=request.pre_processing_overrides, + preprocessor_implementation_id=implementation_id, + preprocessor_max_workers=max_workers, + ) + + result = PreprocessResult( + tensor=tensor, + metadata=metadata, + input_kind="reference", + implementation_id=implementation_id, + ) + + return result diff --git a/inference_models/inference_models/models/rfdetr/optimization/preprocessors/compatibility.py b/inference_models/inference_models/models/rfdetr/optimization/preprocessors/compatibility.py new file mode 100644 index 0000000000..ba047f91f7 --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/optimization/preprocessors/compatibility.py @@ -0,0 +1,100 @@ +"""Compatibility checks shared by RF-DETR reference preprocessors.""" + +from typing import Any, List + +import numpy as np +import torch + +from inference_models.models.optimization.contracts import CompatibilityResult +from inference_models.models.rfdetr.optimization.contracts import PreprocessRequest + + +def _request_items(images) -> List[Any]: + """Expand a supported image or image batch into individual items. + + Args: + images: Single image or batch supplied to RF-DETR preprocessing. + + Returns: + Individual batch items. Unsupported values are retained for validation. + """ + if isinstance(images, list): + items = list(images) + elif isinstance(images, torch.Tensor) and images.ndim == 4: + items = list(images.unbind(0)) + elif isinstance(images, np.ndarray) and images.ndim == 4: + items = list(images) + else: + items = [images] + + return items + + +def check_base_request_compatibility( + request: PreprocessRequest, +) -> CompatibilityResult: + """Check the broad request constraints of the reference preprocessor. + + Args: + request: Typed preprocessing request. + + Returns: + Compatibility result. Detailed shape validation remains in the base path. + """ + items = _request_items(request.images) + reasons = [] + if not items: + reasons.append("empty image batch") + unsupported_types = sorted( + { + type(item).__name__ + for item in items + if not isinstance(item, (np.ndarray, torch.Tensor)) + } + ) + if unsupported_types: + reasons.append(f"unsupported input types: {unsupported_types}") + if reasons: + result = CompatibilityResult.incompatible(*reasons) + else: + result = CompatibilityResult.compatible() + + return result + + +def check_threaded_request_compatibility( + request: PreprocessRequest, +) -> CompatibilityResult: + """Check the exact NumPy batch constraints of threaded preprocessing. + + Args: + request: Typed preprocessing request. + + Returns: + Compatibility result with all discovered limitations. + """ + base_compatibility = check_base_request_compatibility(request=request) + if not base_compatibility.supported: + return base_compatibility + + items = _request_items(request.images) + unsupported = [ + type(item).__name__ for item in items if not isinstance(item, np.ndarray) + ] + invalid = [ + (str(item.dtype), tuple(item.shape)) + for item in items + if isinstance(item, np.ndarray) + and (item.dtype != np.uint8 or item.ndim != 3 or item.shape[-1] != 3) + ] + reasons = [] + if unsupported: + reasons.append(f"threaded preprocessing requires NumPy inputs: {unsupported}") + if invalid: + reasons.append(f"threaded preprocessing requires uint8 HWC images: {invalid}") + if reasons: + result = CompatibilityResult.incompatible(*reasons) + else: + result = CompatibilityResult.compatible() + + return result diff --git a/inference_models/inference_models/models/rfdetr/optimization/preprocessors/threaded_exact.py b/inference_models/inference_models/models/rfdetr/optimization/preprocessors/threaded_exact.py new file mode 100644 index 0000000000..a748df984d --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/optimization/preprocessors/threaded_exact.py @@ -0,0 +1,147 @@ +"""Threaded exact RF-DETR preprocessing choice.""" + +from inference_models.models.common.roboflow.model_packages import ( + ImagePreProcessing, + NetworkInputDefinition, +) +from inference_models.models.optimization.contracts import ( + CompatibilityResult, + DeviceCompatibility, + ExecutionContext, + InputCompatibility, + OptimizationMetadata, + OptimizationStage, + immutable_mapping, + metadata_supports_context, +) +from inference_models.models.rfdetr.optimization.contracts import ( + PreprocessRequest, + PreprocessResult, +) +from inference_models.models.rfdetr.optimization.ids import ( + RFDETR_PREPROCESSOR_BASE, + RFDETR_PREPROCESSOR_THREADED_EXACT_V1, +) +from inference_models.models.rfdetr.optimization.preprocessors.common import ( + run_reference_preprocessor, +) +from inference_models.models.rfdetr.optimization.preprocessors.compatibility import ( + check_threaded_request_compatibility, +) + + +class ThreadedExactPreprocessor: + """Parallelize exact per-image CPU preprocessing for NumPy batches.""" + + metadata = OptimizationMetadata( + implementation_id=RFDETR_PREPROCESSOR_THREADED_EXACT_V1, + stage=OptimizationStage.PREPROCESS, + version="1", + target=DeviceCompatibility( + device_kind="gpu", + device_families=("nvidia_jetson", "nvidia_discrete_gpu"), + ), + inputs=InputCompatibility( + scenarios=("*",), + axis_constraints=immutable_mapping( + {"batch": ">=1", "channels": 3, "source_dimensions": "per image"} + ), + dtypes=("uint8",), + layouts=("HWC", "NHWC"), + ), + dependencies=("Pillow", "torch", "torchvision"), + fallback_id=RFDETR_PREPROCESSOR_BASE, + changes_numerics=False, + supports_concurrency=True, + supports_cuda_graphs=False, + output_contract=immutable_mapping( + { + "device": "selected CUDA device", + "dtype": "float32", + "layout": "contiguous NCHW", + "ownership": "new tensor owned by caller", + } + ), + numerical_behavior="byte-identical per-image reference pipeline", + stream_behavior=( + "CPU work completes before ordered H2D copies are submitted to the " + "caller preprocessing stream" + ), + ) + + def __init__(self, *, max_workers: int) -> None: + self._max_workers = max_workers + + def is_compatible(self, context: ExecutionContext) -> bool: + """Return whether the threaded path supports the runtime context. + + Args: + context: Runtime target and request context. + + Returns: + Whether the target is compatible. + """ + return metadata_supports_context(self.metadata, context) + + def check_model_compatibility( + self, + *, + image_pre_processing: ImagePreProcessing, + network_input: NetworkInputDefinition, + ) -> CompatibilityResult: + """Accept model transformations delegated to the reference path. + + Args: + image_pre_processing: Model-package image transformations. + network_input: Model-package network input definition. + + Returns: + Compatible result. + """ + del image_pre_processing, network_input + result = CompatibilityResult.compatible() + + return result + + def check_request_compatibility( + self, + *, + request: PreprocessRequest, + context: ExecutionContext, + ) -> CompatibilityResult: + """Check the threaded implementation's NumPy request constraints. + + Args: + request: Typed preprocessing request. + context: Runtime target and request context. + + Returns: + Compatibility result with actionable reasons. + """ + del context + result = check_threaded_request_compatibility(request=request) + + return result + + def preprocess( + self, + request: PreprocessRequest, + context: ExecutionContext, + ) -> PreprocessResult: + """Run threaded exact preprocessing. + + Args: + request: Typed preprocessing request. + context: Runtime context containing the preprocessing stream. + + Returns: + Typed preprocessing result. + """ + result = run_reference_preprocessor( + request, + context, + implementation_id=self.metadata.implementation_id, + max_workers=self._max_workers, + ) + + return result diff --git a/inference_models/inference_models/models/rfdetr/optimization/preprocessors/triton_universal.py b/inference_models/inference_models/models/rfdetr/optimization/preprocessors/triton_universal.py new file mode 100644 index 0000000000..fd5e729805 --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/optimization/preprocessors/triton_universal.py @@ -0,0 +1,178 @@ +"""Universal Triton RF-DETR preprocessing choice.""" + +import torch + +from inference_models.models.common.roboflow.model_packages import ( + ImagePreProcessing, + NetworkInputDefinition, +) +from inference_models.models.optimization.contracts import ( + CompatibilityResult, + DeviceCompatibility, + ExecutionContext, + InputCompatibility, + OptimizationMetadata, + OptimizationStage, + immutable_mapping, + metadata_supports_context, +) +from inference_models.models.rfdetr.optimization.contracts import ( + PreprocessRequest, + PreprocessResult, +) +from inference_models.models.rfdetr.optimization.ids import ( + RFDETR_PREPROCESSOR_BASE, + RFDETR_PREPROCESSOR_TRITON_UNIVERSAL_V1, +) +from inference_models.models.rfdetr.triton_universal_preprocess_runtime import ( + UniversalFastPreprocessRuntime, +) + + +class TritonUniversalPreprocessor: + """Run the explicit universal CUDA/Triton preprocessing path.""" + + metadata = OptimizationMetadata( + implementation_id=RFDETR_PREPROCESSOR_TRITON_UNIVERSAL_V1, + stage=OptimizationStage.PREPROCESS, + version="1", + target=DeviceCompatibility( + device_kind="gpu", + device_families=("nvidia_jetson", "nvidia_discrete_gpu"), + ), + inputs=InputCompatibility( + scenarios=("*",), + axis_constraints=immutable_mapping( + { + "batch": ">=1", + "channels": 3, + "source_dimensions": "homogeneous", + "resize_mode": "stretch", + } + ), + dtypes=("uint8", "floating"), + layouts=("HWC", "NHWC", "CHW", "NCHW"), + ), + dependencies=("torch", "torchvision", "triton"), + fallback_id=RFDETR_PREPROCESSOR_BASE, + changes_numerics=False, + supports_concurrency=True, + supports_cuda_graphs=False, + output_contract=immutable_mapping( + { + "device": "selected CUDA device", + "dtype": "float32", + "layout": "contiguous NCHW", + "ownership": "per-call tensor from PyTorch CUDA allocator", + } + ), + numerical_behavior=( + "PIL byte-exact fixed-point resize for uint8; floating tensors preserve " + "RF-DETR tensor-input CUDA resize semantics" + ), + stream_behavior=( + "submits preprocessing to the caller stream and returns a completion event" + ), + ) + + def __init__(self, *, device: torch.device) -> None: + self._runtime = UniversalFastPreprocessRuntime(device=device) + + def is_compatible(self, context: ExecutionContext) -> bool: + """Return whether the Triton path supports the runtime context. + + Args: + context: Runtime target and request context. + + Returns: + Whether the target is compatible. + """ + return metadata_supports_context(self.metadata, context) + + def check_model_compatibility( + self, + *, + image_pre_processing: ImagePreProcessing, + network_input: NetworkInputDefinition, + ) -> CompatibilityResult: + """Check static model configuration supported by Triton preprocessing. + + Args: + image_pre_processing: Model-package image transformations. + network_input: Model-package network input definition. + + Returns: + Compatibility result with actionable reasons. + """ + result = self._runtime.check_model_compatibility( + image_pre_processing=image_pre_processing, + network_input=network_input, + ) + + return result + + def check_request_compatibility( + self, + *, + request: PreprocessRequest, + context: ExecutionContext, + ) -> CompatibilityResult: + """Check request-specific constraints supported by Triton preprocessing. + + Args: + request: Typed preprocessing request. + context: Runtime target and request context. + + Returns: + Compatibility result with actionable reasons. + """ + del context + result = self._runtime.check_request_compatibility( + images=request.images, + pre_processing_overrides=request.pre_processing_overrides, + ) + + return result + + def preprocess( + self, + request: PreprocessRequest, + context: ExecutionContext, + ) -> PreprocessResult: + """Run universal Triton preprocessing. + + Args: + request: Typed preprocessing request. + context: Runtime context containing the preprocessing stream. + + Returns: + Typed preprocessing result and completion event. + """ + stream = context.current_stream + if stream is None: + from inference_models.errors import ModelRuntimeError + + raise ModelRuntimeError( + message="triton-universal-v1 requires a preprocessing CUDA stream.", + help_url=( + "https://inference-models.roboflow.com/errors/models-runtime/" + "#modelruntimeerror" + ), + ) + runtime_result = self._runtime.preprocess( + images=request.images, + input_color_format=request.input_color_format, + image_pre_processing=request.image_pre_processing, + network_input=request.network_input, + pre_processing_overrides=request.pre_processing_overrides, + stream=stream, + ) + result = PreprocessResult( + tensor=runtime_result.tensor, + metadata=runtime_result.metadata, + ready_event=runtime_result.ready_event, + input_kind=runtime_result.input_kind, + implementation_id=self.metadata.implementation_id, + ) + + return result diff --git a/inference_models/inference_models/models/rfdetr/optimization/readiness.py b/inference_models/inference_models/models/rfdetr/optimization/readiness.py new file mode 100644 index 0000000000..91cf43b910 --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/optimization/readiness.py @@ -0,0 +1,48 @@ +"""RF-DETR readiness state carried by the shared tensor-state tracker.""" + +from dataclasses import dataclass +from typing import Optional + +import torch + +from inference_models.models.optimization.torch_readiness import TensorReadinessTracker + + +@dataclass(frozen=True) +class PreprocessReadiness: + """Readiness state recorded for one preprocessed tensor.""" + + ready_event: Optional[torch.cuda.Event] + input_kind: str + implementation_id: str + fallback_reason: Optional[str] = None + + +class PreprocessReadinessTracker(TensorReadinessTracker[PreprocessReadiness]): + """Adapt generic exact-tensor state tracking to RF-DETR preprocessing.""" + + def record( + self, + tensor: torch.Tensor, + *, + ready_event: Optional[torch.cuda.Event], + input_kind: str, + implementation_id: str, + fallback_reason: Optional[str] = None, + ) -> None: + """Record readiness for a tensor returned by preprocessing. + + Args: + tensor: Tensor passed to the protected forward stage. + ready_event: Optional event the inference stream must await. + input_kind: Canonical input-path description. + implementation_id: Preprocessor that produced the tensor. + fallback_reason: Optional reason the requested implementation was skipped. + """ + state = PreprocessReadiness( + ready_event=ready_event, + input_kind=input_kind, + implementation_id=implementation_id, + fallback_reason=fallback_reason, + ) + super().record(tensor, state=state) diff --git a/inference_models/inference_models/models/rfdetr/optimization/selection.py b/inference_models/inference_models/models/rfdetr/optimization/selection.py new file mode 100644 index 0000000000..143d5e00a6 --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/optimization/selection.py @@ -0,0 +1,301 @@ +"""RF-DETR stage selection with declared compatibility fallback.""" + +from dataclasses import dataclass +from typing import Callable, Dict, Generic, Optional, TypeVar, cast + +from inference_models.errors import ModelRuntimeError +from inference_models.models.common.roboflow.model_packages import ( + ImagePreProcessing, + NetworkInputDefinition, +) +from inference_models.models.optimization.contracts import ( + CompatibilityResult, + ExecutionContext, + InferenceStage, + OptimizationStage, +) +from inference_models.models.optimization.registry import ImplementationRegistry +from inference_models.models.rfdetr.optimization.contracts import ( + Postprocessor, + PostprocessRequest, + Preprocessor, + PreprocessRequest, +) + +StageT = TypeVar("StageT", bound=InferenceStage) + + +@dataclass(frozen=True) +class ImplementationSelection(Generic[StageT]): + """Requested and effective RF-DETR stage selection.""" + + implementation: StageT + requested_id: str + fallback_reason: Optional[str] = None + + @property + def effective_id(self) -> str: + """Return the implementation ID that will execute. + + Returns: + Effective stage implementation ID. + """ + return self.implementation.metadata.implementation_id + + @property + def used_fallback(self) -> bool: + """Return whether selection followed the declared fallback. + + Returns: + Whether compatibility resolution followed a declared fallback. + """ + return self.fallback_reason is not None + + def to_dict(self) -> Dict[str, Optional[str]]: + """Serialize requested and effective selection metadata. + + Returns: + JSON-compatible selection metadata. + """ + serialized = { + "requested_id": self.requested_id, + "effective_id": self.effective_id, + "fallback_reason": self.fallback_reason, + } + + return serialized + + +def resolve_preprocessor_for_model( + *, + registry: ImplementationRegistry, + requested_id: str, + context: ExecutionContext, + image_pre_processing: ImagePreProcessing, + network_input: NetworkInputDefinition, + allow_fallback: bool, +) -> ImplementationSelection[Preprocessor]: + """Resolve preprocessing against static model-package configuration. + + Args: + registry: RF-DETR implementation registry. + requested_id: Requested preprocessing implementation ID. + context: Runtime target context. + image_pre_processing: Model-package image transformations. + network_input: Model-package network input definition. + allow_fallback: Whether declared compatibility fallback may be used. + + Returns: + Effective implementation and optional fallback reason. + + Raises: + ModelRuntimeError: If the requested implementation is incompatible and no + permitted compatible fallback exists. + """ + implementation = cast( + Preprocessor, + registry.resolve( + stage=OptimizationStage.PREPROCESS, + requested_id=requested_id, + context=context, + ), + ) + + def check(candidate: Preprocessor) -> CompatibilityResult: + result = candidate.check_model_compatibility( + image_pre_processing=image_pre_processing, + network_input=network_input, + ) + + return result + + selection = _apply_declared_fallback( + registry=registry, + stage=OptimizationStage.PREPROCESS, + implementation=implementation, + requested_id=requested_id, + context=context, + check_compatibility=check, + allow_fallback=allow_fallback, + ) + + return selection + + +def resolve_preprocessor_for_request( + *, + registry: ImplementationRegistry, + implementation: Preprocessor, + request: PreprocessRequest, + context: ExecutionContext, + allow_fallback: bool, +) -> ImplementationSelection[Preprocessor]: + """Resolve preprocessing against one concrete inference request. + + Args: + registry: RF-DETR implementation registry. + implementation: Model-level selected preprocessor. + request: Typed preprocessing request. + context: Runtime target and request context. + allow_fallback: Whether declared compatibility fallback may be used. + + Returns: + Effective request implementation and optional fallback reason. + + Raises: + ModelRuntimeError: If the selected implementation is incompatible and no + permitted compatible fallback exists. + """ + requested_id = implementation.metadata.implementation_id + + def check(candidate: Preprocessor) -> CompatibilityResult: + result = candidate.check_request_compatibility( + request=request, + context=context, + ) + + return result + + selection = _apply_declared_fallback( + registry=registry, + stage=OptimizationStage.PREPROCESS, + implementation=implementation, + requested_id=requested_id, + context=context, + check_compatibility=check, + allow_fallback=allow_fallback, + ) + + return selection + + +def resolve_postprocessor_for_request( + *, + registry: ImplementationRegistry, + implementation: Postprocessor, + request: PostprocessRequest, + context: ExecutionContext, + allow_fallback: bool, +) -> ImplementationSelection[Postprocessor]: + """Resolve postprocessing against one concrete inference request. + + Args: + registry: RF-DETR implementation registry. + implementation: Model-level selected postprocessor. + request: Typed postprocessing request. + context: Runtime target and request context. + allow_fallback: Whether declared compatibility fallback may be used. + + Returns: + Effective request implementation and optional fallback reason. + + Raises: + ModelRuntimeError: If the selected implementation is incompatible and no + permitted compatible fallback exists. + """ + requested_id = implementation.metadata.implementation_id + + def check(candidate: Postprocessor) -> CompatibilityResult: + result = candidate.check_request_compatibility( + request=request, + context=context, + ) + + return result + + selection = _apply_declared_fallback( + registry=registry, + stage=OptimizationStage.POSTPROCESS, + implementation=implementation, + requested_id=requested_id, + context=context, + check_compatibility=check, + allow_fallback=allow_fallback, + ) + + return selection + + +def _apply_declared_fallback( + *, + registry: ImplementationRegistry, + stage: OptimizationStage, + implementation: StageT, + requested_id: str, + context: ExecutionContext, + check_compatibility: Callable[[StageT], CompatibilityResult], + allow_fallback: bool, +) -> ImplementationSelection[StageT]: + compatibility = check_compatibility(implementation) + if compatibility.supported: + selection = ImplementationSelection( + implementation=implementation, + requested_id=requested_id, + ) + + return selection + if not allow_fallback: + raise _unsupported_implementation_error( + stage=stage, + requested_id=requested_id, + requested_reason=compatibility.reason, + fallback_disabled=True, + ) + + fallback_id = implementation.metadata.fallback_id + if fallback_id == implementation.metadata.implementation_id: + raise _unsupported_implementation_error( + stage=stage, + requested_id=requested_id, + requested_reason=compatibility.reason, + ) + fallback = cast( + StageT, + registry.resolve( + stage=stage, + requested_id=fallback_id, + context=context, + ), + ) + fallback_compatibility = check_compatibility(fallback) + if not fallback_compatibility.supported: + raise _unsupported_implementation_error( + stage=stage, + requested_id=requested_id, + requested_reason=compatibility.reason, + fallback_id=fallback_id, + fallback_reason=fallback_compatibility.reason, + ) + + selection = ImplementationSelection( + implementation=fallback, + requested_id=requested_id, + fallback_reason=compatibility.reason, + ) + + return selection + + +def _unsupported_implementation_error( + *, + stage: OptimizationStage, + requested_id: str, + requested_reason: str, + fallback_id: Optional[str] = None, + fallback_reason: Optional[str] = None, + fallback_disabled: bool = False, +) -> ModelRuntimeError: + details = f"{requested_id!r} is unsupported: {requested_reason}." + if fallback_disabled: + details += " Compatibility fallback is disabled by the execution plan." + elif fallback_id is not None: + details += f" Fallback {fallback_id!r} is unsupported: {fallback_reason}." + error = ModelRuntimeError( + message=f"RF-DETR {stage.value} cannot execute this contract. {details}", + help_url=( + "https://inference-models.roboflow.com/errors/models-runtime/" + "#modelruntimeerror" + ), + ) + + return error diff --git a/inference_models/inference_models/models/rfdetr/pre_processing.py b/inference_models/inference_models/models/rfdetr/pre_processing.py index 3e2ceaad09..a5d2d7420e 100644 --- a/inference_models/inference_models/models/rfdetr/pre_processing.py +++ b/inference_models/inference_models/models/rfdetr/pre_processing.py @@ -11,6 +11,8 @@ tensor F.resize → F.normalize """ +from concurrent.futures import ThreadPoolExecutor +from functools import lru_cache from typing import List, Optional, Tuple, Union import numpy as np @@ -39,6 +41,54 @@ make_the_value_divisible, pre_process_numpy_image, ) +from inference_models.models.rfdetr.optimization.ids import ( + RFDETR_PREPROCESSOR_BASE, + RFDETR_PREPROCESSOR_DEFAULT_MAX_WORKERS, + RFDETR_PREPROCESSOR_MAX_WORKERS_ENV_NAME, + RFDETR_PREPROCESSOR_THREADED_EXACT_V1, +) +from inference_models.utils.environment import get_integer_from_env + + +def resolve_rfdetr_preprocessor_max_workers(max_workers: Optional[int] = None) -> int: + """Resolve the explicit or environment-selected preprocessing worker limit. + + Args: + max_workers: Explicit worker limit. When omitted, + ``INFERENCE_MODELS_RFDETR_PREPROCESSOR_MAX_WORKERS`` is used, + defaulting to ``4``. + + Returns: + Positive preprocessing worker limit. + + Raises: + InvalidEnvVariable: If the environment value is not an integer. + ModelRuntimeError: If the resolved worker limit is less than one. + """ + if max_workers is None: + max_workers = get_integer_from_env( + variable_name=RFDETR_PREPROCESSOR_MAX_WORKERS_ENV_NAME, + default=RFDETR_PREPROCESSOR_DEFAULT_MAX_WORKERS, + ) + if max_workers < 1: + raise ModelRuntimeError( + message="RF-DETR preprocessor_max_workers must be at least 1.", + help_url=( + "https://inference-models.roboflow.com/errors/models-runtime/" + "#modelruntimeerror" + ), + ) + + return max_workers + + +@lru_cache(maxsize=None) +def _log_selected_preprocessor(implementation_id: str, max_workers: int) -> None: + LOGGER.warning( + "Selected RF-DETR preprocessor implementation=%s max_workers=%d", + implementation_id, + max_workers, + ) def pre_process_network_input( @@ -49,7 +99,53 @@ def pre_process_network_input( input_color_format: Optional[ColorFormat] = None, image_size_wh: Optional[Union[int, Tuple[int, int]]] = None, pre_processing_overrides: Optional[PreProcessingOverrides] = None, + preprocessor_implementation_id: str = RFDETR_PREPROCESSOR_BASE, + preprocessor_max_workers: int = RFDETR_PREPROCESSOR_DEFAULT_MAX_WORKERS, ) -> Tuple[torch.Tensor, List[PreProcessingMetadata]]: + """Preprocess RF-DETR inputs with the selected implementation. + + Args: + images: Single image or image batch represented by NumPy arrays or tensors. + image_pre_processing: Model-package image preprocessing configuration. + network_input: Model-package network input definition. + target_device: Device receiving the preprocessed batch. + input_color_format: Optional color format supplied by the caller. + image_size_wh: Optional requested network input dimensions. + pre_processing_overrides: Optional request-specific preprocessing overrides. + preprocessor_implementation_id: Explicit implementation ID. + preprocessor_max_workers: Explicit threaded worker limit. + + Returns: + Contiguous NCHW batch and per-image preprocessing metadata. + + Raises: + ModelRuntimeError: If the selected implementation or worker limit is invalid. + TypeError: If an input type is unsupported by the selected implementation. + """ + supported_preprocessors = { + RFDETR_PREPROCESSOR_BASE, + RFDETR_PREPROCESSOR_THREADED_EXACT_V1, + } + if preprocessor_implementation_id not in supported_preprocessors: + raise ModelRuntimeError( + message=( + "The RF-DETR reference preprocessing adapter supports only " + f"{sorted(supported_preprocessors)}, received " + f"{preprocessor_implementation_id!r}." + ), + help_url=( + "https://inference-models.roboflow.com/errors/models-runtime/" + "#modelruntimeerror" + ), + ) + selected_preprocessor = preprocessor_implementation_id + preprocessor_max_workers = resolve_rfdetr_preprocessor_max_workers( + max_workers=preprocessor_max_workers + ) + _log_selected_preprocessor( + implementation_id=selected_preprocessor, + max_workers=preprocessor_max_workers, + ) input_color_mode = ( ColorMode(input_color_format) if input_color_format is not None else None ) @@ -92,43 +188,104 @@ def pre_process_network_input( else: image_list = [images] - tensors: List[torch.Tensor] = [] - metadata: List[PreProcessingMetadata] = [] - for img in image_list: - if isinstance(img, torch.Tensor) and img.is_floating_point(): - tensor, meta = _pre_process_tensor( - image=img, - image_pre_processing=image_pre_processing, - network_input=network_input, - target_size=target_size, - input_color_mode=input_color_mode, - pre_processing_overrides=pre_processing_overrides, - ) - elif isinstance(img, (np.ndarray, torch.Tensor)): - np_img = ( - _tensor_to_hwc_uint8(img) - if isinstance(img, torch.Tensor) - else _ensure_hwc_uint8(img) + def preprocess_one( + image: Union[np.ndarray, torch.Tensor], + ) -> Tuple[torch.Tensor, PreProcessingMetadata]: + return _pre_process_one( + image=image, + image_pre_processing=image_pre_processing, + network_input=network_input, + target_size=target_size, + input_color_mode=input_color_mode, + pre_processing_overrides=pre_processing_overrides, + ) + + if selected_preprocessor == RFDETR_PREPROCESSOR_THREADED_EXACT_V1: + unsupported = [ + type(image).__name__ + for image in image_list + if not isinstance(image, np.ndarray) + ] + if unsupported: + raise ModelRuntimeError( + message=( + f"{RFDETR_PREPROCESSOR_THREADED_EXACT_V1!r} accepts only " + "numpy uint8 HWC/NHWC images; received unsupported entries: " + f"{unsupported}. Select 'base' for torch.Tensor inputs." + ), + help_url=( + "https://inference-models.roboflow.com/errors/models-runtime/" + "#modelruntimeerror" + ), ) - tensor, meta = _pre_process_numpy( - image=np_img, - image_pre_processing=image_pre_processing, - network_input=network_input, - target_size=target_size, - input_color_mode=input_color_mode, - pre_processing_overrides=pre_processing_overrides, + invalid = [ + (str(image.dtype), tuple(image.shape)) + for image in image_list + if image.dtype != np.uint8 or image.ndim != 3 or image.shape[-1] != 3 + ] + if invalid: + raise ModelRuntimeError( + message=( + f"{RFDETR_PREPROCESSOR_THREADED_EXACT_V1!r} requires uint8 " + f"HWC images with 3 channels; received: {invalid}." + ), + help_url=( + "https://inference-models.roboflow.com/errors/models-runtime/" + "#modelruntimeerror" + ), ) + if len(image_list) > 1 and preprocessor_max_workers > 1: + worker_count = min(len(image_list), preprocessor_max_workers) + with ThreadPoolExecutor(max_workers=worker_count) as executor: + processed = list(executor.map(preprocess_one, image_list)) else: - raise TypeError( - f"Unsupported image input type for RFDETR pre-processing: {type(img)}" - ) - tensors.append(tensor.to(device=target_device)) - metadata.append(meta) + processed = [preprocess_one(image) for image in image_list] + else: + processed = [preprocess_one(image) for image in image_list] + + tensors = [tensor.to(device=target_device) for tensor, _ in processed] + metadata = [meta for _, meta in processed] batch = torch.stack(tensors).contiguous() return batch, metadata +def _pre_process_one( + image: Union[np.ndarray, torch.Tensor], + image_pre_processing: ImagePreProcessing, + network_input: NetworkInputDefinition, + target_size: ImageDimensions, + input_color_mode: Optional[ColorMode], + pre_processing_overrides: Optional[PreProcessingOverrides], +) -> Tuple[torch.Tensor, PreProcessingMetadata]: + if isinstance(image, torch.Tensor) and image.is_floating_point(): + return _pre_process_tensor( + image=image, + image_pre_processing=image_pre_processing, + network_input=network_input, + target_size=target_size, + input_color_mode=input_color_mode, + pre_processing_overrides=pre_processing_overrides, + ) + if isinstance(image, (np.ndarray, torch.Tensor)): + np_image = ( + _tensor_to_hwc_uint8(image) + if isinstance(image, torch.Tensor) + else _ensure_hwc_uint8(image) + ) + return _pre_process_numpy( + image=np_image, + image_pre_processing=image_pre_processing, + network_input=network_input, + target_size=target_size, + input_color_mode=input_color_mode, + pre_processing_overrides=pre_processing_overrides, + ) + raise TypeError( + f"Unsupported image input type for RFDETR pre-processing: {type(image)}" + ) + + def _pre_process_numpy( image: np.ndarray, image_pre_processing: ImagePreProcessing, diff --git a/inference_models/inference_models/models/rfdetr/rfdetr_object_detection_trt.py b/inference_models/inference_models/models/rfdetr/rfdetr_object_detection_trt.py index 64497911f9..1448a35c6c 100644 --- a/inference_models/inference_models/models/rfdetr/rfdetr_object_detection_trt.py +++ b/inference_models/inference_models/models/rfdetr/rfdetr_object_detection_trt.py @@ -1,5 +1,6 @@ import threading -from typing import List, Optional, Tuple, Union +from dataclasses import replace +from typing import Any, Dict, List, Mapping, Optional, Tuple, Union, cast import numpy as np import torch @@ -15,6 +16,7 @@ MissingDependencyError, ModelRuntimeError, ) +from inference_models.logger import LOGGER from inference_models.models.common.cuda import ( use_cuda_context, use_primary_cuda_context, @@ -37,12 +39,41 @@ infer_from_trt_engine, load_trt_model, ) +from inference_models.models.optimization.contracts import ( + ExecutionContext, + OptimizationMetadata, + OptimizationStage, +) +from inference_models.models.optimization.fallback_warnings import ( + FallbackWarningTracker, +) +from inference_models.models.optimization.ids import BASE_IMPLEMENTATION_ID from inference_models.models.rfdetr.class_remapping import ( ClassesReMapping, prepare_class_remapping, ) -from inference_models.models.rfdetr.common import post_process_object_detection_results -from inference_models.models.rfdetr.pre_processing import pre_process_network_input +from inference_models.models.rfdetr.optimization.catalog import ( + build_rfdetr_implementation_registry, +) +from inference_models.models.rfdetr.optimization.contracts import ( + Postprocessor, + PostprocessRequest, + PreprocessRequest, +) +from inference_models.models.rfdetr.optimization.execution_plan import ( + RFDetrExecutionPlan, +) +from inference_models.models.rfdetr.optimization.readiness import ( + PreprocessReadinessTracker, +) +from inference_models.models.rfdetr.optimization.selection import ( + resolve_postprocessor_for_request, + resolve_preprocessor_for_model, + resolve_preprocessor_for_request, +) +from inference_models.models.rfdetr.pre_processing import ( + resolve_rfdetr_preprocessor_max_workers, +) from inference_models.weights_providers.entities import RecommendedParameters try: @@ -80,6 +111,8 @@ class RFDetrForObjectDetectionTRT( ] ) ): + """Run RF-DETR object detection through TensorRT with selectable path stages.""" + @classmethod def from_pretrained( cls, @@ -89,9 +122,34 @@ def from_pretrained( trt_cuda_graph_cache: Optional[TRTCudaGraphCache] = None, default_trt_cuda_graph_cache_size: int = 8, rf_detr_max_input_resolution: Optional[Union[int, Tuple[int, int]]] = None, + rfdetr_preprocessor_max_workers: Optional[int] = None, + rfdetr_execution_plan: Optional[RFDetrExecutionPlan] = None, recommended_parameters: Optional[RecommendedParameters] = None, **kwargs, ) -> "RFDetrForObjectDetectionTRT": + """Load an RF-DETR TensorRT model package. + + Args: + model_name_or_path: Local model package directory. + device: CUDA device used for inference. + engine_host_code_allowed: Whether TensorRT may execute engine host code. + trt_cuda_graph_cache: Optional caller-managed CUDA graph cache. + default_trt_cuda_graph_cache_size: Default automatic graph-cache capacity. + rf_detr_max_input_resolution: Optional maximum accepted input resolution. + rfdetr_preprocessor_max_workers: Explicit threaded preprocessing worker + limit. When omitted, the corresponding environment value is used. + rfdetr_execution_plan: Explicit composed execution plan. When omitted, + RF-DETR implementation environment variables are used. + recommended_parameters: Optional model-specific recommended parameters. + **kwargs: Additional loader arguments accepted for API compatibility. + + Returns: + Loaded RF-DETR TensorRT model. + + Raises: + ModelRuntimeError: If the target or implementation selection is invalid. + CorruptedModelPackageError: If required package contents are inconsistent. + """ if device.type != "cuda": raise ModelRuntimeError( message=f"TRT engine only runs on CUDA device - {device} device detected.", @@ -180,6 +238,8 @@ def from_pretrained( cuda_context=cuda_context, execution_context=execution_context, trt_cuda_graph_cache=trt_cuda_graph_cache, + rfdetr_preprocessor_max_workers=rfdetr_preprocessor_max_workers, + rfdetr_execution_plan=rfdetr_execution_plan, recommended_parameters=recommended_parameters, ) @@ -196,6 +256,8 @@ def __init__( cuda_context: cuda.Context, execution_context: trt.IExecutionContext, trt_cuda_graph_cache: Optional[TRTCudaGraphCache], + rfdetr_preprocessor_max_workers: Optional[int] = None, + rfdetr_execution_plan: Optional[RFDetrExecutionPlan] = None, recommended_parameters=None, ): self._engine = engine @@ -209,8 +271,64 @@ def __init__( self._execution_context = execution_context self._trt_config = trt_config self._trt_cuda_graph_cache = trt_cuda_graph_cache + self._rfdetr_preprocessor_max_workers = resolve_rfdetr_preprocessor_max_workers( + max_workers=rfdetr_preprocessor_max_workers + ) + requested_plan = RFDetrExecutionPlan.resolve( + execution_plan=rfdetr_execution_plan, + ) + self._implementation_registry = build_rfdetr_implementation_registry( + device=self._device, + preprocessor_max_workers=self._rfdetr_preprocessor_max_workers, + ) + resolution_context = self._execution_stage_context(current_stream=None) + preprocessor_selection = resolve_preprocessor_for_model( + registry=self._implementation_registry, + requested_id=requested_plan.preprocessor_id, + context=resolution_context, + image_pre_processing=self._inference_config.image_pre_processing, + network_input=self._inference_config.network_input, + allow_fallback=requested_plan.allow_compatibility_fallback, + ) + self._preprocessor = preprocessor_selection.implementation + self._preprocessor_model_selection = preprocessor_selection.to_dict() + if preprocessor_selection.used_fallback: + LOGGER.warning( + "RF-DETR preprocessor fallback requested=%s effective=%s reason=%s", + preprocessor_selection.requested_id, + preprocessor_selection.effective_id, + preprocessor_selection.fallback_reason, + ) + self._postprocessor = cast( + Postprocessor, + self._implementation_registry.resolve( + stage=OptimizationStage.POSTPROCESS, + requested_id=requested_plan.postprocessor_id, + context=resolution_context, + ), + ) + self._rfdetr_execution_plan = RFDetrExecutionPlan( + preprocessor_id=self._preprocessor.metadata.implementation_id, + buffer_strategy_id=requested_plan.buffer_strategy_id, + scheduler_id=requested_plan.scheduler_id, + postprocessor_id=self._postprocessor.metadata.implementation_id, + engine_plugin_id=requested_plan.engine_plugin_id, + allow_compatibility_fallback=(requested_plan.allow_compatibility_fallback), + ) self._lock = threading.Lock() + self._request_fallback_warnings = FallbackWarningTracker() self._inference_stream = torch.cuda.Stream(device=self._device) + self._preprocess_readiness = PreprocessReadinessTracker() + if self.preprocessor_implementation_id != BASE_IMPLEMENTATION_ID: + LOGGER.info( + "Selected RF-DETR preprocessor implementation=%s", + self.preprocessor_implementation_id, + ) + if self.postprocessor_implementation_id != BASE_IMPLEMENTATION_ID: + LOGGER.info( + "Selected RF-DETR postprocessor implementation=%s", + self.postprocessor_implementation_id, + ) self._thread_local_storage = threading.local() self.recommended_parameters = recommended_parameters @@ -218,24 +336,160 @@ def __init__( def class_names(self) -> List[str]: return self._class_names + @property + def preprocessor_implementation_id(self) -> str: + """Return the actually selected preprocessing implementation ID.""" + return self._preprocessor.metadata.implementation_id + + @property + def preprocessor_implementation_metadata(self) -> OptimizationMetadata: + """Return typed metadata for the selected preprocessor.""" + return self._preprocessor.metadata + + @property + def postprocessor_implementation_id(self) -> str: + """Return the actually selected postprocessing implementation ID.""" + return self._postprocessor.metadata.implementation_id + + @property + def postprocessor_implementation_metadata(self) -> OptimizationMetadata: + """Return typed metadata for the selected postprocessor.""" + return self._postprocessor.metadata + + @property + def rfdetr_execution_plan(self) -> RFDetrExecutionPlan: + """Return the resolved composed execution plan.""" + return self._rfdetr_execution_plan + + @property + def optimization_runtime_metadata(self) -> Dict[str, Any]: + """Return machine-readable selected implementation metadata.""" + metadata = { + "execution_plan": self.rfdetr_execution_plan.to_dict(), + "preprocessor": self.preprocessor_implementation_metadata.to_dict(), + "postprocessor": self.postprocessor_implementation_metadata.to_dict(), + "model_selection": { + "preprocessor": dict(self._preprocessor_model_selection), + }, + } + last_execution = {} + for stage in ("preprocessor", "postprocessor"): + selection = getattr( + self._thread_local_storage, + f"last_{stage}_selection", + None, + ) + if selection is not None: + last_execution[stage] = dict(selection) + if last_execution: + metadata["last_execution"] = last_execution + + return metadata + + def infer( + self, + images: Union[torch.Tensor, List[torch.Tensor], np.ndarray, List[np.ndarray]], + **kwargs, + ) -> List[Detections]: + """Run composed inference with an asynchronous preprocessing handoff. + + Args: + images: Single image or image batch represented by arrays or tensors. + **kwargs: Inference arguments forwarded through all stages. + + Returns: + Per-image object detections. + """ + kwargs.pop("independent_stage_execution", None) + pre_processed_images, pre_processing_meta = self.pre_process( + images=images, + independent_stage_execution=False, + **kwargs, + ) + model_results = self.forward(pre_processed_images, **kwargs) + return self.post_process(model_results, pre_processing_meta, **kwargs) + def pre_process( self, images: Union[torch.Tensor, List[torch.Tensor], np.ndarray, List[np.ndarray]], input_color_format: Optional[ColorFormat] = None, pre_processing_overrides: Optional[PreProcessingOverrides] = None, + independent_stage_execution: bool = True, **kwargs, ) -> Tuple[torch.Tensor, List[PreProcessingMetadata]]: - with torch.cuda.stream(self._pre_process_stream): - pre_processed_images, pre_processing_meta = pre_process_network_input( - images=images, - image_pre_processing=self._inference_config.image_pre_processing, - network_input=self._inference_config.network_input, - target_device=self._device, - input_color_format=input_color_format, - pre_processing_overrides=pre_processing_overrides, + """Preprocess inference inputs using the resolved execution plan. + + Args: + images: Single image or image batch represented by arrays or tensors. + input_color_format: Optional caller-supplied color format. + pre_processing_overrides: Optional request preprocessing overrides. + independent_stage_execution: Whether preprocessing must finish before + returning. Composed ``infer()`` sets this to ``False`` and transfers + readiness to ``forward()`` through the exact returned tensor. + **kwargs: Additional request arguments accepted for API compatibility. + + Returns: + Preprocessed tensor and per-image transformation metadata. By default the + tensor is ready before return; otherwise its readiness is tracked for this + model instance's ``forward()`` method. + + Raises: + ModelRuntimeError: If the selected implementation is incompatible. + """ + stream = self._pre_process_stream + request = PreprocessRequest( + images=images, + input_color_format=input_color_format, + image_pre_processing=self._inference_config.image_pre_processing, + network_input=self._inference_config.network_input, + pre_processing_overrides=pre_processing_overrides, + ) + context = self._execution_stage_context( + current_stream=stream, + resolved_axes=self._resolved_preprocess_axes(images), + ) + selection = resolve_preprocessor_for_request( + registry=self._implementation_registry, + implementation=self._preprocessor, + request=request, + context=context, + allow_fallback=self._rfdetr_execution_plan.allow_compatibility_fallback, + ) + self._thread_local_storage.last_preprocessor_selection = selection.to_dict() + if selection.used_fallback and self._request_fallback_warnings.claim( + stage=OptimizationStage.PREPROCESS, + requested_id=selection.requested_id, + effective_id=selection.effective_id, + reason=selection.fallback_reason, + ): + LOGGER.warning( + "RF-DETR request preprocessor fallback requested=%s effective=%s " + "reason=%s", + selection.requested_id, + selection.effective_id, + selection.fallback_reason, ) - self._pre_process_stream.synchronize() - return pre_processed_images, pre_processing_meta + result = selection.implementation.preprocess( + request=request, + context=context, + ) + if selection.fallback_reason is not None: + result = replace(result, fallback_reason=selection.fallback_reason) + if result.ready_event is None: + stream.synchronize() + elif independent_stage_execution: + result.ready_event.synchronize() + + if not independent_stage_execution: + self._preprocess_readiness.record( + result.tensor, + ready_event=result.ready_event, + input_kind=result.input_kind, + implementation_id=result.implementation_id, + fallback_reason=result.fallback_reason, + ) + + return result.tensor, result.metadata def forward( self, @@ -243,9 +497,23 @@ def forward( disable_cuda_graphs: bool = False, **kwargs, ) -> Tuple[torch.Tensor, torch.Tensor]: + """Execute the protected TensorRT model forward pass. + + Args: + pre_processed_images: Ready tensor, or the exact tensor returned by this + model's tracked asynchronous preprocessing path. + disable_cuda_graphs: Whether to bypass the configured graph cache. + **kwargs: Additional request arguments accepted for API compatibility. + + Returns: + TensorRT detection boxes and logits. + """ cache = self._trt_cuda_graph_cache if not disable_cuda_graphs else None with self._lock: with use_cuda_context(context=self._cuda_context): + readiness = self._preprocess_readiness.consume(pre_processed_images) + if readiness is not None and readiness.ready_event is not None: + self._inference_stream.wait_event(readiness.ready_event) detections, labels = infer_from_trt_engine( pre_processed_images=pre_processed_images, trt_config=self._trt_config, @@ -266,27 +534,127 @@ def post_process( confidence: Confidence = "default", **kwargs, ) -> List[Detections]: + """Postprocess TensorRT outputs using the resolved execution plan. + + Args: + model_results: TensorRT detection boxes and logits. + pre_processing_meta: Per-image preprocessing transformations. + confidence: Global or class-specific confidence threshold selection. + **kwargs: Additional request arguments accepted for API compatibility. + + Returns: + Per-image object detections. + + Raises: + ModelRuntimeError: If the selected implementation is incompatible. + """ confidence_filter = ConfidenceFilter( confidence=confidence, recommended_parameters=self.recommended_parameters, default_confidence=INFERENCE_MODELS_RFDETR_DEFAULT_CONFIDENCE, ) - with torch.cuda.stream(self._post_process_stream): + threshold = confidence_filter.get_threshold(self.class_names) + stream = self._post_process_stream + with torch.cuda.stream(stream): for result_element in model_results: - result_element.record_stream(self._post_process_stream) + result_element.record_stream(stream) bboxes, logits = model_results - results = post_process_object_detection_results( + request = PostprocessRequest( bboxes=bboxes, logits=logits, pre_processing_meta=pre_processing_meta, - threshold=confidence_filter.get_threshold(self.class_names), + threshold=threshold, num_classes=len(self.class_names), classes_re_mapping=self._classes_re_mapping, - device=self._device, ) - self._post_process_stream.synchronize() + context = self._execution_stage_context( + current_stream=stream, + resolved_axes={ + "batch": int(logits.shape[0]), + "queries": int(logits.shape[1]), + "classes": int(logits.shape[2]), + }, + ) + selection = resolve_postprocessor_for_request( + registry=self._implementation_registry, + implementation=self._postprocessor, + request=request, + context=context, + allow_fallback=( + self._rfdetr_execution_plan.allow_compatibility_fallback + ), + ) + self._thread_local_storage.last_postprocessor_selection = ( + selection.to_dict() + ) + if selection.used_fallback and self._request_fallback_warnings.claim( + stage=OptimizationStage.POSTPROCESS, + requested_id=selection.requested_id, + effective_id=selection.effective_id, + reason=selection.fallback_reason, + ): + LOGGER.warning( + "RF-DETR request postprocessor fallback requested=%s " + "effective=%s reason=%s", + selection.requested_id, + selection.effective_id, + selection.fallback_reason, + ) + results = selection.implementation.postprocess( + request=request, + context=context, + ) + stream.synchronize() + return results + def _execution_stage_context( + self, + *, + current_stream: Optional[torch.cuda.Stream], + resolved_axes: Optional[Mapping[str, Any]] = None, + ) -> ExecutionContext: + device_index = self._device.index or 0 + context = ExecutionContext( + device_kind="gpu", + device=str(self._device), + device_name=torch.cuda.get_device_name(device_index), + machine_type="runtime", + scenario="runtime", + resolved_axes=resolved_axes or {}, + current_stream=current_stream, + compute_capability=torch.cuda.get_device_capability(device_index), + ) + + return context + + @staticmethod + def _resolved_preprocess_axes( + images: Union[ + torch.Tensor, + List[torch.Tensor], + np.ndarray, + List[np.ndarray], + ], + ) -> Dict[str, Any]: + if isinstance(images, list): + batch_size = len(images) + first = images[0] if images else None + elif isinstance(images, (torch.Tensor, np.ndarray)) and images.ndim == 4: + batch_size = int(images.shape[0]) + first = images[0] if batch_size else None + else: + batch_size = 1 + first = images + shape = tuple(first.shape) if first is not None else () + axes = { + "batch": batch_size, + "input_type": type(first).__name__ if first is not None else "empty", + "source_shape": shape, + } + + return axes + @property def _pre_process_stream(self) -> torch.cuda.Stream: if not hasattr(self._thread_local_storage, "pre_process_stream"): diff --git a/inference_models/inference_models/models/rfdetr/triton_object_detection_postprocess.py b/inference_models/inference_models/models/rfdetr/triton_object_detection_postprocess.py new file mode 100644 index 0000000000..1f32fa981f --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/triton_object_detection_postprocess.py @@ -0,0 +1,499 @@ +"""Explicit fused CUDA postprocessing for RF-DETR object detection. + +The implementation preserves RF-DETR's batched sigmoid and flat top-k +selection, then fuses class filtering/remapping, thresholding, stable +compaction, box conversion, metadata rescaling, and clipping into one Triton +kernel. A single device-to-host count handoff is required because the public +``Detections`` result contains variable-length tensors. + +The runtime exposes a side-effect-free compatibility check so the execution-plan +selector can follow the implementation's declared fallback before launching work. +Direct runtime calls remain strict, and execution failures never fall back. +""" + +from __future__ import annotations + +import threading +from collections import OrderedDict +from dataclasses import dataclass +from typing import Any, List, Optional, Tuple, Union + +import torch + +from inference_models import Detections +from inference_models.errors import ModelRuntimeError +from inference_models.models.common.roboflow.model_packages import PreProcessingMetadata +from inference_models.models.optimization.contracts import CompatibilityResult +from inference_models.models.rfdetr.class_remapping import ClassesReMapping +from inference_models.models.rfdetr.triton_jit_fallback import is_triton_jit_failure + +try: + import triton + import triton.language as tl + + TRITON_AVAILABLE = True +except ImportError: # pragma: no cover - depends on optional GPU package + triton = None + tl = None + TRITON_AVAILABLE = False + + +if TRITON_AVAILABLE: + + @triton.jit + def _compact_transform_detections_kernel( + scores_ptr, + topk_indices_ptr, + bboxes_ptr, + metadata_ptr, + class_mapping_ptr, + thresholds_ptr, + output_scores_ptr, + output_classes_ptr, + output_boxes_ptr, + output_counts_ptr, + num_queries, + num_logits_classes, + num_output_classes, + threshold_scalar, + MAPPING_SIZE: tl.constexpr, + HAS_CLASS_MAPPING: tl.constexpr, + HAS_CLASS_THRESHOLDS: tl.constexpr, + BLOCK_QUERIES: tl.constexpr, + METADATA_STRIDE: tl.constexpr, + ): + """Compact one batch row while preserving sorted top-k order.""" + batch_idx = tl.program_id(0) + offsets = tl.arange(0, BLOCK_QUERIES) + in_bounds = offsets < num_queries + input_offsets = batch_idx * num_queries + offsets + + scores = tl.load(scores_ptr + input_offsets, mask=in_bounds, other=0.0) + flat_indices = tl.load( + topk_indices_ptr + input_offsets, mask=in_bounds, other=0 + ) + raw_classes = flat_indices % num_logits_classes + query_indices = flat_indices // num_logits_classes + + if HAS_CLASS_MAPPING: + mapping_in_bounds = in_bounds & (raw_classes < MAPPING_SIZE) + mapped_classes = tl.load( + class_mapping_ptr + raw_classes, + mask=mapping_in_bounds, + other=-1, + ).to(tl.int32) + valid = mapping_in_bounds & (mapped_classes >= 0) + else: + mapped_classes = raw_classes.to(tl.int32) + valid = in_bounds & (raw_classes < num_output_classes) + + if HAS_CLASS_THRESHOLDS: + class_thresholds = tl.load( + thresholds_ptr + mapped_classes, + mask=valid, + other=0.0, + ) + valid = valid & (scores > class_thresholds) + else: + valid = valid & (scores > threshold_scalar) + + valid_int = valid.to(tl.int32) + compact_offsets = tl.cumsum(valid_int, axis=0) - 1 + output_offsets = batch_idx * num_queries + compact_offsets + + tl.store(output_scores_ptr + output_offsets, scores, mask=valid) + tl.store( + output_classes_ptr + output_offsets, + mapped_classes, + mask=valid, + ) + + box_offsets = (batch_idx * num_queries + query_indices) * 4 + cx = tl.load(bboxes_ptr + box_offsets, mask=in_bounds, other=0.0) + cy = tl.load(bboxes_ptr + box_offsets + 1, mask=in_bounds, other=0.0) + width = tl.load(bboxes_ptr + box_offsets + 2, mask=in_bounds, other=0.0) + height = tl.load(bboxes_ptr + box_offsets + 3, mask=in_bounds, other=0.0) + + metadata_base = batch_idx * METADATA_STRIDE + denorm_width = tl.load(metadata_ptr + metadata_base) + denorm_height = tl.load(metadata_ptr + metadata_base + 1) + pad_left = tl.load(metadata_ptr + metadata_base + 2) + pad_top = tl.load(metadata_ptr + metadata_base + 3) + scale_width = tl.load(metadata_ptr + metadata_base + 4) + scale_height = tl.load(metadata_ptr + metadata_base + 5) + crop_offset_x = tl.load(metadata_ptr + metadata_base + 6) + crop_offset_y = tl.load(metadata_ptr + metadata_base + 7) + original_width = tl.load(metadata_ptr + metadata_base + 8) + original_height = tl.load(metadata_ptr + metadata_base + 9) + + x1 = ((cx - 0.5 * width) * denorm_width - pad_left) / scale_width + y1 = ((cy - 0.5 * height) * denorm_height - pad_top) / scale_height + x2 = ((cx + 0.5 * width) * denorm_width - pad_left) / scale_width + y2 = ((cy + 0.5 * height) * denorm_height - pad_top) / scale_height + x1 = tl.maximum(0.0, tl.minimum(x1 + crop_offset_x, original_width)) + y1 = tl.maximum(0.0, tl.minimum(y1 + crop_offset_y, original_height)) + x2 = tl.maximum(0.0, tl.minimum(x2 + crop_offset_x, original_width)) + y2 = tl.maximum(0.0, tl.minimum(y2 + crop_offset_y, original_height)) + + output_box_offsets = output_offsets * 4 + tl.store(output_boxes_ptr + output_box_offsets, x1, mask=valid) + tl.store(output_boxes_ptr + output_box_offsets + 1, y1, mask=valid) + tl.store(output_boxes_ptr + output_box_offsets + 2, x2, mask=valid) + tl.store(output_boxes_ptr + output_box_offsets + 3, y2, mask=valid) + + count = tl.sum(valid_int, axis=0) + tl.store(output_counts_ptr + batch_idx, count) + + +@dataclass(frozen=True) +class _PreparedInputs: + metadata: torch.Tensor + class_mapping: torch.Tensor + thresholds: torch.Tensor + threshold_scalar: float + has_class_mapping: bool + has_class_thresholds: bool + + +@dataclass(frozen=True) +class _CachedTensor: + tensor: torch.Tensor + ready_event: torch.cuda.Event + + +class FusedObjectDetectionPostprocessor: + """Batched top-k plus one fused Triton object-detection postprocess.""" + + _MAX_CACHE_ENTRIES = 16 + _METADATA_STRIDE = 10 + + def __init__(self, device: torch.device) -> None: + if device.type != "cuda": + raise ModelRuntimeError( + message="triton-fused-v1 requires a CUDA target device.", + help_url=( + "https://inference-models.roboflow.com/errors/models-runtime/" + "#modelruntimeerror" + ), + ) + self._device = device + self._metadata_cache: OrderedDict[Tuple[Any, ...], _CachedTensor] = ( + OrderedDict() + ) + self._threshold_cache: OrderedDict[Tuple[Any, ...], _CachedTensor] = ( + OrderedDict() + ) + self._cache_lock = threading.Lock() + + def postprocess( + self, + bboxes: torch.Tensor, + logits: torch.Tensor, + pre_processing_meta: List[PreProcessingMetadata], + threshold: Union[float, torch.Tensor], + num_classes: int, + classes_re_mapping: Optional[ClassesReMapping], + stream: torch.cuda.Stream, + ) -> List[Detections]: + self._validate_inputs( + bboxes=bboxes, + logits=logits, + pre_processing_meta=pre_processing_meta, + threshold=threshold, + num_classes=num_classes, + classes_re_mapping=classes_re_mapping, + ) + if not TRITON_AVAILABLE: + raise ModelRuntimeError( + message="triton-fused-v1 requires Triton, but Triton is not installed.", + help_url=( + "https://inference-models.roboflow.com/errors/" + "runtime-environment/#missingdependencyerror" + ), + ) + + batch_size, num_queries, num_logits_classes = logits.shape + prepared = self._prepare_inputs( + metadata=pre_processing_meta, + threshold=threshold, + dtype=logits.dtype, + classes_re_mapping=classes_re_mapping, + stream=stream, + ) + + with torch.cuda.stream(stream): + sigmoid_logits = torch.sigmoid(logits) + scores, topk_indices = torch.topk( + sigmoid_logits.reshape(batch_size, -1), + num_queries, + dim=1, + largest=True, + sorted=True, + ) + output_scores = torch.empty_like(scores) + output_classes = torch.empty( + (batch_size, num_queries), + dtype=torch.int32, + device=self._device, + ) + output_boxes = torch.empty( + (batch_size, num_queries, 4), + dtype=bboxes.dtype, + device=self._device, + ) + output_counts = torch.empty( + (batch_size,), dtype=torch.int32, device=self._device + ) + + try: + _compact_transform_detections_kernel[(batch_size,)]( + scores, + topk_indices, + bboxes, + prepared.metadata, + prepared.class_mapping, + prepared.thresholds, + output_scores, + output_classes, + output_boxes, + output_counts, + num_queries, + num_logits_classes, + num_classes, + prepared.threshold_scalar, + MAPPING_SIZE=int(prepared.class_mapping.numel()), + HAS_CLASS_MAPPING=prepared.has_class_mapping, + HAS_CLASS_THRESHOLDS=prepared.has_class_thresholds, + BLOCK_QUERIES=triton.next_power_of_2(num_queries), + METADATA_STRIDE=self._METADATA_STRIDE, + num_warps=8, + ) + except Exception as error: + if not is_triton_jit_failure(error): + raise + raise ModelRuntimeError( + message=( + "triton-fused-v1 failed to compile or launch its " + f"postprocessing kernel: {type(error).__name__}: {error}" + ), + help_url=( + "https://inference-models.roboflow.com/errors/" + "models-runtime/#modelruntimeerror" + ), + ) from error + + # Variable-length Python results require one bounded count handoff. + # This synchronizes the fused kernel once for the whole batch. + counts = output_counts.cpu().tolist() + results = [ + Detections( + xyxy=output_boxes[index, :count].round().int(), + confidence=output_scores[index, :count], + class_id=output_classes[index, :count], + ) + for index, count in enumerate(counts) + ] + for tensor in (output_scores, output_classes, output_boxes): + tensor.record_stream(stream) + return results + + def _validate_inputs( + self, + bboxes: torch.Tensor, + logits: torch.Tensor, + pre_processing_meta: List[PreProcessingMetadata], + threshold: Union[float, torch.Tensor], + num_classes: int, + classes_re_mapping: Optional[ClassesReMapping], + ) -> None: + compatibility = self.check_request_compatibility( + bboxes=bboxes, + logits=logits, + pre_processing_meta=pre_processing_meta, + threshold=threshold, + num_classes=num_classes, + classes_re_mapping=classes_re_mapping, + ) + if compatibility.supported: + return + + raise ModelRuntimeError( + message=( + "triton-fused-v1 cannot preserve this postprocessing contract: " + + "; ".join(compatibility.reasons) + + ". Select 'base' for this configuration." + ), + help_url=( + "https://inference-models.roboflow.com/errors/models-runtime/" + "#modelruntimeerror" + ), + ) + + def check_request_compatibility( + self, + *, + bboxes: torch.Tensor, + logits: torch.Tensor, + pre_processing_meta: List[PreProcessingMetadata], + threshold: Union[float, torch.Tensor], + num_classes: int, + classes_re_mapping: Optional[ClassesReMapping], + ) -> CompatibilityResult: + """Check inputs supported by fused Triton postprocessing. + + Args: + bboxes: Batched CUDA box predictions. + logits: Batched CUDA class logits. + pre_processing_meta: Per-image preprocessing transformations. + threshold: Scalar or per-class confidence threshold. + num_classes: Number of output classes. + classes_re_mapping: Optional class-remapping tensors. + + Returns: + Compatibility result with every unsupported input characteristic. + """ + unsupported = [] + if not TRITON_AVAILABLE: + unsupported.append("Triton is not installed") + if not bboxes.is_cuda or not logits.is_cuda: + unsupported.append("boxes and logits must be CUDA tensors") + if bboxes.device != logits.device or bboxes.device != self._device: + unsupported.append( + "boxes, logits, and target must use the same CUDA device" + ) + if bboxes.dtype != torch.float32 or logits.dtype != torch.float32: + unsupported.append("boxes and logits must use float32") + if not bboxes.is_contiguous() or not logits.is_contiguous(): + unsupported.append("boxes and logits must be contiguous") + if bboxes.ndim != 3 or bboxes.shape[-1] != 4: + unsupported.append(f"boxes shape must be BQ4, got {tuple(bboxes.shape)}") + if logits.ndim != 3: + unsupported.append(f"logits shape must be BQC, got {tuple(logits.shape)}") + if bboxes.ndim == 3 and logits.ndim == 3: + if bboxes.shape[:2] != logits.shape[:2]: + unsupported.append("boxes and logits batch/query dimensions must match") + if logits.shape[1] < 1 or logits.shape[1] > 1024: + unsupported.append("query count must be between 1 and 1024") + if logits.shape[2] < 1: + unsupported.append("logits must contain at least one class") + if len(pre_processing_meta) != logits.shape[0]: + unsupported.append("metadata length must match batch size") + if num_classes < 1: + unsupported.append("num_classes must be positive") + if isinstance(threshold, torch.Tensor): + if threshold.ndim != 1 or threshold.numel() != num_classes: + unsupported.append("per-class threshold must have num_classes values") + if classes_re_mapping is not None: + if not classes_re_mapping.class_mapping.is_cuda: + unsupported.append("class mapping must be a CUDA tensor") + if classes_re_mapping.class_mapping.device != self._device: + unsupported.append("class mapping must use the target CUDA device") + if unsupported: + result = CompatibilityResult.incompatible(*unsupported) + else: + result = CompatibilityResult.compatible() + + return result + + def _prepare_inputs( + self, + metadata: List[PreProcessingMetadata], + threshold: Union[float, torch.Tensor], + dtype: torch.dtype, + classes_re_mapping: Optional[ClassesReMapping], + stream: torch.cuda.Stream, + ) -> _PreparedInputs: + metadata_values = tuple( + value for element in metadata for value in _metadata_values(element) + ) + metadata_key = (self._device.index, dtype, metadata_values) + metadata_tensor = self._cached_device_tensor( + cache=self._metadata_cache, + key=metadata_key, + values=metadata_values, + dtype=dtype, + shape=(len(metadata), self._METADATA_STRIDE), + stream=stream, + ) + + if classes_re_mapping is None: + class_mapping = metadata_tensor + else: + class_mapping = classes_re_mapping.class_mapping + + if isinstance(threshold, torch.Tensor): + if threshold.is_cuda: + thresholds = threshold.to(device=self._device, dtype=dtype) + threshold.record_stream(stream) + thresholds.record_stream(stream) + else: + threshold_values = tuple(float(value) for value in threshold.tolist()) + threshold_key = (self._device.index, dtype, threshold_values) + thresholds = self._cached_device_tensor( + cache=self._threshold_cache, + key=threshold_key, + values=threshold_values, + dtype=dtype, + shape=(len(threshold_values),), + stream=stream, + ) + threshold_scalar = 0.0 + has_class_thresholds = True + else: + thresholds = metadata_tensor + threshold_scalar = float(threshold) + has_class_thresholds = False + + return _PreparedInputs( + metadata=metadata_tensor, + class_mapping=class_mapping, + thresholds=thresholds, + threshold_scalar=threshold_scalar, + has_class_mapping=classes_re_mapping is not None, + has_class_thresholds=has_class_thresholds, + ) + + def _cached_device_tensor( + self, + cache: OrderedDict[Tuple[Any, ...], _CachedTensor], + key: Tuple[Any, ...], + values: Tuple[float, ...], + dtype: torch.dtype, + shape: Tuple[int, ...], + stream: torch.cuda.Stream, + ) -> torch.Tensor: + with self._cache_lock: + cached = cache.get(key) + if cached is not None: + cache.move_to_end(key) + with torch.cuda.stream(stream): + stream.wait_event(cached.ready_event) + cached.tensor.record_stream(stream) + return cached.tensor + with torch.cuda.stream(stream): + tensor = torch.tensor(values, dtype=dtype, device=self._device).reshape( + shape + ) + ready_event = torch.cuda.Event() + ready_event.record(stream) + tensor.record_stream(stream) + cache[key] = _CachedTensor(tensor=tensor, ready_event=ready_event) + while len(cache) > self._MAX_CACHE_ENTRIES: + cache.popitem(last=False) + return tensor + + +def _metadata_values(metadata: PreProcessingMetadata) -> Tuple[float, ...]: + denorm_size = metadata.nonsquare_intermediate_size or metadata.inference_size + return ( + float(denorm_size.width), + float(denorm_size.height), + float(metadata.pad_left), + float(metadata.pad_top), + float(metadata.scale_width), + float(metadata.scale_height), + float(metadata.static_crop_offset.offset_x), + float(metadata.static_crop_offset.offset_y), + float(metadata.original_size.width), + float(metadata.original_size.height), + ) diff --git a/inference_models/inference_models/models/rfdetr/triton_preprocess.py b/inference_models/inference_models/models/rfdetr/triton_preprocess.py index a3d88d0a69..c8c5ebaec7 100644 --- a/inference_models/inference_models/models/rfdetr/triton_preprocess.py +++ b/inference_models/inference_models/models/rfdetr/triton_preprocess.py @@ -169,6 +169,7 @@ def horizontal_resize_uint8_all_channels_kernel( src_w, src_stride_h, src_stride_w, + src_stride_c, crop_offset_y, crop_offset_x, target_w, @@ -192,8 +193,9 @@ def horizontal_resize_uint8_all_channels_kernel( ``(target_w * KSIZE_X,)``. src_h/src_w: Logical source height/width after crop. These drive bounds checks for the resized region. - src_stride_h/src_stride_w: Source strides in elements, used so the - kernel does not assume contiguous row pitch beyond HWC layout. + src_stride_h/src_stride_w/src_stride_c: Source strides in elements, + used so the kernel accepts both contiguous HWC tensors and HWC + views over CUDA CHW tensors without a layout conversion. crop_offset_y/crop_offset_x: Offset into ``src_ptr`` for static crop support. The TRT fast path passes zero. target_w: Width of the resized network input. @@ -232,9 +234,15 @@ def horizontal_resize_uint8_all_channels_kernel( base = sy[:, None] * src_stride_h + sx_c[None, :] * src_stride_w # Load source pixels in the network's channel order so the channel # swap replaces the original PIL image conversion step. - p_r = tl.load(src_ptr + base + CH_R, mask=mask_out, other=0).to(tl.int32) - p_g = tl.load(src_ptr + base + CH_G, mask=mask_out, other=0).to(tl.int32) - p_b = tl.load(src_ptr + base + CH_B, mask=mask_out, other=0).to(tl.int32) + p_r = tl.load( + src_ptr + base + CH_R * src_stride_c, mask=mask_out, other=0 + ).to(tl.int32) + p_g = tl.load( + src_ptr + base + CH_G * src_stride_c, mask=mask_out, other=0 + ).to(tl.int32) + p_b = tl.load( + src_ptr + base + CH_B * src_stride_c, mask=mask_out, other=0 + ).to(tl.int32) wx_2d = wx[None, :] # Fixed-point horizontal convolution: sum(src * PIL_weight_int). hacc_r += p_r * wx_2d @@ -493,6 +501,7 @@ def triton_preprocess_rfdetr_stretch_two_pass_preallocated( src_w = crop_w if crop_w is not None else raw_src_w src_stride_h = int(src.stride(0)) src_stride_w = int(src.stride(1)) + src_stride_c = int(src.stride(2)) dst_stride_c = target_h * target_w dst_stride_h = target_w @@ -524,6 +533,7 @@ def triton_preprocess_rfdetr_stretch_two_pass_preallocated( src_w, src_stride_h, src_stride_w, + src_stride_c, int(crop_offset_y), int(crop_offset_x), target_w, diff --git a/inference_models/inference_models/models/rfdetr/triton_universal_preprocess_runtime.py b/inference_models/inference_models/models/rfdetr/triton_universal_preprocess_runtime.py new file mode 100644 index 0000000000..99c6fc5ade --- /dev/null +++ b/inference_models/inference_models/models/rfdetr/triton_universal_preprocess_runtime.py @@ -0,0 +1,754 @@ +"""Explicit universal CUDA preprocessing for RF-DETR TensorRT adapters. + +The uint8 path uses the PIL-exact RF-DETR Triton kernels. NumPy and CPU tensor +inputs are staged through a two-slot pinned HWC ring; CUDA tensors are +consumed directly. Floating-point tensors retain RF-DETR's tensor-input +semantics and use torchvision's antialiased CUDA resize. + +This runtime remains strict when called directly. The RF-DETR implementation +selector checks its declared compatibility before execution and may choose the +base preprocessor for unsupported contracts. +""" + +from __future__ import annotations + +import threading +from dataclasses import dataclass +from typing import Any, List, Literal, Optional, Tuple, Union + +import numpy as np +import torch +import torchvision.transforms.functional as TF + +from inference_models import PreProcessingOverrides +from inference_models.entities import ColorFormat, ImageDimensions +from inference_models.errors import ModelRuntimeError +from inference_models.models.common.roboflow.model_packages import ( + ColorMode, + ImagePreProcessing, + NetworkInputDefinition, + PreProcessingMetadata, + ResizeMode, + StaticCropOffset, +) +from inference_models.models.optimization.contracts import CompatibilityResult +from inference_models.models.rfdetr.triton_jit_fallback import is_triton_jit_failure +from inference_models.models.rfdetr.triton_preprocess import ( + TRITON_AVAILABLE, + ResampleTables, + build_resample_tables, + resolve_two_pass_launch_config, + triton_preprocess_rfdetr_stretch_two_pass_preallocated, +) + +ImageInput = Union[np.ndarray, torch.Tensor] +_STAGING_RING_SIZE = 2 + + +@dataclass(frozen=True) +class UniversalFastPreprocessResult: + tensor: torch.Tensor + metadata: List[PreProcessingMetadata] + ready_event: torch.cuda.Event + input_kind: str + + +@dataclass(frozen=True) +class _CanonicalBatch: + items: Tuple[ImageInput, ...] + kind: Literal["uint8", "float"] + height: int + width: int + + +class _Uint8State: + """Reusable double-buffered staging and scratch state for one shape pair.""" + + __slots__ = ( + "src_h", + "src_w", + "target_h", + "target_w", + "pinned_host_ring", + "src_gpu_ring", + "slot_events", + "tmp", + "tables", + "launch_config", + "reuse_event", + ) + + def __init__( + self, + src_h: int, + src_w: int, + target_h: int, + target_w: int, + pinned_host_ring: Tuple[torch.Tensor, ...], + src_gpu_ring: Tuple[torch.Tensor, ...], + tmp: torch.Tensor, + tables: ResampleTables, + launch_config: Tuple[int, int, int, int], + ) -> None: + self.src_h = src_h + self.src_w = src_w + self.target_h = target_h + self.target_w = target_w + self.pinned_host_ring = pinned_host_ring + self.src_gpu_ring = src_gpu_ring + self.slot_events: List[Optional[torch.cuda.Event]] = [ + None for _ in range(_STAGING_RING_SIZE) + ] + self.tmp = tmp + self.tables = tables + self.launch_config = launch_config + self.reuse_event: Optional[torch.cuda.Event] = None + + @classmethod + def build( + cls, + src_h: int, + src_w: int, + target_h: int, + target_w: int, + device: torch.device, + ) -> "_Uint8State": + return cls( + src_h=src_h, + src_w=src_w, + target_h=target_h, + target_w=target_w, + pinned_host_ring=tuple( + torch.empty((src_h, src_w, 3), dtype=torch.uint8, pin_memory=True) + for _ in range(_STAGING_RING_SIZE) + ), + src_gpu_ring=tuple( + torch.empty((src_h, src_w, 3), dtype=torch.uint8, device=device) + for _ in range(_STAGING_RING_SIZE) + ), + tmp=torch.empty((3, src_h, target_w), dtype=torch.uint8, device=device), + tables=build_resample_tables( + src_h=src_h, + src_w=src_w, + target_h=target_h, + target_w=target_w, + device=device, + ), + launch_config=resolve_two_pass_launch_config(), + ) + + def matches(self, src_h: int, src_w: int, target_h: int, target_w: int) -> bool: + return ( + self.src_h == src_h + and self.src_w == src_w + and self.target_h == target_h + and self.target_w == target_w + ) + + +class UniversalFastPreprocessRuntime: + """Run one explicit CUDA preprocessing plan for supported RF-DETR inputs. + + The runtime serializes access to its reusable staging/scratch buffers. + Output tensors are allocated per call from PyTorch's CUDA allocator, so + concurrent callers never alias outputs. CUDA inputs are read on the + supplied preprocessing stream and have their allocator lifetime recorded. + """ + + def __init__(self, device: torch.device) -> None: + if device.type != "cuda": + raise ModelRuntimeError( + message="triton-universal-v1 requires a CUDA target device.", + help_url=( + "https://inference-models.roboflow.com/errors/models-runtime/" + "#modelruntimeerror" + ), + ) + self._device = device + self._uint8_state: Optional[_Uint8State] = None + self._state_lock = threading.Lock() + + def preprocess( + self, + images, + input_color_format: Optional[ColorFormat], + image_pre_processing: ImagePreProcessing, + network_input: NetworkInputDefinition, + pre_processing_overrides: Optional[PreProcessingOverrides], + stream: torch.cuda.Stream, + ) -> UniversalFastPreprocessResult: + model_compatibility = self.check_model_compatibility( + image_pre_processing=image_pre_processing, + network_input=network_input, + ) + self._raise_for_incompatibility(model_compatibility) + request_compatibility = self.check_request_compatibility( + images=images, + pre_processing_overrides=pre_processing_overrides, + ) + self._raise_for_incompatibility(request_compatibility) + batch = _canonicalize_batch(images) + caller_mode = ( + ColorMode(input_color_format) + if input_color_format is not None + else ColorMode.BGR + ) + swap_rb = caller_mode != network_input.color_mode + + means, stds = network_input.normalization + means_t = tuple(float(value) for value in means) + stds_t = tuple(float(value) for value in stds) + target_h = network_input.training_input_size.height + target_w = network_input.training_input_size.width + + if batch.kind == "uint8": + return self._preprocess_uint8( + batch=batch, + target_h=target_h, + target_w=target_w, + means=means_t, + stds=stds_t, + swap_rb=swap_rb, + stream=stream, + ) + return self._preprocess_float( + batch=batch, + target_h=target_h, + target_w=target_w, + means=means_t, + stds=stds_t, + swap_rb=swap_rb, + stream=stream, + ) + + def _preprocess_uint8( + self, + batch: _CanonicalBatch, + target_h: int, + target_w: int, + means: Tuple[float, float, float], + stds: Tuple[float, float, float], + swap_rb: bool, + stream: torch.cuda.Stream, + ) -> UniversalFastPreprocessResult: + if not TRITON_AVAILABLE: + raise ModelRuntimeError( + message=( + "triton-universal-v1 requires the existing Triton runtime " + "dependency, but Triton is not installed." + ), + help_url=( + "https://inference-models.roboflow.com/errors/" + "runtime-environment/#missingdependencyerror" + ), + ) + + with self._state_lock: + state = self._uint8_state + if state is not None and state.reuse_event is not None: + state.reuse_event.synchronize() + if state is None or not state.matches( + src_h=batch.height, + src_w=batch.width, + target_h=target_h, + target_w=target_w, + ): + with torch.cuda.stream(stream): + state = _Uint8State.build( + src_h=batch.height, + src_w=batch.width, + target_h=target_h, + target_w=target_w, + device=self._device, + ) + self._uint8_state = state + + output = torch.empty( + (len(batch.items), 3, target_h, target_w), + dtype=torch.float32, + device=self._device, + ) + try: + for index, item in enumerate(batch.items): + staging_slot = index % _STAGING_RING_SIZE + source = self._prepare_uint8_source( + item=item, + state=state, + stream=stream, + staging_slot=staging_slot, + ) + with torch.cuda.stream(stream): + triton_preprocess_rfdetr_stretch_two_pass_preallocated( + src=source, + out=output[index : index + 1], + tmp=state.tmp, + tables=state.tables, + target_h=target_h, + target_w=target_w, + means=means, + stds=stds, + swap_rb=swap_rb, + launch_config=state.launch_config, + ) + if not (isinstance(item, torch.Tensor) and item.is_cuda): + slot_event = torch.cuda.Event() + slot_event.record(stream) + state.slot_events[staging_slot] = slot_event + except Exception as error: + if not is_triton_jit_failure(error): + raise + raise ModelRuntimeError( + message=( + "triton-universal-v1 failed to compile or launch its " + f"preprocessing kernel: {type(error).__name__}: {error}" + ), + help_url=( + "https://inference-models.roboflow.com/errors/" + "models-runtime/#modelruntimeerror" + ), + ) from error + + with torch.cuda.stream(stream): + ready_event = torch.cuda.Event() + ready_event.record(stream) + output.record_stream(stream) + state.reuse_event = ready_event + + return UniversalFastPreprocessResult( + tensor=output, + metadata=_build_metadata_batch( + batch_size=len(batch.items), + source_h=batch.height, + source_w=batch.width, + target_h=target_h, + target_w=target_w, + ), + ready_event=ready_event, + input_kind="uint8-triton", + ) + + def _prepare_uint8_source( + self, + item: ImageInput, + state: _Uint8State, + stream: torch.cuda.Stream, + staging_slot: int, + ) -> torch.Tensor: + if isinstance(item, torch.Tensor) and item.is_cuda: + item.record_stream(stream) + return item + + slot_event = state.slot_events[staging_slot] + if slot_event is not None: + slot_event.synchronize() + pinned_host = state.pinned_host_ring[staging_slot] + src_gpu = state.src_gpu_ring[staging_slot] + + if isinstance(item, np.ndarray): + np.copyto(pinned_host.numpy(), item, casting="no") + else: + pinned_host.copy_(item) + + with torch.cuda.stream(stream): + src_gpu.copy_(pinned_host, non_blocking=True) + # The per-slot event is recorded after the corresponding Triton launch. + # A two-slot ring lets the host fill the next frame while the GPU copies + # and processes the current frame, without overwriting live storage. + return src_gpu + + def _preprocess_float( + self, + batch: _CanonicalBatch, + target_h: int, + target_w: int, + means: Tuple[float, float, float], + stds: Tuple[float, float, float], + swap_rb: bool, + stream: torch.cuda.Stream, + ) -> UniversalFastPreprocessResult: + with torch.cuda.stream(stream): + cuda_items = [] + for item in batch.items: + assert isinstance(item, torch.Tensor) + tensor = item.to( + device=self._device, + dtype=torch.float32, + non_blocking=item.device.type == "cpu" and item.is_pinned(), + ) + if item.is_cuda: + item.record_stream(stream) + cuda_items.append(tensor) + tensor_batch = torch.stack(cuda_items, dim=0) + if swap_rb: + tensor_batch = tensor_batch[:, [2, 1, 0], :, :] + resized = TF.resize( + tensor_batch, + (target_h, target_w), + antialias=True, + ) + output = TF.normalize( + resized, + mean=list(means), + std=list(stds), + ).contiguous() + ready_event = torch.cuda.Event() + ready_event.record(stream) + output.record_stream(stream) + + return UniversalFastPreprocessResult( + tensor=output, + metadata=_build_metadata_batch( + batch_size=len(batch.items), + source_h=batch.height, + source_w=batch.width, + target_h=target_h, + target_w=target_w, + ), + ready_event=ready_event, + input_kind="float-torch-cuda", + ) + + @staticmethod + def check_model_compatibility( + *, + image_pre_processing: ImagePreProcessing, + network_input: NetworkInputDefinition, + ) -> CompatibilityResult: + """Check static model configuration supported by the Triton runtime. + + Args: + image_pre_processing: Model-package image transformations. + network_input: Model-package network input definition. + + Returns: + Compatibility result with every unsupported configuration element. + """ + unsupported = [] + if network_input.resize_mode is not ResizeMode.STRETCH_TO: + unsupported.append(f"resize_mode={network_input.resize_mode!r}") + if network_input.dataset_version_resize_dimensions is not None: + unsupported.append("dataset-version resize") + if network_input.input_channels != 3: + unsupported.append(f"input_channels={network_input.input_channels}") + if network_input.scaling_factor not in (None, 255): + unsupported.append(f"scaling_factor={network_input.scaling_factor}") + if network_input.normalization is None: + unsupported.append("missing normalization") + elif any(len(values) != 3 for values in network_input.normalization): + unsupported.append("normalization must contain three channels") + if ( + image_pre_processing.static_crop is not None + and image_pre_processing.static_crop.enabled + ): + unsupported.append("static crop") + if ( + image_pre_processing.contrast is not None + and image_pre_processing.contrast.enabled + ): + unsupported.append("contrast") + if ( + image_pre_processing.grayscale is not None + and image_pre_processing.grayscale.enabled + ): + unsupported.append("grayscale") + if ( + image_pre_processing.auto_orient is not None + and image_pre_processing.auto_orient.enabled + ): + unsupported.append("auto orient") + if unsupported: + result = CompatibilityResult.incompatible(*unsupported) + else: + result = CompatibilityResult.compatible() + + return result + + @staticmethod + def check_request_compatibility( + *, + images, + pre_processing_overrides: Optional[PreProcessingOverrides], + ) -> CompatibilityResult: + """Check request-specific constraints without performing GPU work. + + Args: + images: Single image or batch supplied to preprocessing. + pre_processing_overrides: Optional request transformation overrides. + + Returns: + Compatibility result with every unsupported request characteristic. + """ + unsupported = [] + if pre_processing_overrides is not None and any( + ( + pre_processing_overrides.disable_contrast_enhancement, + pre_processing_overrides.disable_grayscale, + pre_processing_overrides.disable_static_crop, + ) + ): + unsupported.append("active pre-processing overrides") + + raw_items = _raw_batch_items(images) + if not raw_items: + unsupported.append("empty image batch") + kinds = [] + shapes = [] + for item in raw_items: + item_contract = _inspect_item_contract(item) + if isinstance(item_contract, str): + unsupported.append(item_contract) + continue + kind, shape = item_contract + kinds.append(kind) + shapes.append(shape) + if len(set(kinds)) > 1: + unsupported.append("mixed uint8 and floating tensor semantics") + if len(set(shapes)) > 1: + unsupported.append(f"heterogeneous source dimensions: {shapes}") + if "uint8" in kinds and not TRITON_AVAILABLE: + unsupported.append("Triton is not installed for uint8 preprocessing") + if unsupported: + result = CompatibilityResult.incompatible(*unsupported) + else: + result = CompatibilityResult.compatible() + + return result + + @staticmethod + def _raise_for_incompatibility(compatibility: CompatibilityResult) -> None: + if compatibility.supported: + return + + raise ModelRuntimeError( + message=( + "triton-universal-v1 cannot preserve this preprocessing contract: " + f"{compatibility.reason}. Select 'base' for this configuration." + ), + help_url=( + "https://inference-models.roboflow.com/errors/models-runtime/" + "#modelruntimeerror" + ), + ) + + +def _canonicalize_batch(images) -> _CanonicalBatch: + raw_items = _raw_batch_items(images) + + if not raw_items: + raise ModelRuntimeError( + message="triton-universal-v1 received an empty image batch.", + help_url=( + "https://inference-models.roboflow.com/errors/input-validation/" + "#modelinputerror" + ), + ) + + items = [] + kinds = [] + shapes = [] + for item in raw_items: + if isinstance(item, np.ndarray): + canonical, kind = _canonicalize_numpy(item) + elif isinstance(item, torch.Tensor): + canonical, kind = _canonicalize_tensor(item) + else: + raise ModelRuntimeError( + message=( + "triton-universal-v1 accepts numpy.ndarray and torch.Tensor " + f"inputs, received {type(item).__name__}." + ), + help_url=( + "https://inference-models.roboflow.com/errors/input-validation/" + "#modelinputerror" + ), + ) + items.append(canonical) + kinds.append(kind) + if kind == "uint8": + shapes.append((int(canonical.shape[0]), int(canonical.shape[1]))) + else: + shapes.append((int(canonical.shape[1]), int(canonical.shape[2]))) + + if len(set(kinds)) != 1: + raise ModelRuntimeError( + message=( + "triton-universal-v1 requires a homogeneous batch; mixing uint8 " + "image semantics with floating tensor semantics is unsupported." + ), + help_url=( + "https://inference-models.roboflow.com/errors/input-validation/" + "#modelinputerror" + ), + ) + if len(set(shapes)) != 1: + raise ModelRuntimeError( + message=( + "triton-universal-v1 currently requires equal source dimensions " + f"within one batch; received {shapes}." + ), + help_url=( + "https://inference-models.roboflow.com/errors/input-validation/" + "#modelinputerror" + ), + ) + + height, width = shapes[0] + if height < 1 or width < 1: + raise ModelRuntimeError( + message=( + "triton-universal-v1 requires non-empty image dimensions; " + f"received height={height}, width={width}." + ), + help_url=( + "https://inference-models.roboflow.com/errors/input-validation/" + "#modelinputerror" + ), + ) + return _CanonicalBatch( + items=tuple(items), + kind=kinds[0], + height=height, + width=width, + ) + + +def _raw_batch_items(images) -> List[Any]: + if isinstance(images, list): + items = list(images) + elif isinstance(images, torch.Tensor) and images.ndim == 4: + items = list(images.unbind(0)) + elif isinstance(images, np.ndarray) and images.ndim == 4: + items = list(images) + else: + items = [images] + + return items + + +def _inspect_item_contract( + image, +) -> Union[Tuple[Literal["uint8", "float"], Tuple[int, int]], str]: + if isinstance(image, np.ndarray): + if image.ndim != 3 or image.shape[-1] != 3: + return f"NumPy input must be HWC with three channels: {tuple(image.shape)}" + return "uint8", (int(image.shape[0]), int(image.shape[1])) + if not isinstance(image, torch.Tensor): + return f"unsupported input type: {type(image).__name__}" + if image.ndim != 3: + return f"tensor input must be rank 3: {tuple(image.shape)}" + + is_chw = image.shape[0] == 3 + is_hwc = image.shape[-1] == 3 + if not is_chw and not is_hwc: + return ( + f"tensor input must be CHW or HWC with three channels: {tuple(image.shape)}" + ) + if not image.is_floating_point() and image.dtype != torch.uint8: + return f"integer tensor input must use uint8: {image.dtype}" + + if is_chw: + shape = (int(image.shape[1]), int(image.shape[2])) + else: + shape = (int(image.shape[0]), int(image.shape[1])) + kind = "float" if image.is_floating_point() else "uint8" + + return kind, shape + + +def _canonicalize_numpy(image: np.ndarray) -> Tuple[np.ndarray, Literal["uint8"]]: + if image.ndim != 3 or image.shape[-1] != 3: + raise ModelRuntimeError( + message=( + "triton-universal-v1 requires NumPy images in HWC layout with " + f"three channels; received shape={tuple(image.shape)}." + ), + help_url=( + "https://inference-models.roboflow.com/errors/input-validation/" + "#modelinputerror" + ), + ) + if image.dtype == np.uint8: + return np.ascontiguousarray(image), "uint8" + if np.issubdtype(image.dtype, np.floating): + converted = (image * 255.0).clip(0, 255).astype(np.uint8) + return np.ascontiguousarray(converted), "uint8" + return np.ascontiguousarray(image.astype(np.uint8)), "uint8" + + +def _canonicalize_tensor( + image: torch.Tensor, +) -> Tuple[torch.Tensor, Literal["uint8", "float"]]: + if image.ndim != 3: + raise ModelRuntimeError( + message=( + "triton-universal-v1 requires each tensor image to be rank 3 " + f"(CHW or HWC); received shape={tuple(image.shape)}." + ), + help_url=( + "https://inference-models.roboflow.com/errors/input-validation/" + "#modelinputerror" + ), + ) + + is_chw = image.shape[0] == 3 + is_hwc = image.shape[-1] == 3 + if not is_chw and not is_hwc: + raise ModelRuntimeError( + message=( + "triton-universal-v1 requires three-channel CHW or HWC tensors; " + f"received shape={tuple(image.shape)}." + ), + help_url=( + "https://inference-models.roboflow.com/errors/input-validation/" + "#modelinputerror" + ), + ) + + if image.is_floating_point(): + chw = image if is_chw else image.permute(2, 0, 1) + return chw, "float" + if image.dtype != torch.uint8: + raise ModelRuntimeError( + message=( + "triton-universal-v1 accepts only uint8 integer tensors or " + f"floating tensors; received dtype={image.dtype}." + ), + help_url=( + "https://inference-models.roboflow.com/errors/input-validation/" + "#modelinputerror" + ), + ) + hwc = image.permute(1, 2, 0) if is_chw else image + return hwc, "uint8" + + +def _build_metadata_batch( + batch_size: int, + source_h: int, + source_w: int, + target_h: int, + target_w: int, +) -> List[PreProcessingMetadata]: + original_size = ImageDimensions(width=source_w, height=source_h) + target_size = ImageDimensions(width=target_w, height=target_h) + static_crop_offset = StaticCropOffset( + offset_x=0, + offset_y=0, + crop_width=source_w, + crop_height=source_h, + ) + return [ + PreProcessingMetadata( + pad_left=0, + pad_top=0, + pad_right=0, + pad_bottom=0, + original_size=original_size, + size_after_pre_processing=original_size, + inference_size=target_size, + scale_width=target_w / source_w, + scale_height=target_h / source_h, + static_crop_offset=static_crop_offset, + ) + for _ in range(batch_size) + ] diff --git a/inference_models/mkdocs.yml b/inference_models/mkdocs.yml index 99561a9c54..ec75014d98 100644 --- a/inference_models/mkdocs.yml +++ b/inference_models/mkdocs.yml @@ -25,6 +25,9 @@ extra: extra_css: - stylesheets/extra.css +extra_javascript: + - https://unpkg.com/mermaid@11/dist/mermaid.min.js + nav: - Home: index.md - Getting Started: @@ -111,6 +114,7 @@ nav: - Contributors: - Development Environment: contributors/dev-environment.md - Core Architecture: contributors/core-architecture.md + - Inference-Path Optimization Architecture: contributors/inference-path-optimization-architecture.md - Dependencies and Backends: contributors/dependencies-and-backends.md - Adding a Model: contributors/adding-model.md - Writing Tests: contributors/writing-tests.md @@ -240,9 +244,12 @@ markdown_extensions: anchor_linenums: true - pymdownx.inlinehilite - pymdownx.snippets - - pymdownx.superfences + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format - pymdownx.tabbed: alternate_style: true - toc: permalink: true - diff --git a/inference_models/pyproject.toml b/inference_models/pyproject.toml index 9f0236b7ab..bc4d12c427 100644 --- a/inference_models/pyproject.toml +++ b/inference_models/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "inference-models" -version = "0.31.0" +version = "0.32.0" description = "The new inference engine for Computer Vision models" readme = "README.md" requires-python = ">=3.10,<3.13" diff --git a/inference_models/tests/integration_tests/models/test_rfdetr_predictions_trt.py b/inference_models/tests/integration_tests/models/test_rfdetr_predictions_trt.py index f44e64dce1..0f2a5878ed 100644 --- a/inference_models/tests/integration_tests/models/test_rfdetr_predictions_trt.py +++ b/inference_models/tests/integration_tests/models/test_rfdetr_predictions_trt.py @@ -1,3 +1,5 @@ +from unittest.mock import patch + import numpy as np import pytest import torch @@ -69,6 +71,43 @@ def test_trt_package_numpy( ) +@pytest.mark.slow +@pytest.mark.trt_extras +def test_trt_independent_public_stages_match_composed_inference( + rfdetr_coin_counting_trt_package: str, + coins_counting_image_numpy: np.ndarray, +) -> None: + from inference_models.models.rfdetr.rfdetr_object_detection_trt import ( + RFDetrForObjectDetectionTRT, + ) + + model = RFDetrForObjectDetectionTRT.from_pretrained( + model_name_or_path=rfdetr_coin_counting_trt_package, + engine_host_code_allowed=True, + ) + with patch.object( + model._preprocess_readiness, + "record", + wraps=model._preprocess_readiness.record, + ) as record_readiness: + expected = model(coins_counting_image_numpy) + record_readiness.assert_called_once() + + pre_processed, metadata = model.pre_process(coins_counting_image_numpy) + assert model._preprocess_readiness.consume(pre_processed) is None + raw_predictions = model.forward(pre_processed) + actual = model.post_process(raw_predictions, metadata) + + assert len(actual) == len(expected) + for actual_image, expected_image in zip(actual, expected): + torch.testing.assert_close(actual_image.xyxy, expected_image.xyxy) + torch.testing.assert_close(actual_image.class_id, expected_image.class_id) + torch.testing.assert_close( + actual_image.confidence, + expected_image.confidence, + ) + + @pytest.mark.slow @pytest.mark.trt_extras def test_trt_package_batch_numpy( diff --git a/inference_models/tests/unit_tests/models/optimization/test_fallback_warnings.py b/inference_models/tests/unit_tests/models/optimization/test_fallback_warnings.py new file mode 100644 index 0000000000..3ded6358c4 --- /dev/null +++ b/inference_models/tests/unit_tests/models/optimization/test_fallback_warnings.py @@ -0,0 +1,38 @@ +from concurrent.futures import ThreadPoolExecutor + +from inference_models.models.optimization.contracts import OptimizationStage +from inference_models.models.optimization.fallback_warnings import ( + FallbackWarningTracker, +) + + +def test_fallback_warning_is_claimed_once_per_distinct_key() -> None: + tracker = FallbackWarningTracker() + warning = { + "stage": OptimizationStage.PREPROCESS, + "requested_id": "optimized", + "effective_id": "base", + "reason": "unsupported request", + } + + assert tracker.claim(**warning) + assert not tracker.claim(**warning) + assert tracker.claim(**{**warning, "reason": "different reason"}) + assert tracker.claim(**{**warning, "stage": OptimizationStage.POSTPROCESS}) + + +def test_fallback_warning_claim_is_thread_safe() -> None: + tracker = FallbackWarningTracker() + + def claim() -> bool: + return tracker.claim( + stage=OptimizationStage.PREPROCESS, + requested_id="optimized", + effective_id="base", + reason="unsupported request", + ) + + with ThreadPoolExecutor(max_workers=8) as executor: + claims = list(executor.map(lambda _: claim(), range(32))) + + assert claims.count(True) == 1 diff --git a/inference_models/tests/unit_tests/models/optimization/test_optimization_framework.py b/inference_models/tests/unit_tests/models/optimization/test_optimization_framework.py new file mode 100644 index 0000000000..c894e62e63 --- /dev/null +++ b/inference_models/tests/unit_tests/models/optimization/test_optimization_framework.py @@ -0,0 +1,158 @@ +from dataclasses import dataclass + +import pytest +import torch + +from inference_models.errors import ModelRuntimeError +from inference_models.models.optimization.contracts import ( + CompatibilityResult, + DeviceCompatibility, + ExecutionContext, + InputCompatibility, + OptimizationMetadata, + OptimizationStage, + ValidationEnvironment, +) +from inference_models.models.optimization.execution_plan import InferenceExecutionPlan +from inference_models.models.optimization.registry import ImplementationRegistry +from inference_models.models.optimization.torch_readiness import TensorReadinessTracker + + +@dataclass(frozen=True) +class _State: + implementation_id: str + + +class _Stage: + def __init__( + self, + implementation_id: str, + *, + compatible: bool = True, + validated_environments=(), + ) -> None: + self.metadata = OptimizationMetadata( + implementation_id=implementation_id, + stage=OptimizationStage.PREPROCESS, + version="1", + target=DeviceCompatibility(device_kind="gpu"), + inputs=InputCompatibility(scenarios=("*",)), + dependencies=(), + fallback_id="base", + changes_numerics=False, + supports_concurrency=True, + supports_cuda_graphs=False, + validated_environments=validated_environments, + ) + self._compatible = compatible + + def is_compatible(self, context: ExecutionContext) -> bool: + return self._compatible + + +def _context() -> ExecutionContext: + return ExecutionContext( + device_kind="gpu", + device="cuda:0", + device_name="test-gpu", + machine_type="test-machine", + scenario="batch", + resolved_axes={"batch": 1, "height": 640}, + ) + + +def test_inference_execution_plan_defaults_and_serializes() -> None: + plan = InferenceExecutionPlan() + + assert plan.to_dict() == { + "preprocessor": "base", + "buffer_strategy": "base", + "scheduler": "base", + "postprocessor": "base", + "engine_plugin": "base", + "allow_compatibility_fallback": True, + } + + +def test_compatibility_result_preserves_actionable_reasons() -> None: + result = CompatibilityResult.incompatible("static crop", "grayscale") + + assert not result.supported + assert result.reasons == ("static crop", "grayscale") + assert result.reason == "static crop, grayscale" + + +def test_validation_environment_matches_target_and_scenario() -> None: + validation = ValidationEnvironment( + machine_type="test-machine", + device_kind="gpu", + device_name="test-gpu", + scenario="batch", + resolved_axes={"batch": 1}, + runtime_versions={"torch": "test"}, + source_commit="test", + profiling_bundle="test-bundle", + status="validated", + ) + + assert validation.matches(_context()) + assert not validation.matches( + ExecutionContext( + device_kind="gpu", + device="cuda:0", + device_name="other-gpu", + machine_type="test-machine", + scenario="batch", + resolved_axes={"batch": 1}, + ) + ) + + +def test_registry_uses_scope_in_actionable_errors() -> None: + registry = ImplementationRegistry(scope_name="Example model") + registry.register(_Stage("base")) + + with pytest.raises(ModelRuntimeError, match="Unknown Example model preprocess"): + registry.resolve( + stage=OptimizationStage.PREPROCESS, + requested_id="missing", + context=_context(), + ) + + +def test_registry_auto_selects_only_matching_validated_candidate() -> None: + validation = ValidationEnvironment( + machine_type="test-machine", + device_kind="gpu", + device_name="test-gpu", + scenario="batch", + resolved_axes={"batch": 1}, + runtime_versions={}, + source_commit="test", + profiling_bundle="test-bundle", + status="validated", + ) + registry = ImplementationRegistry(scope_name="Example model") + base = _Stage("base") + candidate = _Stage("candidate", validated_environments=(validation,)) + registry.register(base) + registry.register(candidate) + + selected = registry.resolve( + stage=OptimizationStage.PREPROCESS, + requested_id="auto", + context=_context(), + ) + + assert selected is candidate + + +def test_tensor_readiness_tracker_uses_exact_tensor_identity() -> None: + tracker = TensorReadinessTracker[_State]() + tensor = torch.zeros(1) + other = torch.zeros(1) + tracker.record(tensor, state=_State(implementation_id="candidate")) + + assert tracker.consume(other) is None + assert tracker.consume(tensor) == _State(implementation_id="candidate") + assert tracker.consume(tensor) is None diff --git a/inference_models/tests/unit_tests/models/rfdetr/test_optimization_contracts.py b/inference_models/tests/unit_tests/models/rfdetr/test_optimization_contracts.py new file mode 100644 index 0000000000..c7dedae93b --- /dev/null +++ b/inference_models/tests/unit_tests/models/rfdetr/test_optimization_contracts.py @@ -0,0 +1,446 @@ +import json +from dataclasses import FrozenInstanceError + +import numpy as np +import pytest +import torch + +from inference_models.errors import ModelRuntimeError +from inference_models.models.common.roboflow.model_packages import ( + ColorMode, + ImagePreProcessing, + NetworkInputDefinition, + ResizeMode, + TrainingInputSize, +) +from inference_models.models.optimization.contracts import ( + CompatibilityResult, + DeviceCompatibility, + ExecutionContext, + InputCompatibility, + OptimizationMetadata, + OptimizationStage, + ValidationEnvironment, + immutable_mapping, +) +from inference_models.models.optimization.registry import ImplementationRegistry +from inference_models.models.rfdetr.optimization.catalog import ( + RFDETR_POSTPROCESSOR_IMPLEMENTATIONS, + RFDETR_PREPROCESSOR_IMPLEMENTATIONS, +) +from inference_models.models.rfdetr.optimization.contracts import ( + PostprocessRequest, + PreprocessRequest, +) +from inference_models.models.rfdetr.optimization.execution_plan import ( + RFDetrExecutionPlan, +) +from inference_models.models.rfdetr.optimization.ids import ( + RFDETR_POSTPROCESSOR_BASE, + RFDETR_POSTPROCESSOR_TRITON_FUSED_V1, + RFDETR_PREPROCESSOR_BASE, + RFDETR_PREPROCESSOR_TRITON_UNIVERSAL_V1, +) +from inference_models.models.rfdetr.optimization.readiness import ( + PreprocessReadinessTracker, +) +from inference_models.models.rfdetr.optimization.selection import ( + resolve_postprocessor_for_request, + resolve_preprocessor_for_model, + resolve_preprocessor_for_request, +) + + +class _Stage: + def __init__( + self, + implementation_id: str, + *, + compatible: bool = True, + validated: bool = False, + model_supported: bool = True, + request_supported: bool = True, + stage: OptimizationStage = OptimizationStage.PREPROCESS, + ) -> None: + validation_environments = ( + ( + ValidationEnvironment( + machine_type="test", + device_kind="gpu", + device_name="test-gpu", + scenario="runtime", + resolved_axes={}, + runtime_versions={}, + source_commit="test", + profiling_bundle="test-bundle", + status="validated", + ), + ) + if validated + else () + ) + self.metadata = OptimizationMetadata( + implementation_id=implementation_id, + stage=stage, + version="1", + target=DeviceCompatibility(device_kind="gpu"), + inputs=InputCompatibility(scenarios=("*",)), + dependencies=(), + fallback_id="base", + changes_numerics=False, + supports_concurrency=True, + supports_cuda_graphs=False, + validated_environments=validation_environments, + ) + self._compatible = compatible + self._model_supported = model_supported + self._request_supported = request_supported + + def is_compatible(self, context: ExecutionContext) -> bool: + return self._compatible + + def check_model_compatibility( + self, + *, + image_pre_processing: ImagePreProcessing, + network_input: NetworkInputDefinition, + ) -> CompatibilityResult: + del image_pre_processing, network_input + if self._model_supported: + return CompatibilityResult.compatible() + + return CompatibilityResult.incompatible("static crop") + + def check_request_compatibility( + self, + *, + request: PreprocessRequest, + context: ExecutionContext, + ) -> CompatibilityResult: + del request, context + if self._request_supported: + return CompatibilityResult.compatible() + + return CompatibilityResult.incompatible("heterogeneous source dimensions") + + +def _context() -> ExecutionContext: + return ExecutionContext( + device_kind="gpu", + device="cuda:0", + device_name="test-gpu", + machine_type="test", + scenario="runtime", + ) + + +def _network_input() -> NetworkInputDefinition: + return NetworkInputDefinition( + training_input_size=TrainingInputSize(height=64, width=64), + dataset_version_resize_dimensions=None, + dynamic_spatial_size_supported=False, + color_mode=ColorMode.RGB, + resize_mode=ResizeMode.STRETCH_TO, + input_channels=3, + scaling_factor=255, + normalization=[[0.0, 0.0, 0.0], [1.0, 1.0, 1.0]], + ) + + +def test_execution_plan_defaults_to_optimized_implementations(monkeypatch) -> None: + monkeypatch.delenv("INFERENCE_MODELS_RFDETR_PREPROCESSOR", raising=False) + monkeypatch.delenv("INFERENCE_MODELS_RFDETR_POSTPROCESSOR", raising=False) + + default_plan = RFDetrExecutionPlan() + resolved_plan = RFDetrExecutionPlan.resolve() + + for plan in (default_plan, resolved_plan): + assert plan.preprocessor_id == RFDETR_PREPROCESSOR_TRITON_UNIVERSAL_V1 + assert plan.postprocessor_id == RFDETR_POSTPROCESSOR_TRITON_FUSED_V1 + + +def test_execution_plan_reads_environment_overrides(monkeypatch) -> None: + monkeypatch.setenv( + "INFERENCE_MODELS_RFDETR_PREPROCESSOR", + RFDETR_PREPROCESSOR_BASE, + ) + monkeypatch.setenv( + "INFERENCE_MODELS_RFDETR_POSTPROCESSOR", + RFDETR_POSTPROCESSOR_BASE, + ) + + plan = RFDetrExecutionPlan.resolve() + + assert plan.preprocessor_id == RFDETR_PREPROCESSOR_BASE + assert plan.postprocessor_id == RFDETR_POSTPROCESSOR_BASE + + +def test_explicit_plan_ignores_environment(monkeypatch) -> None: + monkeypatch.setenv( + "INFERENCE_MODELS_RFDETR_PREPROCESSOR", + RFDETR_PREPROCESSOR_TRITON_UNIVERSAL_V1, + ) + plan = RFDetrExecutionPlan( + preprocessor_id=RFDETR_PREPROCESSOR_BASE, + postprocessor_id=RFDETR_POSTPROCESSOR_BASE, + ) + + resolved = RFDetrExecutionPlan.resolve(execution_plan=plan) + + assert resolved is plan + + +def test_execution_plan_rejects_unimplemented_stage_category() -> None: + with pytest.raises(ModelRuntimeError, match="does not yet provide"): + RFDetrExecutionPlan.resolve( + execution_plan=RFDetrExecutionPlan(scheduler_id="future-scheduler") + ) + + +def test_registry_resolves_explicit_and_auto_base() -> None: + registry = ImplementationRegistry(scope_name="RF-DETR") + base = _Stage("base") + candidate = _Stage("candidate") + registry.register(base) + registry.register(candidate) + + assert ( + registry.resolve( + stage=OptimizationStage.PREPROCESS, + requested_id="candidate", + context=_context(), + ) + is candidate + ) + assert ( + registry.resolve( + stage=OptimizationStage.PREPROCESS, + requested_id="auto", + context=_context(), + ) + is base + ) + + +def test_registry_auto_selects_a_validated_compatible_candidate() -> None: + registry = ImplementationRegistry(scope_name="RF-DETR") + base = _Stage("base") + candidate = _Stage("candidate", validated=True) + registry.register(base) + registry.register(candidate) + + assert ( + registry.resolve( + stage=OptimizationStage.PREPROCESS, + requested_id="auto", + context=_context(), + ) + is candidate + ) + + +def test_registry_rejects_unknown_and_incompatible_explicit_selection() -> None: + registry = ImplementationRegistry(scope_name="RF-DETR") + registry.register(_Stage("base")) + registry.register(_Stage("incompatible", compatible=False)) + + with pytest.raises(ModelRuntimeError, match="Unknown RF-DETR preprocess"): + registry.resolve( + stage=OptimizationStage.PREPROCESS, + requested_id="unknown", + context=_context(), + ) + with pytest.raises(ModelRuntimeError, match="not compatible"): + registry.resolve( + stage=OptimizationStage.PREPROCESS, + requested_id="incompatible", + context=_context(), + ) + + +def test_model_contract_incompatibility_resolves_declared_base_fallback() -> None: + registry = ImplementationRegistry(scope_name="RF-DETR") + base = _Stage("base") + candidate = _Stage("candidate", model_supported=False) + registry.register(base) + registry.register(candidate) + + selection = resolve_preprocessor_for_model( + registry=registry, + requested_id="candidate", + context=_context(), + image_pre_processing=ImagePreProcessing(), + network_input=_network_input(), + allow_fallback=True, + ) + + assert selection.implementation is base + assert selection.effective_id == "base" + assert selection.fallback_reason == "static crop" + + +def test_auto_selection_is_not_reported_as_fallback_when_candidate_is_supported() -> ( + None +): + registry = ImplementationRegistry(scope_name="RF-DETR") + registry.register(_Stage("base")) + candidate = _Stage("candidate", validated=True) + registry.register(candidate) + + selection = resolve_preprocessor_for_model( + registry=registry, + requested_id="auto", + context=_context(), + image_pre_processing=ImagePreProcessing(), + network_input=_network_input(), + allow_fallback=True, + ) + + assert selection.implementation is candidate + assert selection.effective_id == "candidate" + assert not selection.used_fallback + + +def test_request_incompatibility_resolves_declared_base_fallback() -> None: + registry = ImplementationRegistry(scope_name="RF-DETR") + base = _Stage("base") + candidate = _Stage("candidate", request_supported=False) + registry.register(base) + registry.register(candidate) + request = PreprocessRequest( + images=np.zeros((8, 9, 3), dtype=np.uint8), + input_color_format=ColorMode.RGB, + image_pre_processing=ImagePreProcessing(), + network_input=_network_input(), + pre_processing_overrides=None, + ) + + selection = resolve_preprocessor_for_request( + registry=registry, + implementation=candidate, + request=request, + context=_context(), + allow_fallback=True, + ) + + assert selection.implementation is base + assert selection.effective_id == "base" + assert selection.fallback_reason == "heterogeneous source dimensions" + + +def test_fallback_is_rejected_when_base_is_also_incompatible() -> None: + registry = ImplementationRegistry(scope_name="RF-DETR") + registry.register(_Stage("base", model_supported=False)) + registry.register(_Stage("candidate", model_supported=False)) + + with pytest.raises(ModelRuntimeError, match="Fallback 'base' is unsupported"): + resolve_preprocessor_for_model( + registry=registry, + requested_id="candidate", + context=_context(), + image_pre_processing=ImagePreProcessing(), + network_input=_network_input(), + allow_fallback=True, + ) + + +def test_strict_plan_rejects_compatibility_fallback() -> None: + registry = ImplementationRegistry(scope_name="RF-DETR") + registry.register(_Stage("base")) + registry.register(_Stage("candidate", model_supported=False)) + + with pytest.raises(ModelRuntimeError, match="fallback is disabled"): + resolve_preprocessor_for_model( + registry=registry, + requested_id="candidate", + context=_context(), + image_pre_processing=ImagePreProcessing(), + network_input=_network_input(), + allow_fallback=False, + ) + + +def test_postprocessor_contract_uses_same_declared_fallback_policy() -> None: + registry = ImplementationRegistry(scope_name="RF-DETR") + base = _Stage("base", stage=OptimizationStage.POSTPROCESS) + candidate = _Stage( + "candidate", + request_supported=False, + stage=OptimizationStage.POSTPROCESS, + ) + registry.register(base) + registry.register(candidate) + request = PostprocessRequest( + bboxes=torch.zeros((1, 2, 4)), + logits=torch.zeros((1, 2, 3)), + pre_processing_meta=[], + threshold=0.5, + num_classes=3, + classes_re_mapping=None, + ) + + selection = resolve_postprocessor_for_request( + registry=registry, + implementation=candidate, + request=request, + context=_context(), + allow_fallback=True, + ) + + assert selection.implementation is base + assert selection.effective_id == "base" + assert selection.fallback_reason == "heterogeneous source dimensions" + + with pytest.raises(ModelRuntimeError, match="fallback is disabled"): + resolve_postprocessor_for_request( + registry=registry, + implementation=candidate, + request=request, + context=_context(), + allow_fallback=False, + ) + + +def test_implementation_metadata_is_typed_and_immutable() -> None: + preprocessor = RFDETR_PREPROCESSOR_IMPLEMENTATIONS[ + RFDETR_PREPROCESSOR_TRITON_UNIVERSAL_V1 + ] + postprocessor = RFDETR_POSTPROCESSOR_IMPLEMENTATIONS[ + RFDETR_POSTPROCESSOR_TRITON_FUSED_V1 + ] + + assert preprocessor.stage is OptimizationStage.PREPROCESS + assert postprocessor.stage is OptimizationStage.POSTPROCESS + assert preprocessor.inputs.axis_constraints["channels"] == 3 + with pytest.raises(FrozenInstanceError): + preprocessor.version = "2" + with pytest.raises(TypeError): + preprocessor.inputs.axis_constraints["channels"] = 4 + assert json.loads(json.dumps(preprocessor.to_dict()))["stage"] == "preprocess" + + +def test_readiness_tracker_consumes_only_the_exact_tensor() -> None: + tracker = PreprocessReadinessTracker() + tensor = torch.zeros(1) + other = torch.zeros(1) + tracker.record( + tensor, + ready_event=None, + input_kind="test", + implementation_id="candidate", + ) + + assert tracker.consume(other) is None + readiness = tracker.consume(tensor) + assert readiness is not None + assert readiness.input_kind == "test" + assert readiness.implementation_id == "candidate" + assert tracker.consume(tensor) is None + + +def test_immutable_mapping_detaches_from_source() -> None: + source = {"batch": 1} + immutable = immutable_mapping(source) + source["batch"] = 2 + + assert immutable["batch"] == 1 diff --git a/inference_models/tests/unit_tests/models/rfdetr/test_pre_processing.py b/inference_models/tests/unit_tests/models/rfdetr/test_pre_processing.py index 2bc433814c..ca48ee8b37 100644 --- a/inference_models/tests/unit_tests/models/rfdetr/test_pre_processing.py +++ b/inference_models/tests/unit_tests/models/rfdetr/test_pre_processing.py @@ -148,6 +148,34 @@ def test_one_step_stretch_tensor_matches_reference_pipeline() -> None: torch.testing.assert_close(actual_tensor, expected, atol=1e-6, rtol=0) +def test_shared_preprocessor_does_not_consume_trt_selection_environment( + monkeypatch, +) -> None: + monkeypatch.setenv( + "INFERENCE_MODELS_RFDETR_PREPROCESSOR", + "triton-universal-v1", + ) + image_pre_processing = ImagePreProcessing() + network_input = _build_network_input( + training_h=16, + training_w=16, + resize_mode=ResizeMode.STRETCH_TO, + dataset_version_dims=None, + ) + image = np.zeros((16, 16, 3), dtype=np.uint8) + + actual_tensor, _ = pre_process_network_input( + images=image, + image_pre_processing=image_pre_processing, + network_input=network_input, + target_device=torch.device("cpu"), + input_color_format="rgb", + ) + + expected = _reference_pipeline(Image.fromarray(image), target_h=16, target_w=16) + torch.testing.assert_close(actual_tensor, expected, atol=0, rtol=0) + + def test_one_step_stretch_with_bgr_input_matches_rgb_reference() -> None: """When caller passes BGR data with input_color_format='bgr', preprocessor swaps to RGB and the resulting tensor matches the RGB reference exactly.""" diff --git a/inference_models/tests/unit_tests/models/rfdetr/test_triton_object_detection_postprocess.py b/inference_models/tests/unit_tests/models/rfdetr/test_triton_object_detection_postprocess.py new file mode 100644 index 0000000000..5833631adb --- /dev/null +++ b/inference_models/tests/unit_tests/models/rfdetr/test_triton_object_detection_postprocess.py @@ -0,0 +1,205 @@ +import pytest +import torch + +from inference_models.entities import ImageDimensions +from inference_models.errors import ModelRuntimeError +from inference_models.models.common.roboflow.model_packages import ( + PreProcessingMetadata, + StaticCropOffset, +) +from inference_models.models.rfdetr.class_remapping import ClassesReMapping +from inference_models.models.rfdetr.common import post_process_object_detection_results +from inference_models.models.rfdetr.optimization.catalog import ( + RFDETR_POSTPROCESSOR_IMPLEMENTATIONS, +) +from inference_models.models.rfdetr.optimization.ids import ( + RFDETR_POSTPROCESSOR_TRITON_FUSED_V1, +) +from inference_models.models.rfdetr.triton_object_detection_postprocess import ( + TRITON_AVAILABLE, + FusedObjectDetectionPostprocessor, + _metadata_values, +) + + +def _metadata( + source_height: int = 480, + source_width: int = 640, + target_height: int = 512, + target_width: int = 512, +) -> PreProcessingMetadata: + return PreProcessingMetadata( + pad_left=0, + pad_top=0, + pad_right=0, + pad_bottom=0, + original_size=ImageDimensions(width=source_width, height=source_height), + size_after_pre_processing=ImageDimensions( + width=source_width, height=source_height + ), + inference_size=ImageDimensions(width=target_width, height=target_height), + scale_width=target_width / source_width, + scale_height=target_height / source_height, + static_crop_offset=StaticCropOffset( + offset_x=0, + offset_y=0, + crop_width=source_width, + crop_height=source_height, + ), + ) + + +def test_fused_postprocessor_is_explicitly_selectable() -> None: + metadata = RFDETR_POSTPROCESSOR_IMPLEMENTATIONS[ + RFDETR_POSTPROCESSOR_TRITON_FUSED_V1 + ] + assert metadata.validated_environments == () + + +def test_fused_postprocessor_requires_cuda() -> None: + with pytest.raises(ModelRuntimeError, match="requires a CUDA target"): + FusedObjectDetectionPostprocessor(torch.device("cpu")) + + +def test_fused_postprocessor_reports_request_incompatibility_without_cuda() -> None: + postprocessor = FusedObjectDetectionPostprocessor.__new__( + FusedObjectDetectionPostprocessor + ) + postprocessor._device = torch.device("cuda:0") + + compatibility = postprocessor.check_request_compatibility( + bboxes=torch.zeros((1, 2, 4)), + logits=torch.zeros((1, 2, 3)), + pre_processing_meta=[_metadata()], + threshold=0.5, + num_classes=3, + classes_re_mapping=None, + ) + + assert not compatibility.supported + assert "boxes and logits must be CUDA tensors" in compatibility.reasons + + +def test_fused_postprocessor_reports_missing_triton(monkeypatch) -> None: + monkeypatch.setattr( + "inference_models.models.rfdetr.triton_object_detection_postprocess." + "TRITON_AVAILABLE", + False, + ) + postprocessor = FusedObjectDetectionPostprocessor.__new__( + FusedObjectDetectionPostprocessor + ) + postprocessor._device = torch.device("cuda:0") + + compatibility = postprocessor.check_request_compatibility( + bboxes=torch.zeros((1, 2, 4)), + logits=torch.zeros((1, 2, 3)), + pre_processing_meta=[_metadata()], + threshold=0.5, + num_classes=3, + classes_re_mapping=None, + ) + + assert not compatibility.supported + assert "Triton is not installed" in compatibility.reasons + + +def test_fused_postprocessor_direct_call_remains_strict_for_incompatibility() -> None: + postprocessor = FusedObjectDetectionPostprocessor.__new__( + FusedObjectDetectionPostprocessor + ) + postprocessor._device = torch.device("cuda:0") + + with pytest.raises(ModelRuntimeError, match="cannot preserve this.*contract"): + postprocessor._validate_inputs( + bboxes=torch.zeros((1, 2, 4)), + logits=torch.zeros((1, 2, 3)), + pre_processing_meta=[_metadata()], + threshold=0.5, + num_classes=3, + classes_re_mapping=None, + ) + + +def test_metadata_values_match_reference_transform_parameters() -> None: + values = _metadata_values(_metadata()) + + assert values == ( + 512.0, + 512.0, + 0.0, + 0.0, + 0.8, + 512 / 480, + 0.0, + 0.0, + 640.0, + 480.0, + ) + + +@pytest.mark.skipif( + not torch.cuda.is_available() or not TRITON_AVAILABLE, + reason="CUDA and Triton are required", +) +@pytest.mark.parametrize("per_class_threshold", [False, True]) +@pytest.mark.parametrize("with_class_remapping", [False, True]) +def test_fused_postprocessor_matches_reference( + per_class_threshold: bool, + with_class_remapping: bool, +) -> None: + device = torch.device("cuda:0") + generator = torch.Generator(device=device).manual_seed(123) + batch_size, num_queries, logits_classes = 4, 17, 6 + bboxes = torch.rand( + (batch_size, num_queries, 4), device=device, generator=generator + ) + logits = torch.randn( + (batch_size, num_queries, logits_classes), + device=device, + generator=generator, + ) + metadata = [_metadata() for _ in range(batch_size)] + + if with_class_remapping: + class_mapping = torch.tensor([0, -1, 1, 2, 3, -1], device=device) + remapping = ClassesReMapping( + remaining_class_ids=torch.tensor([0, 2, 3, 4], device=device), + class_mapping=class_mapping, + ) + num_classes = 4 + else: + remapping = None + num_classes = logits_classes - 1 + + threshold = torch.linspace(0.1, 0.5, num_classes) if per_class_threshold else 0.25 + expected = post_process_object_detection_results( + bboxes=bboxes, + logits=logits, + pre_processing_meta=metadata, + threshold=threshold, + num_classes=num_classes, + classes_re_mapping=remapping, + device=device, + ) + stream = torch.cuda.Stream(device=device) + actual = FusedObjectDetectionPostprocessor(device).postprocess( + bboxes=bboxes, + logits=logits, + pre_processing_meta=metadata, + threshold=threshold, + num_classes=num_classes, + classes_re_mapping=remapping, + stream=stream, + ) + stream.synchronize() + + for actual_image, expected_image in zip(actual, expected): + torch.testing.assert_close(actual_image.xyxy, expected_image.xyxy) + torch.testing.assert_close(actual_image.class_id, expected_image.class_id) + torch.testing.assert_close( + actual_image.confidence, + expected_image.confidence, + rtol=0, + atol=0, + ) diff --git a/inference_models/tests/unit_tests/models/rfdetr/test_triton_universal_preprocess_runtime.py b/inference_models/tests/unit_tests/models/rfdetr/test_triton_universal_preprocess_runtime.py new file mode 100644 index 0000000000..470a203980 --- /dev/null +++ b/inference_models/tests/unit_tests/models/rfdetr/test_triton_universal_preprocess_runtime.py @@ -0,0 +1,326 @@ +import numpy as np +import pytest +import torch + +from inference_models import PreProcessingOverrides +from inference_models.errors import ModelRuntimeError +from inference_models.models.common.roboflow.model_packages import ( + ColorMode, + Contrast, + ContrastType, + Grayscale, + ImagePreProcessing, + NetworkInputDefinition, + ResizeMode, + StaticCrop, + TrainingInputSize, +) +from inference_models.models.rfdetr.optimization.catalog import ( + RFDETR_PREPROCESSOR_IMPLEMENTATIONS, +) +from inference_models.models.rfdetr.optimization.ids import ( + RFDETR_PREPROCESSOR_TRITON_UNIVERSAL_V1, +) +from inference_models.models.rfdetr.pre_processing import ( + resolve_rfdetr_preprocessor_max_workers, +) +from inference_models.models.rfdetr.triton_universal_preprocess_runtime import ( + UniversalFastPreprocessRuntime, + _build_metadata_batch, + _canonicalize_batch, +) + +_IMAGENET_MEAN = (0.485, 0.456, 0.406) +_IMAGENET_STD = (0.229, 0.224, 0.225) + + +def _network_input( + *, + resize_mode: ResizeMode = ResizeMode.STRETCH_TO, +) -> NetworkInputDefinition: + return NetworkInputDefinition( + training_input_size=TrainingInputSize(height=64, width=64), + dataset_version_resize_dimensions=None, + dynamic_spatial_size_supported=False, + color_mode=ColorMode.RGB, + resize_mode=resize_mode, + input_channels=3, + scaling_factor=255, + normalization=[list(_IMAGENET_MEAN), list(_IMAGENET_STD)], + ) + + +@pytest.mark.parametrize( + "images", + [ + np.zeros((2, 8, 9, 3), dtype=np.uint8), + [np.zeros((8, 9, 3), dtype=np.uint8) for _ in range(2)], + torch.zeros((2, 3, 8, 9), dtype=torch.uint8), + torch.zeros((2, 8, 9, 3), dtype=torch.uint8), + ], +) +def test_canonicalize_uint8_cpu_batches(images) -> None: + batch = _canonicalize_batch(images) + + assert batch.kind == "uint8" + assert (batch.height, batch.width) == (8, 9) + assert len(batch.items) == 2 + assert all(tuple(item.shape) == (8, 9, 3) for item in batch.items) + + +@pytest.mark.parametrize( + "images", + [ + torch.zeros((2, 3, 8, 9), dtype=torch.float32), + torch.zeros((2, 8, 9, 3), dtype=torch.float16), + ], +) +def test_canonicalize_float_cpu_batches(images) -> None: + batch = _canonicalize_batch(images) + + assert batch.kind == "float" + assert (batch.height, batch.width) == (8, 9) + assert all(tuple(item.shape) == (3, 8, 9) for item in batch.items) + + +def test_canonicalize_float_numpy_matches_reference_uint8_conversion() -> None: + image = np.array([[[0.0, 0.5, 1.5]]], dtype=np.float32) + + batch = _canonicalize_batch(image) + + np.testing.assert_array_equal( + batch.items[0], + np.array([[[0, 127, 255]]], dtype=np.uint8), + ) + + +def test_canonicalize_rejects_mixed_semantics() -> None: + with pytest.raises(ModelRuntimeError, match="homogeneous batch"): + _canonicalize_batch( + [ + torch.zeros((3, 8, 9), dtype=torch.uint8), + torch.zeros((3, 8, 9), dtype=torch.float32), + ] + ) + + +def test_canonicalize_rejects_mixed_source_dimensions() -> None: + with pytest.raises(ModelRuntimeError, match="equal source dimensions"): + _canonicalize_batch( + [ + np.zeros((8, 9, 3), dtype=np.uint8), + np.zeros((10, 9, 3), dtype=np.uint8), + ] + ) + + +def test_metadata_batch_describes_stretch() -> None: + metadata = _build_metadata_batch( + batch_size=2, + source_h=8, + source_w=10, + target_h=16, + target_w=40, + ) + + assert len(metadata) == 2 + assert metadata[0].scale_height == 2 + assert metadata[0].scale_width == 4 + assert metadata[0].static_crop_offset.crop_width == 10 + assert metadata[0].static_crop_offset.crop_height == 8 + + +def test_universal_candidate_is_explicitly_selectable() -> None: + metadata = RFDETR_PREPROCESSOR_IMPLEMENTATIONS[ + RFDETR_PREPROCESSOR_TRITON_UNIVERSAL_V1 + ] + assert metadata.validated_environments == () + assert metadata.fallback_id == "base" + + +@pytest.mark.parametrize( + ("image_pre_processing", "reason"), + [ + ( + ImagePreProcessing.model_validate( + { + "static-crop": StaticCrop( + enabled=True, + x_min=10, + x_max=90, + y_min=10, + y_max=90, + ) + } + ), + "static crop", + ), + (ImagePreProcessing(grayscale=Grayscale(enabled=True)), "grayscale"), + ( + ImagePreProcessing( + contrast=Contrast( + enabled=True, + type=ContrastType.CONTRAST_STRETCHING, + ) + ), + "contrast", + ), + ], +) +def test_model_compatibility_reports_base_supported_transformations( + image_pre_processing: ImagePreProcessing, + reason: str, +) -> None: + compatibility = UniversalFastPreprocessRuntime.check_model_compatibility( + image_pre_processing=image_pre_processing, + network_input=_network_input(), + ) + + assert not compatibility.supported + assert reason in compatibility.reasons + + +def test_model_compatibility_reports_non_stretch_resize() -> None: + compatibility = UniversalFastPreprocessRuntime.check_model_compatibility( + image_pre_processing=ImagePreProcessing(), + network_input=_network_input(resize_mode=ResizeMode.LETTERBOX), + ) + + assert not compatibility.supported + assert any(reason.startswith("resize_mode=") for reason in compatibility.reasons) + + +def test_request_compatibility_reports_active_overrides_and_heterogeneous_shapes() -> ( + None +): + compatibility = UniversalFastPreprocessRuntime.check_request_compatibility( + images=[ + np.zeros((8, 9, 3), dtype=np.uint8), + np.zeros((10, 9, 3), dtype=np.uint8), + ], + pre_processing_overrides=PreProcessingOverrides(disable_static_crop=True), + ) + + assert not compatibility.supported + assert "active pre-processing overrides" in compatibility.reasons + assert any( + reason.startswith("heterogeneous source dimensions") + for reason in compatibility.reasons + ) + + +def test_request_compatibility_accepts_no_op_overrides(monkeypatch) -> None: + monkeypatch.setattr( + "inference_models.models.rfdetr.triton_universal_preprocess_runtime." + "TRITON_AVAILABLE", + True, + ) + + compatibility = UniversalFastPreprocessRuntime.check_request_compatibility( + images=np.zeros((8, 9, 3), dtype=np.uint8), + pre_processing_overrides=PreProcessingOverrides(), + ) + + assert compatibility.supported + + +@pytest.mark.parametrize( + "pre_processing_overrides", + [ + PreProcessingOverrides(disable_contrast_enhancement=True), + PreProcessingOverrides(disable_grayscale=True), + PreProcessingOverrides(disable_static_crop=True), + ], +) +def test_request_compatibility_rejects_each_active_override( + monkeypatch, + pre_processing_overrides: PreProcessingOverrides, +) -> None: + monkeypatch.setattr( + "inference_models.models.rfdetr.triton_universal_preprocess_runtime." + "TRITON_AVAILABLE", + True, + ) + + compatibility = UniversalFastPreprocessRuntime.check_request_compatibility( + images=np.zeros((8, 9, 3), dtype=np.uint8), + pre_processing_overrides=pre_processing_overrides, + ) + + assert not compatibility.supported + assert "active pre-processing overrides" in compatibility.reasons + + +def test_supported_uint8_request_is_compatible_when_triton_is_available( + monkeypatch, +) -> None: + monkeypatch.setattr( + "inference_models.models.rfdetr.triton_universal_preprocess_runtime." + "TRITON_AVAILABLE", + True, + ) + compatibility = UniversalFastPreprocessRuntime.check_request_compatibility( + images=np.zeros((2, 8, 9, 3), dtype=np.uint8), + pre_processing_overrides=None, + ) + + assert compatibility.supported + + +def test_uint8_request_reports_missing_triton(monkeypatch) -> None: + monkeypatch.setattr( + "inference_models.models.rfdetr.triton_universal_preprocess_runtime." + "TRITON_AVAILABLE", + False, + ) + + compatibility = UniversalFastPreprocessRuntime.check_request_compatibility( + images=np.zeros((8, 9, 3), dtype=np.uint8), + pre_processing_overrides=None, + ) + + assert not compatibility.supported + assert "Triton is not installed for uint8 preprocessing" in compatibility.reasons + + +def test_float_request_remains_compatible_without_triton(monkeypatch) -> None: + monkeypatch.setattr( + "inference_models.models.rfdetr.triton_universal_preprocess_runtime." + "TRITON_AVAILABLE", + False, + ) + + compatibility = UniversalFastPreprocessRuntime.check_request_compatibility( + images=torch.zeros((3, 8, 9), dtype=torch.float32), + pre_processing_overrides=None, + ) + + assert compatibility.supported + + +def test_preprocessor_worker_limit_can_be_selected_from_environment( + monkeypatch, +) -> None: + monkeypatch.setenv("INFERENCE_MODELS_RFDETR_PREPROCESSOR_MAX_WORKERS", "7") + + assert resolve_rfdetr_preprocessor_max_workers() == 7 + + +def test_explicit_preprocessor_worker_limit_overrides_environment(monkeypatch) -> None: + monkeypatch.setenv("INFERENCE_MODELS_RFDETR_PREPROCESSOR_MAX_WORKERS", "7") + + assert resolve_rfdetr_preprocessor_max_workers(2) == 2 + + +def test_preprocessor_worker_limit_rejects_non_positive_environment_value( + monkeypatch, +) -> None: + monkeypatch.setenv("INFERENCE_MODELS_RFDETR_PREPROCESSOR_MAX_WORKERS", "0") + + with pytest.raises(ModelRuntimeError, match="must be at least 1"): + resolve_rfdetr_preprocessor_max_workers() + + +def test_universal_runtime_requires_cuda_device() -> None: + with pytest.raises(ModelRuntimeError, match="requires a CUDA target"): + UniversalFastPreprocessRuntime(device=torch.device("cpu")) diff --git a/inference_models/uv.lock b/inference_models/uv.lock index c4474aa9ff..3a27640102 100644 --- a/inference_models/uv.lock +++ b/inference_models/uv.lock @@ -1,5 +1,5 @@ version = 1 -revision = 2 +revision = 3 requires-python = ">=3.10, <3.13" resolution-markers = [ "python_full_version >= '3.12' and platform_machine != 's390x' and sys_platform == 'darwin'", @@ -139,7 +139,7 @@ name = "anyio" version = "4.12.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "exceptiongroup", marker = "python_full_version < '3.11' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126')" }, + { name = "exceptiongroup", marker = "python_full_version < '3.11' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, { name = "idna" }, { name = "typing-extensions" }, ] @@ -390,7 +390,7 @@ name = "click" version = "8.3.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "colorama", marker = "sys_platform == 'win32' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126')" }, + { name = "colorama", marker = "sys_platform == 'win32' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/3d/fa/656b739db8587d7b5dfa22e22ed02566950fbfbcdc20311993483657a5c0/click-8.3.1.tar.gz", hash = "sha256:12ff4785d337a1bb490bb7e9c2b1ee5da3112e94a8622f26a6c77f5d2fc6842a", size = 295065, upload-time = "2025-11-15T20:45:42.706Z" } wheels = [ @@ -411,7 +411,7 @@ name = "coloredlogs" version = "15.0.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "humanfriendly", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, + { name = "humanfriendly", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (platform_machine != 'aarch64' and extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (sys_platform != 'linux' and extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126') or extra == 'extra-16-inference-models-onnx-cpu' or extra == 'extra-16-inference-models-onnx-cu118' or extra == 'extra-16-inference-models-onnx-cu12' or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/cc/c7/eed8f27100517e8c0e6b923d5f0845d0cb99763da6fdee00478f91db7325/coloredlogs-15.0.1.tar.gz", hash = "sha256:7c991aa71a4577af2f82600d8f8f3a89f936baeaf9b50a9c197da014e5bf16b0", size = 278520, upload-time = "2021-06-11T10:22:45.202Z" } wheels = [ @@ -590,7 +590,7 @@ name = "exceptiongroup" version = "1.3.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "typing-extensions", marker = "python_full_version < '3.11' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126')" }, + { name = "typing-extensions", marker = "python_full_version < '3.11' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/0b/9f/a65090624ecf468cdca03533906e7c69ed7588582240cfe7cc9e770b50eb/exceptiongroup-1.3.0.tar.gz", hash = "sha256:b241f5885f560bc56a59ee63ca4c6a8bfa46ae4ad651af316d4e81817bb9fd88", size = 29749, upload-time = "2025-05-10T17:42:51.123Z" } wheels = [ @@ -829,7 +829,7 @@ source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "filelock" }, { name = "fsspec" }, - { name = "hf-xet", marker = "platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126')" }, + { name = "hf-xet", marker = "platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, { name = "httpx" }, { name = "packaging" }, { name = "pyyaml" }, @@ -913,7 +913,7 @@ wheels = [ [[package]] name = "inference-models" -version = "0.31.0" +version = "0.32.0" source = { virtual = "." } dependencies = [ { name = "accelerate" }, @@ -1896,7 +1896,7 @@ resolution-markers = [ "(python_full_version < '3.11' and platform_machine != 'aarch64' and platform_machine != 's390x' and sys_platform == 'linux') or (python_full_version < '3.11' and platform_machine != 's390x' and sys_platform != 'darwin' and sys_platform != 'linux')", ] dependencies = [ - { name = "numpy", marker = "platform_machine != 's390x' or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, + { name = "numpy", marker = "platform_machine != 's390x' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/32/49/6e67c334872d2c114df3020e579f3718c333198f8312290e09ec0216703a/ml_dtypes-0.5.1.tar.gz", hash = "sha256:ac5b58559bb84a95848ed6984eb8013249f90b6bab62aa5acbad876e256002c9", size = 698772, upload-time = "2025-01-07T03:34:55.613Z" } wheels = [ @@ -1927,7 +1927,7 @@ resolution-markers = [ "python_full_version < '3.11' and platform_machine == 's390x' and sys_platform != 'darwin'", ] dependencies = [ - { name = "numpy", marker = "platform_machine == 's390x' or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, + { name = "numpy", marker = "platform_machine == 's390x' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/0e/4a/c27b42ed9b1c7d13d9ba8b6905dece787d6259152f2309338aed29b2447b/ml_dtypes-0.5.4.tar.gz", hash = "sha256:8ab06a50fb9bf9666dd0fe5dfb4676fa2b0ac0f31ecff72a6c3af8e22c063453", size = 692314, upload-time = "2025-11-17T22:32:31.031Z" } wheels = [ @@ -3032,15 +3032,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/bc/60/5382c03e1970de634027cee8e1b7d39776b778b81812aaf45b694dfe9e28/pillow-12.2.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:bfa9c230d2fe991bed5318a5f119bd6780cda2915cca595393649fc118ab895e", size = 7080946, upload-time = "2026-04-01T14:46:11.734Z" }, ] -[[package]] -name = "pkgconfig" -version = "1.6.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/52/fd/0adde075cd3bfecd557bc7d757e00e231d34d8a6edb4c8d1642759254c21/pkgconfig-1.6.0.tar.gz", hash = "sha256:4a5a6631ce937fafac457104a40d558785a658bbdca5c49b6295bc3fd651907f", size = 5691, upload-time = "2026-03-06T11:26:01.194Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/75/6f/f7ec07fba48f07c555cc4099481df644fbbc12067879072c17ac229f6556/pkgconfig-1.6.0-py3-none-any.whl", hash = "sha256:98e71754855e9563838d952a160eb577edabb57782e49853edb5381927e6bea1", size = 7086, upload-time = "2026-03-06T11:26:07.688Z" }, -] - [[package]] name = "platformdirs" version = "4.3.8" @@ -3064,7 +3055,7 @@ name = "portalocker" version = "3.2.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "pywin32", marker = "sys_platform == 'win32' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126')" }, + { name = "pywin32", marker = "sys_platform == 'win32' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/5e/77/65b857a69ed876e1951e88aaba60f5ce6120c33703f7cb61a3c894b8c1b6/portalocker-3.2.0.tar.gz", hash = "sha256:1f3002956a54a8c3730586c5c77bf18fae4149e07eaf1c29fc3faf4d5a3f89ac", size = 95644, upload-time = "2025-06-14T13:20:40.03Z" } wheels = [ @@ -3548,7 +3539,6 @@ version = "2.2.3" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "cffi" }, - { name = "pkgconfig" }, ] sdist = { url = "https://files.pythonhosted.org/packages/7b/88/f73dae807ec68b228fba72507105e3ba80a561dc0bade0004ce24fd118fc/pyvips-2.2.3.tar.gz", hash = "sha256:43bceced0db492654c93008246a58a508e0373ae1621116b87b322f2ac72212f", size = 56626, upload-time = "2024-04-28T11:19:58.158Z" } @@ -4570,9 +4560,9 @@ resolution-markers = [ "python_full_version < '3.11' and platform_machine == 's390x' and sys_platform != 'darwin'", ] dependencies = [ - { name = "huggingface-hub", marker = "sys_platform != 'darwin' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126')" }, - { name = "pyyaml", marker = "sys_platform != 'darwin' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126')" }, - { name = "safetensors", marker = "sys_platform != 'darwin' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126')" }, + { name = "huggingface-hub", marker = "sys_platform != 'darwin' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, + { name = "pyyaml", marker = "sys_platform != 'darwin' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, + { name = "safetensors", marker = "sys_platform != 'darwin' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, { name = "torch", version = "2.6.0+cu124", source = { registry = "https://download.pytorch.org/whl/cu124" }, marker = "(sys_platform != 'darwin' and extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra != 'extra-16-inference-models-torch-cpu' and extra != 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (sys_platform != 'darwin' and extra != 'extra-16-inference-models-onnx-cpu' and extra != 'extra-16-inference-models-onnx-cu118' and extra != 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, { name = "torch", version = "2.7.1+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "(sys_platform != 'darwin' and extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra == 'extra-16-inference-models-torch-cpu') or (sys_platform != 'darwin' and extra != 'extra-16-inference-models-onnx-cpu' and extra != 'extra-16-inference-models-onnx-cu118' and extra != 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-torch-cpu') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, { name = "torch", version = "2.7.1+cu118", source = { registry = "https://download.pytorch.org/whl/cu118" }, marker = "(sys_platform != 'darwin' and extra != 'extra-16-inference-models-onnx-jp6-cu126' and extra != 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (sys_platform != 'darwin' and extra != 'extra-16-inference-models-onnx-cpu' and extra != 'extra-16-inference-models-onnx-cu118' and extra != 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-torch-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, @@ -5261,7 +5251,7 @@ name = "tqdm" version = "4.67.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "colorama", marker = "sys_platform == 'win32' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126')" }, + { name = "colorama", marker = "sys_platform == 'win32' or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu118') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cpu' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-cu12') or (extra == 'extra-16-inference-models-onnx-cu118' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-onnx-cu12' and extra == 'extra-16-inference-models-onnx-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu118') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cpu' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu124') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu118' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu126') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu124' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu128') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu126' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-cu130') or (extra == 'extra-16-inference-models-torch-cu128' and extra == 'extra-16-inference-models-torch-jp6-cu126') or (extra == 'extra-16-inference-models-torch-cu130' and extra == 'extra-16-inference-models-torch-jp6-cu126')" }, ] sdist = { url = "https://files.pythonhosted.org/packages/a8/4b/29b4ef32e036bb34e4ab51796dd745cdba7ed47ad142a9f4a1eb8e0c744d/tqdm-4.67.1.tar.gz", hash = "sha256:f8aef9c52c08c13a65f30ea34f4e5aac3fd1a34959879d7e59e63027286627f2", size = 169737, upload-time = "2024-11-24T20:12:22.481Z" } wheels = [ diff --git a/requirements/requirements.cpu.txt b/requirements/requirements.cpu.txt index 5235d319ce..78a211734c 100644 --- a/requirements/requirements.cpu.txt +++ b/requirements/requirements.cpu.txt @@ -1,3 +1,3 @@ onnxruntime>=1.15.1,<1.22.0 nvidia-ml-py<13.0.0 -inference-models[torch-cpu,onnx-cpu]~=0.31.0 # keep in sync between requirements.gpu.txt, requirements.cpu.txt, requirements.vino.txt +inference-models[torch-cpu,onnx-cpu]~=0.32.0rc3 # keep in sync between requirements.gpu.txt, requirements.cpu.txt, requirements.vino.txt diff --git a/requirements/requirements.gpu.txt b/requirements/requirements.gpu.txt index 1206f06644..6e5bd7c61b 100644 --- a/requirements/requirements.gpu.txt +++ b/requirements/requirements.gpu.txt @@ -1,2 +1,2 @@ onnxruntime-gpu>=1.15.1,<1.22.0 -inference-models[torch-cu124,onnx-cu12]~=0.31.0 # keep in sync between requirements.jetson requirements.gpu.txt, requirements.cpu.txt, requirements.vino.txt +inference-models[torch-cu124,onnx-cu12]~=0.32.0rc3 # keep in sync between requirements.jetson requirements.gpu.txt, requirements.cpu.txt, requirements.vino.txt diff --git a/requirements/requirements.jetson.txt b/requirements/requirements.jetson.txt index 8b907b36fb..f94a3d7e03 100644 --- a/requirements/requirements.jetson.txt +++ b/requirements/requirements.jetson.txt @@ -1,3 +1,3 @@ pypdfium2>=4.11.0,<5.0.0 PyYAML~=6.0.0 -inference-models~=0.31.0 # keep in sync between requirements.jetson requirements.gpu.txt, requirements.cpu.txt, requirements.vino.txt +inference-models~=0.32.0rc3 # keep in sync between requirements.jetson requirements.gpu.txt, requirements.cpu.txt, requirements.vino.txt diff --git a/requirements/requirements.vino.txt b/requirements/requirements.vino.txt index 720022fe46..fc2e6bcdf6 100644 --- a/requirements/requirements.vino.txt +++ b/requirements/requirements.vino.txt @@ -1,2 +1,2 @@ onnxruntime-openvino>=1.15.0,<1.22.0 -inference-models[torch-cpu]~=0.31.0 # keep in sync between requirements.jetson requirements.gpu.txt, requirements.cpu.txt, requirements.vino.txt +inference-models[torch-cpu]~=0.32.0rc3 # keep in sync between requirements.jetson requirements.gpu.txt, requirements.cpu.txt, requirements.vino.txt