Nuke inference plugin

[jmccarten-wetafx]

For inference of a Image to Image network in Nuke - RMTC CAT building is in the pipeline - however using CAT to execute a network does not allow RMTC to directly track the inference result, this would still be required to be populated, in addition CAT files operate on specifically wrapped Torchscript files.

Creating an RMTC inference plugin that indexes off solutions rather than weights or models will permit flexible 'best run' interfacing, multiple model type support and allow the inferences to be correctly recorded to RMTC as they occur.

[acherbetji]

I'd be interested to understand whether we could enable this by extending the current inference node or make changes to the NDK.

1. Could we address the need to monitor output with callbacks on the inference node?
2. Is the torch script requirement a blocker / do you expect to implement other inference frameworks?

If building a new inference plugin is still necessary, which of these would be beneficial
3. Shipping PyTorch headers and exposing the bundled PyTorch instance, for example, via an API or an NDK MLPlanarIop?
4. Including the python torch library in the Nuke bundle?
Anything else?

One of the reasons I think it would be useful to integrate rather than replace, is Nuke 17 implements a shared allocator between Blink and PyTorch for better gpu memory management.

Best,

Adam (Foundry)

[jmccarten-wetafx]

Hi Adam,

There are a few points to consider as to why we were thinking of a level above CAT model execution.

Tracking inferences - when running a CAT model it does not populate RMTC with the inference tracking information - we need a mechanism to to inject it, for DCCs we were thinking it would be done as a single batched inference lodged when you save/export any specific inferred asset in the DCC scene. This is really the main missing aspect.

Solutions first inferencing - to give an opportunity for DCCs to pull in the best quality model with permissions for use in a given context - we have suggested focusing on 'solution' as the entry point for a DCC to find models. This means if a particular dataset, model or weights is later found in violation of license permissions (e.g. through a GDPR, CA942 or general PII request) the indirection will permit RMTC to locate an alternative model automatically. This isn't a requirement, but it is one of the features the system provides.

Processor stack - models have various representations of tensor ins/outs - take images: Keras style (B)HWC or PyTorch (B)CHW. In an attempt to allow for richer interaction of models - RMTC attempts to bring a common denominator format to allow for assets to move interchangeably between them (currently (B)HWC - a nod to cache coherency for possible future inference/train optimisations). This is paired with a reversible processor stack attached to each model instance. This stack takes the common RMTC representation to the model specific representation and is ran at pre-inference. This stack is currently Python but we aim to move it to a native implementation for direct execution in a DCC. Ideally we avoid the stack completely by running a pre-process step during the pipeline - where we take this stack as a specification to manipulate the model so it is aligned with a given DCC's data format and we reduce the inference overhead with models (this is a speculative idea at present and likely not possibly in general).

Models - RMTC is framework agnostic, we currently support PyTorch for training with ONNX (coming) & PyTorch inference but these are added as an extension. If we indirect to RMTC for inference then we open up to support historical models in other frameworks that may already exist in the pipeline. Of course there's a perf impact and an environment management challenge for any such abstraction.

Another lesser consideration is RMTC's specific separation of weights and models in the provenance tree - CAT represents a combination of both. This is modelled through variant/ancestors in RMTC's provenance tree - though we do see a TD wanting to swap out combinations potentially - we would have to rebuild the CAT model each time if these are not separated.

Ideally - what we seek from a DCC when running a model in RMTC

• Inject RMTC tracking information at an appropriate cadence
• Work with the solution concept over direct model execution
• Support the processor stack

Now this could be a change to the way the NDK works, this could be a plugin, this could be some scripting that sits around the current CAT model. Our thinking was a 2 tier plugin system - a generic DCC agnostic representation with DCC specific implementations that marshal to the RMTC stack input.