Skip to content

Add int64_t support to BackendOptions for pointer-sized handles (#19232)#19232

Open
shoumikhin wants to merge 1 commit intopytorch:mainfrom
shoumikhin:export-D103241865
Open

Add int64_t support to BackendOptions for pointer-sized handles (#19232)#19232
shoumikhin wants to merge 1 commit intopytorch:mainfrom
shoumikhin:export-D103241865

Conversation

@shoumikhin
Copy link
Copy Markdown
Contributor

@shoumikhin shoumikhin commented Apr 30, 2026 •
edited by meta-codesync Bot
Loading

Summary:

Add int64_t as a new variant arm to OptionValue in BackendOptions
and extend BackendInitContext::get_runtime_spec<T> to support it. Lets
backends receive pointer-sized opaque handles (e.g., cudaStream_t,
CUgreenCtx) as runtime specs at delegate init() time.

Motivation: GPU delegates today have no public surface to receive
caller-owned device resources, so they default to allocating their own
streams from the device's primary-context stream pool. This silently
bypasses caller-set CUDA Green Contexts, which is exactly the gap MRS's
MLRT runtime hits when partitioning a Jetson Thor across concurrent
models. With this change, callers that own a GPU context can pass a
green-ctx-bound stream into a Module's load path via BackendOptions,
and GPU delegates that opt in (TensorRT, CUDA, future Vulkan / Metal)
honor it instead of allocating their own.

Fully additive:

  • New int64_t variant arm + matching set_option(int64_t) overload.
  • get_option<int64_t> falls through the existing bool/int path.
  • BackendInitContext::get_runtime_spec<int64_t> extends the existing
    static_assert.
  • No behavior change for existing bool / int / const char* callers.
  • No public API removal or signature change.

Documentation:

  • The int64_t arm docstring spells out the canonical uintptr_t
    round-trip pattern so callers don't bikeshed the cast — raw
    reinterpret_cast<int64_t>(ptr) is implementation-defined per the
    standard; via uintptr_t it is well-defined on every platform.
  • Calls out the literal-overload-resolution gotcha (42 → int,
    42L → platform-dependent) and recommends static_cast<int64_t> at
    the call site.

Defensive scaffolding:

  • static_assert(std::variant_size_v<OptionValue> == 4, ...) in
    set_option_impl guards against future variant arm additions silently
    breaking call sites.
  • Companion static_assert constrains the impl template T to the four
    valid variant arms so accidental misuse fails at compile time.

Differential Revision: D103241865

Copilot AI review requested due to automatic review settings April 30, 2026 18:10
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 30, 2026
@pytorch-bot
Copy link
Copy Markdown

pytorch-bot Bot commented Apr 30, 2026 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19232

Note: Links to docs will display an error until the docs builds have been completed.

❌ 83 Cancelled Jobs, 4 Unrelated Failures

As of commit b478c32 with merge base e84a418 (image):

CANCELLED JOBS - The following jobs were cancelled. Please retry:

FLAKY - The following job failed but was likely due to flakiness present on trunk:

BROKEN TRUNK - The following jobs failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-codesync
Copy link
Copy Markdown
Contributor

meta-codesync Bot commented Apr 30, 2026

@shoumikhin has exported this pull request. If you are a Meta employee, you can view the originating Diff in D103241865.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 30, 2026

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Copilot AI reviewed Apr 30, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds int64_t as a supported runtime backend option type so backends can receive pointer-sized opaque handles via runtime_specs at Method::load_method() time.

Changes:

  • Extend OptionValue to include an int64_t alternative and add a matching BackendOptions::set_option(..., int64_t) overload.
  • Extend BackendInitContext::get_runtime_spec<T> to allow T = int64_t.
  • Add a unit test covering BackendOptions int64_t set/get behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
runtime/backend/options.h Adds int64_t to OptionValue and a new set_option overload; updates docs.
runtime/backend/backend_init_context.h Allows get_runtime_spec<int64_t> and adds <cstdint>.
runtime/backend/test/backend_options_test.cpp Adds int64_t BackendOptions test coverage.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread runtime/backend/backend_init_context.h
Comment thread runtime/backend/options.h Outdated
Comment thread runtime/backend/test/backend_options_test.cpp Outdated
@meta-codesync meta-codesync Bot changed the title Support int64_t in BackendOption variant for pointer-sized opaque handles Support int64_t in BackendOption variant for pointer-sized opaque handles (#19232) Apr 30, 2026
shoumikhin added a commit to shoumikhin/executorch that referenced this pull request Apr 30, 2026
@shoumikhin @facebook-github-bot
Support int64_t in BackendOption variant for pointer-sized opaque han…
6ab86f1 
…dles (pytorch#19232)

Summary:

Add an `int64_t` alternative to the `OptionValue` variant in
`executorch/runtime/backend/options.h`, plus a matching `set_option(int64_t)`
overload on `BackendOptions` and an extended `get_runtime_spec<int64_t>`
template in `BackendInitContext`.

## Motivation

Backends increasingly need to receive pointer-sized opaque handles at
`Method::load_method()` time via the existing `runtime_specs` channel:
- NVIDIA CUDA driver handles (e.g., `CUgreenCtx` for green-context-aware
  TensorRT inference, where the runtime user pre-creates a green context
  and binds the engine's stream to it via `cuGreenCtxStreamCreate`).
- Externally-created CUDA streams shared across delegates.
- Vulkan/Metal command queue handles, etc.

The current `OptionValue` variant supports `bool`, `int`, and a 256-byte
char array. None of these safely round-trips a 64-bit pointer on LP64
platforms — `int` is 32-bit and would silently truncate the upper half of
a pointer, producing use-after-free or segfaults at the consumer side.

A workaround using the `const char*` arm (stringify the pointer to hex,
parse with `strtoull` in the backend) is possible but ugly, error-prone,
and unnecessary given how trivial the additive fix is.

## Scope of change

- Add `int64_t` to the `OptionValue` variant alternatives.
- Add a `set_option(const char (&key)[N], int64_t value)` overload on
  `BackendOptions`.
- Extend `BackendInitContext::get_runtime_spec<T>` template's
  `static_assert` to allow `T = int64_t` in addition to `bool`, `int`,
  `const char*`. The internal `std::get_if<T>` already works with the new
  variant alternative — no other body changes needed.
- Add `<cstdint>` include in both headers.

## Properties

This change is purely additive:

- **Existing consumers see no behavior change.** The `bool`, `int`, and
  `const char*` paths are untouched. Variant alternatives are appended,
  not reordered, so `std::variant::index()` semantics for existing values
  are preserved.
- **No serialized format changes.** `runtime_specs` are passed in-process
  at `Method::load_method` and are never serialized into the .pte file.
  AOT `CompileSpec` lives in a separate Python schema (flatbuffer-backed)
  and is unchanged here.
- **No public API removed or renamed.** The new `int64_t` overload is
  selected only when callers explicitly pass `int64_t`-typed values;
  integer literals continue to bind to the `int` overload.

## Caller usage

The intended usage pattern, illustrated for the green-context case:

  // 1) User creates the opaque resource and owns its lifetime
  CUgreenCtx green = nullptr;
  cuGreenCtxCreate(&green, desc, device, CU_GREEN_CTX_DEFAULT_STREAM);

  // 2) Pass as a runtime spec at Method::load_method time
  BackendOption opt{};
  std::strcpy(opt.key, "green_context");
  opt.value =
      static_cast<int64_t>(reinterpret_cast<intptr_t>(green));
  Span<const BackendOption> specs(&opt, 1);
  // ... thread `specs` through Module::load_program / load_method

  // 3) In the backend's init():
  auto h = context.get_runtime_spec<int64_t>("green_context");
  if (h.ok()) {
    auto green = reinterpret_cast<CUgreenCtx>(h.get());
    CUstream raw_stream;
    cuGreenCtxStreamCreate(
        &raw_stream, green, CU_STREAM_NON_BLOCKING, /*priority=*/0);
    // bind raw_stream to the engine for enqueueV3...
  }

Differential Revision: D103241865
@meta-codesync meta-codesync Bot changed the title Support int64_t in BackendOption variant for pointer-sized opaque handles (#19232) Intermediate commit for 1777572697 Apr 30, 2026
Copilot AI review requested due to automatic review settings April 30, 2026 18:48
Copilot AI reviewed Apr 30, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds int64_t support to backend option plumbing so runtimes/backends can pass and retrieve pointer-sized opaque handles (and other 64-bit values) with strict type checking.

Changes:

  • Extend BackendOptions / BackendInitContext::get_runtime_spec<T> to support int64_t values in the underlying std::variant.
  • Update API documentation around string storage semantics and supported option types.
  • Add unit tests covering int64_t set/get behavior and type-mismatch errors.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
runtime/backend/options.h Adds int64_t to OptionValue, adds set_option(..., int64_t), updates docs for supported types and string handling
runtime/backend/backend_init_context.h Extends get_runtime_spec<T> static_assert and retrieval logic to allow int64_t
runtime/backend/test/backend_options_test.cpp Adds int64_t round-trip and type-mismatch tests for BackendOptions
runtime/backend/test/backend_init_context_test.cpp Adds get_runtime_spec<int64_t> tests and expands type-mismatch coverage

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread runtime/backend/options.h
Comment thread runtime/backend/options.h Outdated
Comment thread runtime/backend/backend_init_context.h
@meta-codesync meta-codesync Bot changed the title Intermediate commit for 1777572697 Add int64_t support to BackendOptions for pointer-sized handles Apr 30, 2026
Copilot AI review requested due to automatic review settings April 30, 2026 22:17
@shoumikhin @facebook-github-bot
Add int64_t support to BackendOptions for pointer-sized handles (pyto…
305af49 
…rch#19232)

Summary:

Add `int64_t` as a new variant arm to `OptionValue` in `BackendOptions`
and extend `BackendInitContext::get_runtime_spec<T>` to support it. Lets
backends receive pointer-sized opaque handles (e.g., `cudaStream_t`,
`CUgreenCtx`) as runtime specs at delegate `init()` time.

Motivation: GPU delegates today have no public surface to receive
caller-owned device resources, so they default to allocating their own
streams from the device's primary-context stream pool. This silently
bypasses caller-set CUDA Green Contexts, which is exactly the gap MRS's
MLRT runtime hits when partitioning a Jetson Thor across concurrent
models. With this change, callers that own a GPU context can pass a
green-ctx-bound stream into a Module's load path via `BackendOptions`,
and GPU delegates that opt in (TensorRT, CUDA, future Vulkan / Metal)
honor it instead of allocating their own.

Fully additive:
- New `int64_t` variant arm + matching `set_option(int64_t)` overload.
- `get_option<int64_t>` falls through the existing `bool/int` path.
- `BackendInitContext::get_runtime_spec<int64_t>` extends the existing
  `static_assert`.
- No behavior change for existing `bool` / `int` / `const char*` callers.
- No public API removal or signature change.

Documentation:
- The int64_t arm docstring spells out the canonical `uintptr_t`
  round-trip pattern so callers don't bikeshed the cast — raw
  `reinterpret_cast<int64_t>(ptr)` is implementation-defined per the
  standard; via `uintptr_t` it is well-defined on every platform.
- Calls out the literal-overload-resolution gotcha (`42` → `int`,
  `42L` → platform-dependent) and recommends `static_cast<int64_t>` at
  the call site.

Defensive scaffolding:
- `static_assert(std::variant_size_v<OptionValue> == 4, ...)` in
  `set_option_impl` guards against future variant arm additions silently
  breaking call sites.
- Companion `static_assert` constrains the impl template `T` to the four
  valid variant arms so accidental misuse fails at compile time.

Differential Revision: D103241865
@meta-codesync meta-codesync Bot changed the title Add int64_t support to BackendOptions for pointer-sized handles Add int64_t support to BackendOptions for pointer-sized handles (#19232) Apr 30, 2026
Copilot AI reviewed Apr 30, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds int64_t as a supported BackendOptions value type so backends can receive pointer-sized opaque handles (e.g., CUDA streams/contexts) via runtime specs at delegate init time.

Changes:

  • Extend OptionValue to include an int64_t variant arm and add a set_option(..., int64_t) overload.
  • Extend BackendInitContext::get_runtime_spec<T> to allow T = int64_t.
  • Add unit tests covering int64_t option set/get and type-mismatch behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
runtime/backend/options.h Adds int64_t option support and updates option documentation/validation scaffolding.
runtime/backend/backend_init_context.h Allows retrieving int64_t runtime specs via get_runtime_spec<int64_t>().
runtime/backend/test/backend_options_test.cpp Adds tests verifying int64_t options round-trip and type mismatch with int.
runtime/backend/test/backend_init_context_test.cpp Adds tests for get_runtime_spec<int64_t>() and mismatch behavior between int and int64_t.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread runtime/backend/options.h
Comment on lines +61 to +62
* arm allows callers to pass pointer-sized opaque handles (e.g., driver
* handles like CUgreenCtx) by reinterpret_cast to/from int64_t.
@Gasoonjia
Copy link
Copy Markdown
Contributor

Gasoonjia commented Apr 30, 2026

it looks good to me, thanks @shoumikhin !
Maybe @metascroy or @JacobSzwejbka you can also take a look?

@metascroy
Copy link
Copy Markdown
Contributor

metascroy commented Apr 30, 2026

Adding an int64_t to backend options seems fine to me, but not sure how I feel about the intended use-case of passing in caller-owned device resources. Seems a bit hacky.

Maybe @JacobSzwejbka has thoughts there, I know he had some concerns about people overusing backend options.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

ciflow/periodic ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. fb-exported meta-exported

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@shoumikhin @Gasoonjia @metascroy