exir: add flatbuffer Program serialization for performance by chizkiyahu · Pull Request #17333 · pytorch/executorch

chizkiyahu · 2026-02-10T13:47:06Z

Title

Introduce flatbuffer serializer as default for program serialization

Summary

Add a FlatBuffers-based serializer at exir/_serialize/_flatbuffer_program.py and make it the default serializer for program artifacts. The new serializer achieves large efficiency gains across our model suite while retaining robustness via a JSON fallback and detailed error logging.

Before

python -> json -> binary

After

python -> binary

Performance

Measured across ~600 models:

~36.3× average speedup (serialize latency).
~61.6% average reduction in memory usage during serialization.

Tests

Unit tests added at:

exir/_serialize/test/test_flatbuffer_program.py

cc @digantdesai @SS-JIA @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell

pytorch-bot · 2026-02-10T13:47:11Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/17333

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 2 Unrelated Failures

As of commit b313378 with merge base d660dce ():

NEW FAILURES - The following jobs have failed:

pull / test-samsung-models-linux / linux-job (gh)
RuntimeError: Command docker exec -t d17fe174f97e2765432e2977c6d27de361379f9470f55286cc108bc4754a7d90 /exec failed with exit code 1
pull / test-samsung-quantmodels-linux / linux-job (gh)
RuntimeError: Command docker exec -t 3e3865b5c9dc4a6b918fb927df1ecaec09b36a35ff2d85e68fa074300d8c792c /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / unittest / macos / macos-job (gh) (trunk failure)
AttributeError: '_OpNamespace' 'mkldnn' object has no attribute '_is_mkldnn_acl_supported'
pull / unittest-editable / macos / macos-job (gh) (trunk failure)
AttributeError: '_OpNamespace' 'mkldnn' object has no attribute '_is_mkldnn_acl_supported'

This comment was automatically generated by Dr. CI and updates every 15 minutes.

chizkiyahu · 2026-02-10T13:48:30Z

@pytorchbot label "partner: arm"

chizkiyahu · 2026-02-10T13:49:24Z

@pytorchbot label ciflow/trunk

pytorch-bot · 2026-02-10T13:49:30Z

To add these label(s) (ciflow/trunk) to the PR, please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

Copilot

Pull request overview

Adds a new FlatBuffer-based serializer for Program and switches the default serialization path to it for major performance/memory improvements, while keeping a JSON-based fallback for robustness.

Changes:

Introduces direct Python FlatBuffer packing for Program (_flatbuffer_program.py) and uses it by default with JSON fallback + logging.
Adds tooling/docs + CI for regenerating FlatBuffer Python bindings from schema/*.fbs.
Adds unit tests + BUCK targets for FlatBuffer serialization behavior.

Reviewed changes

Copilot reviewed 11 out of 54 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
schema/README.md	Documents how to regenerate committed FlatBuffer Python bindings.
exir/_serialize/test/test_flatbuffer_program.py	Adds unit tests for roundtrip and parity between direct vs JSON-based FlatBuffer generation.
exir/_serialize/test/BUCK	Registers the new unit test target.
exir/_serialize/generate_program.py	Adds a generator script to run `flatc` and rewrite imports/init files for vendored bindings.
exir/_serialize/_program.py	Switches default serialization to FlatBuffer fast path with JSON fallback + logging.
exir/_serialize/_flatbuffer_program.py	Implements direct FlatBuffer packing for `Program` without invoking `flatc`.
exir/_serialize/_flatbuffer.py	Extends schema prep to extract file_identifier and effective alignment values.
exir/_serialize/BUCK	Exposes the new serializer module in the library target.
.lintrunner.toml	Excludes generated FlatBuffer bindings from lintrunner.
.github/workflows/validate_flatbuffer_gen.yml	Adds CI workflow intended to validate generated bindings are up to date.
.flake8	Excludes generated FlatBuffer bindings from flake8.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

chizkiyahu · 2026-02-10T14:53:59Z

@pytorchbot label "release notes: exir"

pytorch-bot · 2026-02-10T14:58:14Z

To add the ciflow label ciflow/trunk please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

Copilot

Pull request overview

Copilot reviewed 11 out of 54 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

exir/_serialize/_flatbuffer_program.py:1

The shebang line '#!/usr/bin/env fbpython' is specific to Facebook's internal Python environment. For an open source repository, this should be either removed or changed to a standard Python shebang like '#!/usr/bin/env python3'.

# Copyright 2026 Arm Limited and/or its affiliates.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

chizkiyahu · 2026-02-10T17:14:01Z

@mergennachin

this is the updated PR of #16691
thanks

JacobSzwejbka · 2026-02-11T22:27:02Z

+    return _BackendDelegateInlineData.BackendDelegateInlineDataEnd(builder)
+
+
+def _install_fast_packers() -> None:


Does this make a big difference in serialization speed?

yes it give very big improvement at speed

quick test on resnet18 show improve from ~3.54 sec to 0.07 sec ~50x

wow, that's a big speed up!

chizkiyahu · 2026-02-11T23:26:24Z

@pytorchbot label ciflow/trunk

pytorch-bot · 2026-02-11T23:26:29Z

To add these label(s) (ciflow/trunk) to the PR, please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

Copilot

Pull request overview

Copilot reviewed 11 out of 54 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 13 out of 56 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-25T08:48:15Z

+def _convert_evalue(val: EValue) -> Any:
+    result = _flatbuffer_t_class(EValue)()
+    union_val = val.val
+    if union_val is None:
+        result.valType = _KernelTypes.KernelTypes.NONE
+        result.val = None
+        return result
+    union_name = type(union_val).__name__
+    result.valType = getattr(_KernelTypes.KernelTypes, union_name)
+    result.val = _convert_value(union_val)
+    return result
+
+
+def _convert_instruction(val: Instruction) -> Any:
+    result = _flatbuffer_t_class(Instruction)()
+    union_val = val.instr_args
+    if union_val is None:
+        result.instrArgsType = _InstructionArguments.InstructionArguments.NONE
+        result.instrArgs = None
+        return result
+    union_name = type(union_val).__name__
+    result.instrArgsType = getattr(
+        _InstructionArguments.InstructionArguments, union_name
+    )
+    result.instrArgs = _convert_value(union_val)
+    return result


The functions _convert_evalue and _convert_instruction use type(union_val).__name__ to get the union type name and then use getattr() to retrieve the corresponding enum value. If the union contains a type that doesn't have a corresponding enum value in _KernelTypes.KernelTypes or _InstructionArguments.InstructionArguments, this will raise an AttributeError at runtime. Consider adding error handling or validation to provide a more informative error message if an unexpected type is encountered.

Copilot · 2026-02-25T08:48:16Z

+    return val
+
+
+def convert_program(val: Program) -> ProgramT:


The function convert_program is publicly exposed (no underscore prefix) but lacks documentation and type annotations for its return type. While it's used internally, if it's intended to be part of the public API, it should have proper documentation. If it's meant to be internal, it should be prefixed with an underscore.

Suggested change

def convert_program(val: Program) -> ProgramT:

def convert_program(val: Program) -> ProgramT:

"""

Convert a :class:`executorch.exir.schema.Program` instance into its

corresponding FlatBuffers table type (:class:`ProgramT`).

This helper is used when serializing a high-level Program schema object

into the low-level FlatBuffers representation expected by the generated

executorch flatbuffer bindings.

Args:

val: The high-level Program object to convert.

Returns:

The corresponding ProgramT instance suitable for FlatBuffers

serialization.

"""

pytorch-bot · 2026-02-25T09:09:12Z

To add the ciflow label ciflow/trunk please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

JacobSzwejbka · 2026-02-27T18:50:49Z

Pulling internally again. Will play around with the bitwise differences to make sure they all checkout.

Gasoonjia · 2026-03-02T20:28:56Z

@@ -19,8 +20,8 @@
 from executorch.exir._serialize._flatbuffer import (
    _FlatbufferResult,
    _program_flatbuffer_to_json,


do we still need this?

This is used in the deserialize_pte_binary function, which still relies on JSON:

program: Program = _json_to_program( _program_flatbuffer_to_json(program_data[:program_size]) )

As I understand it, deserialize_pte_binary is used only for testing and tooling, not in production code. However, since it is part of the public API, I am not entirely sure whether that assumption is correct.

@Gasoonjia

Please let me know if you would like me to remove _program_flatbuffer_to_json from deserialize_pte_binary, or even remove it entirely from the codebase.

Alternatively, let me know if you think it is fine to keep it as it is.

We should keep deserialize_pte_binary - is it possible to deserialize using the new python path instead of json?

is a not a small change
I will open another PR for this

JacobSzwejbka · 2026-03-03T00:32:25Z

Its all good on my side. Just need the merge conflict to be resolved. Im happy if you want to take a look at deserialize in a followup I dont think its blocking for this one. @chizkiyahu

Signed-off-by: Chizkiyahu Raful <chizkiyahu.raful@arm.com> Change-Id: Ie72323e17bcab7999bed89702bc42dc9df665b93

Change-Id: I23787d086aa9229f5d93d1e16c6f6297119e7f5a Signed-off-by: Chizkiyahu Raful <chizkiyahu.raful@arm.com>

chizkiyahu · 2026-03-03T10:42:49Z

Just need the merge conflict to be resolved

done

Copilot

Pull request overview

Copilot reviewed 13 out of 56 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-03T10:43:54Z

+@functools.lru_cache(maxsize=1)
+def _install_fast_packers() -> None:
+    _Buffer.BufferT.Pack = _pack_buffer
+    _BackendDelegateInlineData.BackendDelegateInlineDataT.Pack = (
+        _pack_backend_delegate_inline_data
+    )


The _install_fast_packers function is decorated with @functools.lru_cache(maxsize=1), which permanently monkey-patches the Pack methods on BufferT and BackendDelegateInlineDataT class objects the first time it is called. Because these are class-level mutations (not instance-level), the patched Pack method will be active for all subsequent uses of those classes throughout the entire process lifetime — including any parallel uses, tests that import these classes independently, or any code path that calls BufferT.Pack or BackendDelegateInlineDataT.Pack without going through _program_to_flatbuffer. This is a global side effect that is never undone, which makes the code fragile and hard to reason about in concurrent or test environments.

Copilot · 2026-03-03T10:43:55Z

+def _pack_buffer(self: Any, builder: Any) -> int:
+    storage = 0
+    if self.storage is not None:
+        storage = _create_aligned_byte_vector(
+            builder, _coerce_bytes(self.storage), _BUFFER_ALIGNMENT.get()
+        )
+    _Buffer.BufferStart(builder)
+    if storage:
+        _Buffer.BufferAddStorage(builder, storage)
+    return _Buffer.BufferEnd(builder)


The _pack_buffer function checks if storage: (line 97) to decide whether to call BufferAddStorage. Since storage is a FlatBuffers offset (an integer), this check is falsy when the offset is 0. A FlatBuffers offset of 0 would only occur for an empty vector and 0 is not a valid table offset in practice, but this subtle reliance on the offset never being 0 for a present field is fragile. Using an explicit if self.storage is not None: guard mirroring the outer check would be clearer and safer. The same issue applies to _pack_backend_delegate_inline_data at line 109.

Copilot

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

JacobSzwejbka · 2026-03-11T18:09:52Z

Just wanted to say thanks again @chizkiyahu. This is an awesome change!

This continues the work from pytorch#17333. Change-Id: I35ac4cd5f6430ea89939453344c13e056b5c746c Signed-off-by: Chizkiyahu Raful <chizkiyahu.raful@arm.com>

@digantdesai

…7333) ## **Title** Introduce flatbuffer serializer as default for program serialization ## Summary Add a FlatBuffers-based serializer at `exir/_serialize/_flatbuffer_program.py` and make it the default serializer for program artifacts. The new serializer achieves large efficiency gains across our model suite while retaining robustness via a JSON fallback and detailed error logging. ### Before python -> json -> binary ### After python -> binary ## Performance Measured across ~600 models: * ~36.3× average speedup (serialize latency). * ~61.6% average reduction in memory usage during serialization. ## Tests Unit tests added at: * `exir/_serialize/test/test_flatbuffer_program.py` cc @digantdesai @SS-JIA @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell --------- Signed-off-by: Chizkiyahu Raful <chizkiyahu.raful@arm.com> Co-authored-by: Jacob Szwejbka <jakeszwe@meta.com>

This continues the work from pytorch#17333. Change-Id: I35ac4cd5f6430ea89939453344c13e056b5c746c Signed-off-by: Chizkiyahu Raful <chizkiyahu.raful@arm.com>

Copilot AI review requested due to automatic review settings February 10, 2026 13:47

chizkiyahu requested review from JacobSzwejbka, larryliu0820 and lucylq as code owners February 10, 2026 13:47

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 10, 2026

pytorch-bot Bot added the partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm label Feb 10, 2026

Copilot AI reviewed Feb 10, 2026

View reviewed changes

chizkiyahu mentioned this pull request Feb 10, 2026

exir: add flatbuffer Program serialization for performance #16691

Closed

pytorch-bot Bot added the release notes: exir Changes to any dialects and passes on these dialects, such as memory planning label Feb 10, 2026

mansnils added the ciflow/trunk label Feb 10, 2026

pytorch-bot Bot removed the ciflow/trunk label Feb 10, 2026

Copilot AI review requested due to automatic review settings February 10, 2026 16:27

Copilot AI reviewed Feb 10, 2026

View reviewed changes

Comment thread exir/_serialize/_flatbuffer.py

mergennachin requested review from GregoryComer and metascroy February 10, 2026 17:16

JacobSzwejbka reviewed Feb 11, 2026

View reviewed changes

Comment thread exir/_serialize/_flatbuffer_program.py Outdated

Copilot AI review requested due to automatic review settings February 11, 2026 23:25

Copilot AI reviewed Feb 11, 2026

View reviewed changes

Comment thread exir/_serialize/_flatbuffer_program.py

Comment thread exir/_serialize/_flatbuffer_program.py

Comment thread exir/_serialize/_flatbuffer_program.py

Comment thread exir/_serialize/_program.py Outdated

Comment thread exir/_serialize/test/test_flatbuffer_program.py

lucylq reviewed Feb 11, 2026

View reviewed changes

Comment thread exir/_serialize/_flatbuffer_program.py

Comment thread exir/_serialize/generate_program.py

chizkiyahu force-pushed the exir-flatbuffer-serialize-fastpath_v2 branch from 96943c2 to 247294d Compare February 12, 2026 08:24

Copilot started reviewing on behalf of chizkiyahu February 25, 2026 08:46 View session

Copilot AI reviewed Feb 25, 2026

View reviewed changes

mansnils added the ciflow/trunk label Feb 25, 2026

pytorch-bot Bot removed the ciflow/trunk label Feb 25, 2026

chizkiyahu requested a review from JacobSzwejbka February 27, 2026 01:24

JacobSzwejbka approved these changes Mar 2, 2026

View reviewed changes

Merge branch 'main' into exir-flatbuffer-serialize-fastpath_v2

e28f0c5

Gasoonjia reviewed Mar 2, 2026

View reviewed changes

chizkiyahu added 2 commits March 3, 2026 10:27

exir: resolve conflict in exir/_serialize/test/BUCK

bce3db3

Signed-off-by: Chizkiyahu Raful <chizkiyahu.raful@arm.com> Change-Id: Ie72323e17bcab7999bed89702bc42dc9df665b93

exir: merge main into exir-flatbuffer-serialize-fastpath_v2

c82f1ab

Change-Id: I23787d086aa9229f5d93d1e16c6f6297119e7f5a Signed-off-by: Chizkiyahu Raful <chizkiyahu.raful@arm.com>

Copilot AI review requested due to automatic review settings March 3, 2026 10:40

Copilot started reviewing on behalf of chizkiyahu March 3, 2026 10:41 View session

Copilot AI reviewed Mar 3, 2026

View reviewed changes

JacobSzwejbka added 2 commits March 4, 2026 10:35

Merge branch 'main' into exir-flatbuffer-serialize-fastpath_v2

2766d9a

Merge branch 'main' into exir-flatbuffer-serialize-fastpath_v2

b313378

Copilot AI review requested due to automatic review settings March 6, 2026 04:37

Copilot AI reviewed Mar 6, 2026

View reviewed changes

JacobSzwejbka merged commit be6b986 into pytorch:main Mar 6, 2026
155 of 161 checks passed

chizkiyahu mentioned this pull request Mar 12, 2026

Serialize/flatbuffer to program #18129

Open

		return _BackendDelegateInlineData.BackendDelegateInlineDataEnd(builder)


		def _install_fast_packers() -> None:

-def convert_program(val: Program) -> ProgramT:
+def convert_program(val: Program) -> ProgramT:
+    """
+    Convert a :class:`executorch.exir.schema.Program` instance into its
+    corresponding FlatBuffers table type (:class:`ProgramT`).
+    This helper is used when serializing a high-level Program schema object
+    into the low-level FlatBuffers representation expected by the generated
+    executorch flatbuffer bindings.
+    Args:
+        val: The high-level Program object to convert.
+    Returns:
+        The corresponding ProgramT instance suitable for FlatBuffers
+        serialization.
+    """

Conversation

chizkiyahu commented Feb 10, 2026 • edited by pytorch-bot Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Title

Summary

Before

After

Performance

Tests

Uh oh!

pytorch-bot Bot commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/17333

❌ 2 New Failures, 2 Unrelated Failures

Uh oh!

chizkiyahu commented Feb 10, 2026

Uh oh!

chizkiyahu commented Feb 10, 2026

Uh oh!

pytorch-bot Bot commented Feb 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chizkiyahu commented Feb 10, 2026

Uh oh!

pytorch-bot Bot commented Feb 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

chizkiyahu commented Feb 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chizkiyahu commented Feb 11, 2026

Uh oh!

pytorch-bot Bot commented Feb 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

pytorch-bot Bot commented Feb 25, 2026

chizkiyahu commented Feb 10, 2026 •

edited by pytorch-bot Bot

Loading

pytorch-bot Bot commented Feb 10, 2026 •

edited

Loading

JacobSzwejbka commented Mar 3, 2026 •

edited

Loading