MINIFICPP-2740 Rust bindings based on stable C API#2186
Draft
martinzink wants to merge 1 commit into
Draft
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a new Rust workspace (minifi_rust/) that provides Rust bindings and an idiomatic safe API on top of the MiNiFi stable C API, along with a sample Rust extension and CI integration (unit tests + Behave-based Docker integration tests).
Changes:
- Adds
minifi_native_sys(bindgen-generated raw FFI) andminifi_native(safe Rust API + C-FFI glue + mocks/macros) crates. - Adds
minifi_rs_playgroundexample extension (processors/controller services) with unit tests and Behave integration tests. - Integrates Rust into the build/CI pipeline (CMake/Corrosion, Docker build scripts, GitHub Actions jobs/artifacts).
Reviewed changes
Copilot reviewed 127 out of 128 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| minifi_rust/rustfmt.toml | Rust formatting configuration for the workspace. |
| minifi_rust/README.md | Documents the Rust workspace architecture and extension authoring/deployment. |
| minifi_rust/minifi_rs_behave/src/main.rs | Rust test runner for Behave integration tests and optional Linux build trigger. |
| minifi_rust/minifi_rs_behave/linux_build.sh | Docker buildx wrapper to build Linux .so artifacts with SDK injection. |
| minifi_rust/minifi_rs_behave/debian.dockerfile | Debian-based container build for producing the Rust extension .so. |
| minifi_rust/minifi_rs_behave/Cargo.toml | Cargo manifest for the Behave runner crate. |
| minifi_rust/minifi_rs_behave/build.rs | Build script to expose Behave framework path to the runner. |
| minifi_rust/minifi_rs_behave/alpine.dockerfile | Alpine-based container build for producing the Rust extension .so. |
| minifi_rust/minifi_native/src/mock/mock_process_session.rs | Mock ProcessSession implementation for unit testing processors. |
| minifi_rust/minifi_native/src/mock/mock_process_context.rs | Mock ProcessContext/property map/controller service lookup for unit tests. |
| minifi_rust/minifi_native/src/mock/mock_logger.rs | Mock/stdout logger implementations + macro laziness test. |
| minifi_rust/minifi_native/src/mock/mock_flow_file.rs | Mock FlowFile implementation used by unit tests. |
| minifi_rust/minifi_native/src/mock/mock_controller_service_context.rs | Mock controller service context implementing GetProperty. |
| minifi_rust/minifi_native/src/mock.rs | Exposes mock types from the minifi_native crate. |
| minifi_rust/minifi_native/src/lib.rs | Crate root exports + extension registration macro + API version symbol. |
| minifi_rust/minifi_native/src/c_ffi/c_ffi_streams.rs | C-FFI input/output stream adapters implementing BufRead/Write. |
| minifi_rust/minifi_native/src/c_ffi/c_ffi_relationship.rs | Converts Rust relationship definitions into C representations. |
| minifi_rust/minifi_native/src/c_ffi/c_ffi_property.rs | Converts Rust property definitions + validators into C representations. |
| minifi_rust/minifi_native/src/c_ffi/c_ffi_processor_list.rs | Holds processor definitions and exposes a C array pointer/count. |
| minifi_rust/minifi_native/src/c_ffi/c_ffi_process_context.rs | C-FFI ProcessContext implementation (properties/controller services). |
| minifi_rust/minifi_native/src/c_ffi/c_ffi_primitives.rs | Shared C-FFI primitives (string views, conversions, enums). |
| minifi_rust/minifi_native/src/c_ffi/c_ffi_output_attribute.rs | Converts output attributes into C representations with stable backing storage. |
| minifi_rust/minifi_native/src/c_ffi/c_ffi_logger.rs | C-FFI logger adapter delegating to MiNiFi’s logging callbacks. |
| minifi_rust/minifi_native/src/c_ffi/c_ffi_flow_file.rs | C-FFI flow file wrapper type. |
| minifi_rust/minifi_native/src/c_ffi/c_ffi_controller_service_list.rs | Holds controller service definitions and exposes a C array pointer/count. |
| minifi_rust/minifi_native/src/c_ffi/c_ffi_controller_service_definition.rs | C-FFI controller service definition + callbacks wiring. |
| minifi_rust/minifi_native/src/c_ffi/c_ffi_controller_service_context.rs | C-FFI controller service context property retrieval. |
| minifi_rust/minifi_native/src/c_ffi.rs | Module wiring/re-exports for the C-FFI layer. |
| minifi_rust/minifi_native/src/api/relationship.rs | Defines the Relationship API type. |
| minifi_rust/minifi_native/src/api/raw_processor.rs | Defines raw processor traits + threading model markers. |
| minifi_rust/minifi_native/src/api/raw_controller_service.rs | Defines raw controller service trait. |
| minifi_rust/minifi_native/src/api/property.rs | Defines property schema + typed getters/validators. |
| minifi_rust/minifi_native/src/api/processor.rs | Defines scheduling/metrics/advanced features + processor wrapper struct. |
| minifi_rust/minifi_native/src/api/processor_wrappers/utils/flow_file_content.rs | Defines content representation (buffer/stream) for wrapper APIs. |
| minifi_rust/minifi_native/src/api/processor_wrappers/utils/context_session_flowfile_bundle.rs | Bundles context/session/flowfile to unify property/attribute access. |
| minifi_rust/minifi_native/src/api/processor_wrappers/utils.rs | Exposes utility modules used by wrapper APIs. |
| minifi_rust/minifi_native/src/api/processor_wrappers/flow_file_transform.rs | Higher-level “transform” wrapper API for buffer/stream replacements. |
| minifi_rust/minifi_native/src/api/processor_wrappers/flow_file_stream_transform.rs | Higher-level streaming transform wrapper API. |
| minifi_rust/minifi_native/src/api/processor_wrappers/flow_file_source.rs | Higher-level source/generator wrapper API. |
| minifi_rust/minifi_native/src/api/processor_wrappers/complex_processor.rs | Wrapper for “complex” processors using trigger traits. |
| minifi_rust/minifi_native/src/api/processor_wrappers.rs | Module wiring for processor wrapper APIs. |
| minifi_rust/minifi_native/src/api/process_session.rs | Defines ProcessSession/stream traits + IO state. |
| minifi_rust/minifi_native/src/api/process_context.rs | Defines ProcessContext + typed property getters. |
| minifi_rust/minifi_native/src/api/logger.rs | Defines log levels, Logger trait, and logging macros. |
| minifi_rust/minifi_native/src/api/flow_file.rs | Defines the FlowFile marker trait. |
| minifi_rust/minifi_native/src/api/errors.rs | Defines MinifiError and conversions/display behavior. |
| minifi_rust/minifi_native/src/api/controller_service.rs | Defines controller service wrapper + enable behavior. |
| minifi_rust/minifi_native/src/api/component_definition_traits.rs | Defines processor/controller-service definition traits and IDs. |
| minifi_rust/minifi_native/src/api/attribute.rs | Defines output attributes and attribute getter trait. |
| minifi_rust/minifi_native/src/api.rs | API module wiring and public re-exports. |
| minifi_rust/minifi_native/Cargo.toml | Cargo manifest for the safe Rust API crate. |
| minifi_rust/minifi_native/.gitignore | Rust crate ignore rules. |
| minifi_rust/minifi_native_sys/src/lib.rs | Raw bindgen include wrapper for the generated FFI bindings. |
| minifi_rust/minifi_native_sys/Cargo.toml | Cargo manifest for the raw FFI crate and build dependencies. |
| minifi_rust/minifi_native_sys/build.rs | Build script to locate/download SDK, generate bindings, and emit metadata. |
| minifi_rust/minifi_native_sys/.gitignore | Rust crate ignore rules. |
| minifi_rust/minifi_native_macros/src/lib.rs | Proc-macros for component IDs, default metrics, and feature defaults. |
| minifi_rust/minifi_native_macros/Cargo.toml | Cargo manifest for the proc-macro crate. |
| minifi_rust/generate_docs/generate_docs.sh | Script to generate docs via containerized MiNiFi run. |
| minifi_rust/generate_docs/generate_docs.dockerfile | Dockerfile for docs generation against a MiNiFi container image. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/put_file/unix_only_properties.rs | PutFile Unix-only properties (permissions). |
| minifi_rust/extensions/minifi_rs_playground/src/processors/put_file/tests.rs | Unit tests for the PutFile processor. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/put_file/relationships.rs | Relationship definitions for PutFile. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/put_file/properties.rs | Property definitions for PutFile. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/put_file/processor_definition.rs | ProcessorDefinition wiring for PutFile. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/put_file.rs | PutFile processor implementation using the Rust API. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/mod.rs | Processor module wiring for the sample extension. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/lorem_ipsum_cs_user/tests.rs | Unit tests for controller-service usage processor. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/lorem_ipsum_cs_user/relationships.rs | Relationship definitions for LoremIpsumCSUser. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/lorem_ipsum_cs_user/properties.rs | Property definitions for LoremIpsumCSUser. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/lorem_ipsum_cs_user/processor_definition.rs | ProcessorDefinition wiring for LoremIpsumCSUser. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/lorem_ipsum_cs_user.rs | FlowFile source processor exercising controller services + content modes. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/log_attribute/tests.rs | Unit tests for LogAttribute processor behavior. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/log_attribute/relationships.rs | Relationship definitions for LogAttribute. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/log_attribute/properties.rs | Property definitions for LogAttribute. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/log_attribute/processor_definition.rs | ProcessorDefinition wiring for LogAttribute. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/log_attribute.rs | LogAttribute processor implementation using the Rust API. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/kamikaze_processor/tests.rs | Unit tests for failure/panic handling paths. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/kamikaze_processor/relationships.rs | Relationship definitions for KamikazeProcessor. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/kamikaze_processor/properties.rs | Property definitions for KamikazeProcessor. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/kamikaze_processor/processor_definition.rs | ProcessorDefinition wiring for KamikazeProcessor. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/kamikaze_processor.rs | Kamikaze test processor implementation. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/get_file/tests.rs | Unit tests for GetFile processor. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/get_file/relationships.rs | Relationship definitions for GetFile. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/get_file/properties.rs | Property definitions for GetFile. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/get_file/processor_definition.rs | ProcessorDefinition wiring for GetFile. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/get_file/output_attributes.rs | Output attribute definitions for GetFile. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/generate_flow_file/tests.rs | Unit tests for GenerateFlowFile behavior. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/generate_flow_file/relationships.rs | Relationship definitions for GenerateFlowFile. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/generate_flow_file/properties.rs | Property definitions for GenerateFlowFile. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/generate_flow_file/processor_definition.rs | ProcessorDefinition wiring for GenerateFlowFile. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/generate_flow_file.rs | GenerateFlowFile processor implementation. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/duplicate_text.rs | Streaming transform example processor. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/count_actual_logging.rs | Lazy logging test processor. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/asciify_german/tests.rs | Unit tests for streaming transform cancellation/behavior. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/asciify_german/relationships.rs | Relationship definitions for AsciifyGerman. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/asciify_german/processor_definition.rs | ProcessorDefinition wiring for AsciifyGerman. |
| minifi_rust/extensions/minifi_rs_playground/src/processors/asciify_german.rs | Streaming transform processor implementation. |
| minifi_rust/extensions/minifi_rs_playground/src/lib.rs | Declares and registers processors/controller services via the macro. |
| minifi_rust/extensions/minifi_rs_playground/src/controller_services/mod.rs | Controller service module wiring. |
| minifi_rust/extensions/minifi_rs_playground/src/controller_services/lorem_ipsum_controller_service/properties.rs | Property definitions for the lorem ipsum controller service. |
| minifi_rust/extensions/minifi_rs_playground/src/controller_services/lorem_ipsum_controller_service.rs | Controller service implementation generating lorem ipsum. |
| minifi_rust/extensions/minifi_rs_playground/src/controller_services/dummy_controller_service.rs | Dummy controller service for negative test cases. |
| minifi_rust/extensions/minifi_rs_playground/features/streaming.feature | Behave scenarios for streaming transforms. |
| minifi_rust/extensions/minifi_rs_playground/features/steps/steps.py | Custom Behave step implementations for new scenarios. |
| minifi_rust/extensions/minifi_rs_playground/features/metrics.feature | Behave scenarios validating metrics logging. |
| minifi_rust/extensions/minifi_rs_playground/features/lazy-logging.feature | Behave scenarios validating lazy logging behavior. |
| minifi_rust/extensions/minifi_rs_playground/features/error-handling.feature | Behave scenarios for error propagation and crash behavior. |
| minifi_rust/extensions/minifi_rs_playground/features/environment.py | Behave environment hooks to build/attach the Rust extension into containers. |
| minifi_rust/extensions/minifi_rs_playground/features/basic.feature | Basic Behave scenarios verifying extension loading and simple flows. |
| minifi_rust/extensions/minifi_rs_playground/Cargo.toml | Cargo manifest for the sample extension crate. |
| minifi_rust/extensions/minifi_rs_playground/.gitignore | Extension crate ignore rules. |
| minifi_rust/extensions/minifi_rs_playground/.cargo/config.toml | macOS linker flags for dynamic lookup in extensions. |
| minifi_rust/CMakeLists.txt | CMake integration for Rust via Corrosion and cargo test as a CTest. |
| minifi_rust/Cargo.toml | Workspace definition and profiles for panic behavior. |
| minifi_rust/.gitignore | Workspace ignore rules. |
| minifi_rust/.dockerignore | Docker ignore rules for the Rust workspace builds. |
| minifi_rust/.cargo/config.toml | Cargo aliases and env defaults for SDK/test execution. |
| docker/rockylinux/Dockerfile | Updates base image and installs rust tooling conditionally for Rust builds. |
| CMakeLists.txt | Adds minifi_rust subdirectory to the top-level build. |
| cmake/MiNiFiOptions.cmake | Adds MINIFI_RUST CMake option to control Rust integration. |
| .github/workflows/create-release-artifacts.yml | Builds Rust artifacts and uploads .so extension outputs. |
| .github/workflows/ci.yml | Adds Rust docker integration test job and Rust formatting check in linters. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1
to
+6
| fn main() { | ||
| println!("cargo:rerun-if-env-changed=DEP_MINIFI_BEHAVE_PATH"); | ||
| let behave_path = | ||
| std::env::var("DEP_MINIFI_BEHAVE_PATH").expect("DEP_MINIFI_BEHAVE_PATH is required"); | ||
| println!("cargo:rustc-env=MINIFI_BEHAVE_PATH={}", behave_path); | ||
| } |
Comment on lines
+114
to
+124
| impl<S> GetControllerService for S | ||
| where | ||
| S: ProcessContext, | ||
| { | ||
| fn get_controller_service<Cs>(&self, property: &Property) -> Result<Option<&Cs>, MinifiError> | ||
| where | ||
| Cs: EnableControllerService + ComponentIdentifier + 'static, | ||
| { | ||
| self.get_controller_service(property) | ||
| } | ||
| } |
Comment on lines
+70
to
+73
| unsafe { | ||
| let slice = std::slice::from_raw_parts(self.data as *const u8, self.length); | ||
| str::from_utf8(slice).map_err(|_| FfiConversionError::InvalidUtf8) | ||
| } |
Comment on lines
+88
to
+100
| fn get_destination_path<Ctx>(context: &Ctx) -> Result<PathBuf, MinifiError> | ||
| where | ||
| Ctx: GetProperty + GetAttribute, | ||
| { | ||
| let directory = context | ||
| .get_property(&properties::DIRECTORY)? | ||
| .expect("required property"); | ||
|
|
||
| let file_name = context | ||
| .get_attribute("filename")? | ||
| .unwrap_or("foo.txt".to_string()); // TODO fallback to UUID | ||
| Ok(PathBuf::from(directory + "/" + file_name.as_str())) | ||
| } |
Comment on lines
602
to
+633
| @@ -578,8 +625,14 @@ jobs: | |||
| continue-on-error: true | |||
| run: ./run_flake8.sh . | |||
|
|
|||
| - id: cargo_check | |||
| name: Cargo check | |||
| continue-on-error: true | |||
| run: cargo fmt --check | |||
| working-directory: minifi_rust | |||
|
|
|||
Comment on lines
116
to
120
| add_minifi_option(ENABLE_CONTROLLER "Enables the build of MiNiFi controller binary." ON) | ||
| add_minifi_option(ENABLE_LLAMACPP "Enables llama.cpp support." ON) | ||
| add_minifi_option(ENABLE_OPC "Instructs the build system to enable the OPC extension" ON) | ||
| add_minifi_option(MINIFI_RUST "Enables the build of rust based extension." ON) | ||
|
|
martinzink
commented
May 28, 2026
| add_minifi_option(ENABLE_CONTROLLER "Enables the build of MiNiFi controller binary." ON) | ||
| add_minifi_option(ENABLE_LLAMACPP "Enables llama.cpp support." ON) | ||
| add_minifi_option(ENABLE_OPC "Instructs the build system to enable the OPC extension" ON) | ||
| add_minifi_option(MINIFI_RUST "Enables the build of rust based extension." ON) |
Member
Author
There was a problem hiding this comment.
It might be better to keep it OFF since it requires rust compiler. (also just we probably need to handle this in bootstrap)
| from minifi_behave.core.minifi_test_context import MinifiTestContext | ||
|
|
||
|
|
||
| @when("the MiNiFi instance is started without assertions") |
Member
Author
There was a problem hiding this comment.
We should probably move these into the framework later, but I didnt want to touch anything outside of minifi_rust dir
| @@ -0,0 +1,13 @@ | |||
| use minifi_native::{Property, StandardPropertyValidator}; | |||
|
|
|||
| pub(crate) const LENGTH: Property = Property { | |||
Member
Author
There was a problem hiding this comment.
We should discuss what kind of style we want to follow. Single file per processor or splitting the properties, relationships, processor definitions into the mods folder
| @@ -0,0 +1,2 @@ | |||
| pub(crate) mod dummy_controller_service; | |||
| pub(crate) mod lorem_ipsum_controller_service; | |||
Member
Author
There was a problem hiding this comment.
test controller services, (multiple so we can test setting and quering the wrong one)
| @@ -0,0 +1,67 @@ | |||
| use crate::processors::asciify_german::relationships::FAILURE; | |||
Member
Author
There was a problem hiding this comment.
This is used to test streaming flow file transforms, with changing flowfiles sizes
| @@ -0,0 +1,20 @@ | |||
| [package] | |||
| name = "minifi_rs_playground" | |||
Member
Author
There was a problem hiding this comment.
This extension contains reimplentations of already existing processors and test only processors. These are not intended for production usage, therefore they probably don't require that much scrutiny in the review process.
If for some reason we ever want to ship something from this extension than that processor will have to be extracted and rereviewed.
| } | ||
|
|
||
| #[proc_macro_derive(NoAdvancedProcessorFeatures)] | ||
| pub fn derive_no_advanced_processor_features(input: TokenStream) -> TokenStream { |
Member
Author
There was a problem hiding this comment.
The API was simplified so there is only a single advanced prcessor feature (get_trigger_when_empty) so we should probably want to rename this
| @@ -0,0 +1,94 @@ | |||
| #!/usr/bin/env bash | |||
Member
Author
There was a problem hiding this comment.
This shell script was written partly using llm
Comment on lines
+8
to
+9
| minifi_rs_playground = { path = "../extensions/minifi_rs_playground" } | ||
| minifi_native_sys = { path = "../minifi_native_sys" } |
Member
Author
There was a problem hiding this comment.
we currently list minifi_rs_playground as a dependancy which is technically untrue (see unused_crate_dependencies = "allow"), but this allows a linux/windows user to run cargo behave and have all the required artifacts be built before behave tests.
(this also means we are building everything twice in the CI (once bare metal due to dependency and once in docker due to CI using alpine))
Member
Author
There was a problem hiding this comment.
We should probably remove this dependancy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:
Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically main)?
Is your initial contribution a single, squashed commit?
For code changes:
For documentation related changes:
Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.