MINIFICPP-2807 Fix Conan build

fgerlits · szaszm · commit 7e0d31bfdea3 · 2026-05-26T20:05:41.000+02:00
Updated dependency versions and added small fixes. A few extensions still don't build with Conan, these are disabled in conanfile.py. TODO (in upcoming PRs): * enable the remaining extensions * use Conan in CI * integrate Conan into bootstrap. Closes #2181 Signed-off-by: Marton Szasz <szaszm@apache.org>
diff --git a/CONAN.md b/CONAN.md
@@ -36,26 +36,36 @@ source, it uses CMake to install the third party external libraries that are ena
 
 ## Build MiNiFi C++ with Conan
 
-To build MiNiFi using conan, first we install conan version 2, then we create a **default** conan profile that will later be ignored for our custom conan profile, create a MINIFI_HOME environment variable, then we install prebuilt conan packages representing the MiNiFi external libraries and finally we compile MiNiFi.
+### Install Conan
 
 ~~~bash
-sudo pip install --force-reinstall -v "conan==2.0.17"
+sudo pip install --force-reinstall -v "conan==2.28.1"
 
 # create a "default" conan profile, so conan has it on record in ~/.conan2/, before using your own custom profile.
 conan profile detect
+~~~
+
+### Create Custom RocksDB Conan Package
+The default RocksDB conan package is built with -fno-rtti, which makes it incompatible with MiNiFi. So we need to create a custom RocksDB conan package that is built with -frtti.
+
+~~~bash
+cd $HOME/nifi-minifi-cpp/thirdparty/rocksdb/all
+
+conan create . --user=minifi --channel=develop --test-folder="" --version=11.1.1 --profile=../../../etc/conan/profiles/release-linux
+~~~
+
+### Build MiNiFi
 
+~~~bash
 # conanfile.py is in root dir of MiNiFi C++ project
 cd $HOME/nifi-minifi-cpp
 
-# create MINIFI_HOME env variable for binary executable minifi
-export MINIFI_HOME=$(pwd)
-
 # install conan packages for MiNiFi C++ using conanfile.py invoking Conan
 # since we created default profile earlier, we can override it with our own minifi profile
-conan install . --build=missing --output-folder=build_conan -pr=etc/conan/profiles/release-linux
+conan install . --build=missing --output-folder=build_conan --profile=etc/conan/profiles/release-linux
 
 # build MiNiFi C++ using conanfile.py invoking Conan & CMake
-conan build . --output-folder=build_conan -pr=etc/conan/profiles/release-linux
+conan build . --output-folder=build_conan --profile=etc/conan/profiles/release-linux
 ~~~
 
 - **NOTE**: After building MiNiFi, we must have the MINIFI_HOME environment variable created in order to successfully run the minifi binary executable.
@@ -76,6 +86,10 @@ popd # build_conan/
 
 ~~~bash
 # verify we can run minifi binary executable
+cd $HOME/nifi-minifi-cpp
+
+export MINIFI_HOME=$(pwd)
+
 ./build_conan/bin/minifi
 ~~~
 
@@ -85,7 +99,7 @@ To create a MiNiFi package, we will follow the similar steps we took to build Mi
 
 ~~~bash
 # make sure to install conan2 for your environment
-sudo pip install --force-reinstall -v "conan==2.0.17"
+sudo pip install --force-reinstall -v "conan==2.28.1"
 
 # create a "default" conan profile, so conan has it on record, before using your own custom profile. Gets created in ~/.conan2/
 conan profile detect
@@ -99,10 +113,10 @@ export MINIFI_HOME=$(pwd)
 # install conan packages for MiNiFi C++ using conanfile.py invoking Conan
 # since we created default profile earlier, we can override it with our own minifi profile
 # make sure path is correct
-conan install . --build=missing --output-folder=build_conan -pr=etc/conan/profiles/release-linux
+conan install . --build=missing --output-folder=build_conan --profile=etc/conan/profiles/release-linux
 
 # create MiNiFi C++ conan package using conanfile.py invoking Conan & CMake
-conan create . --user=minifi --channel=develop -pr=etc/conan/profiles/release-linux
+conan create . --user=minifi --channel=develop --profile=etc/conan/profiles/release-linux
 ~~~
 
 - **NOTE**: When we tell conan to create the MiNiFi conan package, conan first installs prebuilt conan packages, then it compiles MiNiFi from source inside `~/.conan2/p/b/minif<UUID>/b`, and then it copies over MiNiFi's libraries and its binary executables into the conan package folder `~/.conan2/p/b/minif<UUID>/p`. Once we have the MiNiFi conan package, we can integrate it into other C++ infrastructure using CMake.
@@ -123,15 +137,3 @@ To have a more consistent quick build process for MiNiFi, we can use conan versi
 There are multiple benefits of having MiNiFi prebuilt conan packages. We can upload these MiNiFi conan packages to a conan repository like jfrog for version management. We can easily integrate MiNiFi's edge data pipeline features into other C++ software infrastructure using conan's CMake support. We can still use MiNiFi for edge data collection from the IoT devices embedded on robotic systems. We can integrate MiNiFi into self-driving cars (sensor examples: cameras, lidar, radar, inertial measurement unit (IMU), electronic speed controller (ESC), steering servo, etc), into medical imaging robots (sensor examples: depth cameras, ultrasound, gamma detector, force/torque sensor, joint position sensor, etc) or some other real-time robotic system.
 
 By leveraging MiNiFi as a conan package, we can leverage MiNiFi that comes with the best practices of building data pipelines from NiFi and bring them into existing C++ real-time robotics infrastructure. Some teams across companies typically have their own custom edge data pipelines that process data for the different events to eventually perform actions on that data. As an alternative to all these companies and their teams having their own custom edge data pipeline libraries, MiNiFi C++, which is like a headless NiFi, can provide a more consistent standard approach for team's to build edge pipelines. Through all stages of the edge data pipelines, MiNiFi can still provide telemetry to NiFi instances running in the cloud.
-
-## Appendix
-
-### Create Custom RocksDB Conan Package
-
-~~~bash
-pushd nifi-minifi-cpp/thirdparty/rocksdb/all
-
-conan create . --user=minifi --channel=develop --version=8.10.2 -pr=../../../etc/conan/profiles/release-linux
-
-popd
-~~~
diff --git a/cmake/BundledRocksDB.cmake b/cmake/BundledRocksDB.cmake
@@ -19,8 +19,8 @@ function(use_bundled_rocksdb SOURCE_DIR BINARY_DIR)
     message("Using bundled RocksDB")
 
     if (NOT WIN32)
-        include(Zstd)
-        include(LZ4)
+        include(GetZstd)
+        include(GetLZ4)
     endif()
 
     set(PATCH_FILE "${SOURCE_DIR}/thirdparty/rocksdb/all/patches/dboptions_equality_operator.patch")
diff --git a/cmake/Couchbase.cmake b/cmake/Couchbase.cmake
@@ -17,8 +17,8 @@
 
 include(FetchContent)
 
-include(fmt)
-include(Spdlog)
+include(GetFmt)
+include(GetSpdlog)
 include(Asio)
 
 set(COUCHBASE_CXX_CLIENT_BUILD_STATIC ON CACHE BOOL "" FORCE)
diff --git a/cmake/Fetchlibrdkafka.cmake b/cmake/Fetchlibrdkafka.cmake
@@ -16,9 +16,9 @@
 # under the License.
 
 include(FetchContent)
-include(Zstd)
 
-include(LZ4)
+include(GetZstd)
+include(GetLZ4)
 
 set(WITH_SSL "ON" CACHE STRING "" FORCE)
 set(WITH_SASL "OFF" CACHE STRING "" FORCE)
@@ -46,7 +46,10 @@ FetchContent_Declare(libkafka
 FetchContent_MakeAvailable(libkafka)
 
 get_target_property(ZSTD_INCLUDE_DIRS zstd::zstd INTERFACE_INCLUDE_DIRECTORIES)
-get_target_property(LZ4_INCLUDE_DIRS lz4::lz4 INCLUDE_DIRECTORIES)
+
+if(NOT DEFINED LZ4_INCLUDE_DIRS)
+    get_target_property(LZ4_INCLUDE_DIRS lz4::lz4 INCLUDE_DIRECTORIES)
+endif()
 
 target_include_directories(rdkafka SYSTEM PRIVATE ${ZSTD_INCLUDE_DIRS})
 target_include_directories(rdkafka SYSTEM PRIVATE ${LZ4_INCLUDE_DIRS})
diff --git a/cmake/GetLZ4.cmake b/cmake/GetLZ4.cmake
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+if(MINIFI_LZ4_SOURCE STREQUAL "CONAN")
+    message("Using Conan to install lz4")
+    find_package(lz4 REQUIRED)
+    if(NOT TARGET lz4::lz4)
+        add_library(lz4::lz4 ALIAS LZ4::liblz4_static)
+    endif()
+    if(NOT TARGET LZ4::LZ4)
+        add_library(LZ4::LZ4 ALIAS lz4::lz4)
+    endif()
+    set(LZ4_LIBRARIES "${lz4_LIBRARIES}" CACHE STRING "" FORCE)
+    set(LZ4_INCLUDE_DIRS "${lz4_INCLUDE_DIRS}" CACHE STRING "" FORCE)
+elseif(MINIFI_LZ4_SOURCE STREQUAL "BUILD")
+    message("Using CMake to build lz4 from source")
+    include(LZ4)
+endif()
diff --git a/cmake/GetLibArchive.cmake b/cmake/GetLibArchive.cmake
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+if(MINIFI_LIBARCHIVE_SOURCE STREQUAL "CONAN")
+    message("Using Conan to install LibArchive")
+    find_package(LibArchive REQUIRED)
+elseif(MINIFI_LIBARCHIVE_SOURCE STREQUAL "BUILD")
+    message("Using CMake to build LibArchive from source")
+    include(BundledLibArchive)
+    use_bundled_libarchive(${CMAKE_SOURCE_DIR} ${CMAKE_BINARY_DIR})
+endif()
diff --git a/cmake/GetZstd.cmake b/cmake/GetZstd.cmake
@@ -15,13 +15,15 @@
 # specific language governing permissions and limitations
 # under the License.
 
-function(get_zstd)
-    if(MINIFI_ZSTD_SOURCE STREQUAL "CONAN")
-        message("Using Conan to install zstd")
-        find_package(zstd REQUIRED)
+if(MINIFI_ZSTD_SOURCE STREQUAL "CONAN")
+    message("Using Conan to install zstd")
+    find_package(zstd REQUIRED)
+    if(NOT TARGET zstd::zstd)
         add_library(zstd::zstd ALIAS zstd::libzstd_static)
-    elseif(MINIFI_ZSTD_SOURCE STREQUAL "BUILD")
-        message("Using CMake to build zstd from source")
-        include(Zstd)
     endif()
-endfunction(get_zstd)
+    set(ZSTD_LIBRARIES "${zstd_LIBRARIES}" CACHE STRING "" FORCE)
+    set(ZSTD_INCLUDE_DIRS "${zstd_INCLUDE_DIRS}" CACHE STRING "" FORCE)
+elseif(MINIFI_ZSTD_SOURCE STREQUAL "BUILD")
+    message("Using CMake to build zstd from source")
+    include(Zstd)
+endif()
diff --git a/cmake/MiNiFiOptions.cmake b/cmake/MiNiFiOptions.cmake
@@ -135,10 +135,12 @@ set(STRICT_GSL_CHECKS "DEBUG_ONLY" CACHE STRING "Contract checking and assertion
 list(APPEND STRICT_GSL_CHECKS_Values AUDIT ON DEBUG_ONLY OFF)
 set_property(CACHE STRICT_GSL_CHECKS PROPERTY STRINGS ${STRICT_GSL_CHECKS_Values})
 
+add_minifi_multi_option(MINIFI_PACKAGING_TYPE "Selects the packaging format for the final artifact." "TGZ;RPM" "TGZ")
+
 # BUILD: Fetch and build from source using CMake FetchContent or ExternalProject
 # SYSTEM: Use find_package to use the system version
 # CONAN: Use Conan packages
-add_minifi_multi_option(MINIFI_PACKAGING_TYPE "Selects the packaging format for the final artifact." "TGZ;RPM" "TGZ")
+add_minifi_multi_option(MINIFI_LZ4_SOURCE "Retrieves lz4 from provided source" "BUILD;SYSTEM;CONAN" "BUILD")
 add_minifi_multi_option(MINIFI_LIBCURL_SOURCE "Retrieves LibCURL from provided source" "BUILD;SYSTEM;CONAN" "BUILD")
 add_minifi_multi_option(MINIFI_OPENSSL_SOURCE "Retrieves OpenSSL from provided source" "BUILD;SYSTEM;CONAN" "BUILD")
 add_minifi_multi_option(MINIFI_ZLIB_SOURCE "Retrieves ZLib from provided source" "BUILD;SYSTEM;CONAN" "BUILD")
@@ -150,6 +152,7 @@ add_minifi_multi_option(MINIFI_LIBXML2_SOURCE "Retrieves LibXml2 from provided s
 add_minifi_multi_option(MINIFI_FMT_SOURCE "Retrieves Fmt from provided source" "BUILD;SYSTEM;CONAN" "BUILD")
 add_minifi_multi_option(MINIFI_SPDLOG_SOURCE "Retrieves Spdlog from provided source" "BUILD;SYSTEM;CONAN" "BUILD")
 add_minifi_multi_option(MINIFI_CATCH2_SOURCE "Retrieves Catch2 from provided source" "BUILD;SYSTEM;CONAN" "BUILD")
+add_minifi_multi_option(MINIFI_LIBARCHIVE_SOURCE "Retrieves libarchive from provided source" "BUILD;SYSTEM;CONAN" "BUILD")
 
 # Docker options
 
diff --git a/conanfile.py b/conanfile.py
@@ -7,9 +7,14 @@
 
 required_conan_version = ">=2.0"
 
-shared_requires = ("openssl/3.2.1", "libcurl/8.4.0", "civetweb/1.16", "libxml2/2.12.6", "fmt/10.2.1", "spdlog/1.14.0", "catch2/3.5.4", "zlib/1.2.11", "zstd/1.5.2", "bzip2/1.0.8", "rocksdb/8.10.2@minifi/develop")
+shared_requires = ("lz4/1.10.0", "openssl/3.6.2", "libcurl/8.20.0", "civetweb/1.16", "libxml2/2.15.3", "fmt/12.1.0", "spdlog/1.17.0", "catch2/3.14.0", "zlib/1.3.2", "zstd/1.5.7",
+                   "bzip2/1.0.8", "rocksdb/11.1.1@minifi/develop", "libarchive/3.8.7")
 
-shared_sources = ("CMakeLists.txt", "libminifi/*", "extensions/*", "minifi_main/*", "nanofi/*", "bin/*", "bootstrap/*", "cmake/*", "conf/*", "controller/*", "encrypt-config/*", "etc/*", "examples/*", "packaging/msi/*", "thirdparty/*", "docker/*", "LICENSE", "NOTICE", "README.md", "C2.md", "CONFIGURE.md", "CONTRIBUTING.md", "CONTROLLERS.md", "EXPRESSIONS.md", "Extensions.md", "JNI.md", "METRICS.md", "OPS.md", "PROCESSORS.md", "ThirdParties.md", "Windows.md", "CPPLINT.cfg", "generateVersion.sh", "run_clang_tidy.sh", "run_clang_tidy.sh", "run_flake8.sh", "run_shellcheck.sh", "versioninfo.rc.in")
+shared_sources = ("CMakeLists.txt", "libminifi/*", "extensions/*", "minifi_main/*", "behave_framework/*", "bin/*", "bootstrap/*", "cmake/*", "conf/*", "controller/*", "core-framework/*",
+                  "docs/*", "encrypt-config/*", "etc/*", "examples/*", "extension-framework/*", "fips/*", "minifi-api/*", "packaging/*", "thirdparty/*", "docker/*", "LICENSE", "NOTICE",
+                  "README.md", "C2.md", "CONAN.md", "CONFIGURE.md", "CONTRIBUTING.md", "CONTROLLERS.md", "EXPRESSIONS.md", "Extensions.md", "METRICS.md", "OPS.md", "PARAMETER_PROVIDERS.md",
+                  "PROCESSORS.md", "SITE_TO_SITE.md", "ThirdParties.md", "Windows.md", "CPPLINT.cfg", "generateVersion.bat", "generateVersion.sh", "run_clang_tidy.sh", "run_flake8.sh",
+                  "run_shellcheck.sh", "versioninfo.rc.in")
 
 
 class MiNiFiCppMain(ConanFile):
@@ -28,6 +33,7 @@ class MiNiFiCppMain(ConanFile):
     def generate(self):
         tc = CMakeToolchain(self)
 
+        tc.variables["MINIFI_LZ4_SOURCE"] = "CONAN"
         tc.variables["MINIFI_LIBCURL_SOURCE"] = "CONAN"
         tc.variables["MINIFI_OPENSSL_SOURCE"] = "CONAN"
         tc.variables["MINIFI_ZLIB_SOURCE"] = "CONAN"
@@ -39,13 +45,10 @@ def generate(self):
         tc.variables["MINIFI_FMT_SOURCE"] = "CONAN"
         tc.variables["MINIFI_SPDLOG_SOURCE"] = "CONAN"
         tc.variables["MINIFI_CATCH2_SOURCE"] = "CONAN"
+        tc.variables["MINIFI_LIBARCHIVE_SOURCE"] = "CONAN"
 
-        tc.variables["SKIP_TESTS"] = "OFF"
-        tc.variables["ENABLE_CIVET"] = "ON"
-        tc.variables["ENABLE_LIBARCHIVE"] = "OFF"
         tc.variables["ENABLE_AWS"] = "OFF"
-        tc.variables["ENABLE_SQL"] = "OFF"
-        tc.variables["ENABLE_GCP"] = "OFF"
+        tc.variables["ENABLE_LLAMACPP"] = "OFF"
         tc.variables["ENABLE_LUA_SCRIPTING"] = "OFF"
 
         tc.generate()
diff --git a/etc/conan/profiles/release-linux b/etc/conan/profiles/release-linux
@@ -3,10 +3,11 @@ os=Linux
 arch=x86_64
 build_type=Release
 compiler=gcc
-compiler.cppstd=20
+compiler.cppstd=23
 compiler.libcxx=libstdc++11
-compiler.version=11
+compiler.version=13
 [conf]
 tools.system.package_manager:mode=install
 tools.system.package_manager:sudo=True
+tools.cmake.cmaketoolchain:generator=Ninja
 [options]
diff --git a/etc/conan/profiles/release-windows b/etc/conan/profiles/release-windows
@@ -0,0 +1,12 @@
+[settings]
+os=Windows
+arch=x86_64
+build_type=Release
+compiler=msvc
+compiler.cppstd=23
+compiler.runtime=dynamic
+compiler.version=194
+[conf]
+tools.cmake.cmaketoolchain:generator=Ninja
+tools.cmake.cmaketoolchain:extra_variables={"CMAKE_POLICY_DEFAULT_CMP0091": "NEW"}
+[options]
diff --git a/extensions/libarchive/CMakeLists.txt b/extensions/libarchive/CMakeLists.txt
@@ -26,8 +26,7 @@ if (ENABLE_LZMA)
     list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake/liblzma/dummy")
 endif()
 
-include(BundledLibArchive)
-use_bundled_libarchive(${CMAKE_SOURCE_DIR} ${CMAKE_BINARY_DIR})
+include(GetLibArchive)
 
 include(${CMAKE_SOURCE_DIR}/extensions/ExtensionHeader.txt)
 
diff --git a/thirdparty/rocksdb/all/conandata.yml b/thirdparty/rocksdb/all/conandata.yml
@@ -1,15 +1,10 @@
 sources:
-  "8.10.2":
-    url: "https://github.com/facebook/rocksdb/archive/refs/tags/v8.10.2.tar.gz"
-    sha256: "44b6ec2f4723a0d495762da245d4a59d38704e0d9d3d31c45af4014bee853256"
+  "11.1.1":
+    url: "https://github.com/facebook/rocksdb/archive/refs/tags/v11.1.1.tar.gz"
+    sha256: "63f11183fe40725a0e89a9e392f2c86c94b90064f1d95f5173a02cfe40de13f7"
 patches:
-  "8.10.2":
-    - patch_file: "patches/arm7.patch"
-      patch_description: "Fix for ARM7"
-      patch_type: "portability"
-      patch_source: "https://github.com/apache/nifi-minifi-cpp/commit/36244376bb46c5c85657c6dfe92a6a0c4cea650a"
+  "11.1.1":
     - patch_file: "patches/dboptions_equality_operator.patch"
-      patch_description: "Fix for missing equality operator in DBOptions and make DBOptions configurable"
-      patch_type: "portability"
-      patch_source: "https://github.com/apache/nifi-minifi-cpp/commit/545236fd06b613f2d2b38f74ff9ee85df9190d59"
-    - patch_file: "patches/cstdint.patch"
+      patch_description: "Add equality operator to DBOptions"
+      patch_type: "add-missing-functionality"
+      patch_source: "https://github.com/apache/nifi-minifi-cpp/pull/1731"
diff --git a/thirdparty/rocksdb/all/conanfile.py b/thirdparty/rocksdb/all/conanfile.py
@@ -87,21 +87,21 @@ def layout(self):
 
     def requirements(self):
         if self.options.with_gflags:
-            self.requires("gflags/2.2.2")
+            self.requires("gflags/2.3.0")
         if self.options.with_snappy:
-            self.requires("snappy/1.1.10")
+            self.requires("snappy/1.2.1")
         if self.options.with_lz4:
-            self.requires("lz4/1.9.4")
+            self.requires("lz4/1.10.0")
         if self.options.with_zlib:
-            self.requires("zlib/[>=1.2.11 <2]")
+            self.requires("zlib/[>=1.3.2 <2]")
         if self.options.with_bz2:
             self.requires("bzip2/1.0.8")
         if self.options.with_zstd:
-            self.requires("zstd/1.5.2")
+            self.requires("zstd/1.5.7")
         if self.options.get_safe("with_tbb"):
-            self.requires("onetbb/2021.10.0")
+            self.requires("onetbb/2023.0.0")
         if self.options.with_jemalloc:
-            self.requires("jemalloc/5.3.0")
+            self.requires("jemalloc/5.3.1")
 
     def validate(self):
         if self.settings.compiler.get_safe("cppstd"):
@@ -174,7 +174,7 @@ def generate(self):
 
     def build(self):
         for patch_data in self.conan_data.get("patches", {}).get(self.version, []):
-            patch(self, patch_file=patch_data["patch_file"], base_path=self.source_folder, strip=1, fuzz=True)
+            patch(self, patch_file=patch_data["patch_file"], base_path=self.source_folder, strip=0, fuzz=True)
         cmake = CMake(self)
         cmake.configure()
         cmake.build()
