feat(osgi): add opt-in Apache Felix plugin framework by kiran536 · Pull Request #6917 · opensearch-project/data-prepper

kiran536 · 2026-06-10T09:44:48Z

Add an opt-in plugin-loading mode backed by an embedded Apache Felix OSGi framework, enabled with the system property -Dplugin.framework=osgi. The default classpath loader remains the default when the flag is not set. When the flag is not set, plugin discovery is functionally identical to today: the same ServiceLoader-based providers are returned in the same order, the OSGi framework never starts, and no OSGi code path executes. The only added work in legacy mode is the construction of a couple of inert Spring beans.

The new plugin-framework-osgi module embeds the Felix lifecycle (FelixPluginManager), bootstraps it in OSGi mode through a Spring-managed runner (OsgiFrameworkRunner), and exposes a PluginProvider backed by the OSGi service registry (OsgiPluginRegistry). At startup in OSGi mode, StaticBundleLoader installs each legacy plugin JAR through BundleAdapter, which installs already-OSGi JARs directly and wraps legacy (non-OSGi) JARs by generating a versioned OSGi manifest. Wrapped plugins' @DataPrepperPlugin classes are registered as OSGi services (LegacyPluginBundleActivator). Data Prepper API packages are exported to bundles with semver version attributes read from data-prepper-api build metadata (DataPrepperOsgiPackages, DataPrepperApiVersion), consistent with the data-prepper-api backward-compatibility contract from #6607, so a plugin built on 2.15 resolves on a 2.16 host. DataPrepperOsgiPackages also exports the framework package org.opensearch.dataprepper.plugin.osgi so adapted bundles can load the LegacyPluginBundleActivator from the system bundle.

StaticBundleLoader installs, resolves, and starts bundles with fail-fast behavior, a startup summary log, and Micrometer metrics. BundleResolutionErrorTranslator turns Felix resolution failures into readable diagnostics, and BundleHealthCheck verifies framework, bundle, and classloader isolation state. BundleClassLoaderScope manages the thread context classloader at the plugin invocation boundary so ServiceLoader and SPI resolve against the bundle classloader. PluginHealthProbe is a seam for future functional health checks.

The OsgiFrameworkRunner bean is always instantiated by Spring but does nothing unless -Dplugin.framework=osgi is set, because it returns early in its @PostConstruct method. The Felix framework JAR ships on the runtime classpath as a small (<1MB) library with zero transitive dependencies, but it is not exercised in legacy mode.

Scope is limited to classloader isolation and deploy-time validation. Bundles install once at startup and stop once at shutdown, the same lifecycle as classpath plugins today. No production hot-reload is included. The PluginHotLoader helper is test-scoped only and not on the production classpath.

Wiring changes: settings.gradle adds the module, a Felix version-catalog entry, and the bnd plugin; build-resources.gradle adds the module to coreProjects; data-prepper-core declares a runtimeOnly dependency on the new module; PluginProviderLoader is made public with a registerProvider() hook and caches the framework-mode flag at construction; a test in data-prepper-plugin-framework's PluginProviderLoaderTest verifies the mode-caching behavior. Legacy behavior is unchanged. A new, permanent, minimal example plugin data-prepper-plugins/echo-processor is added to the build to demonstrate a minimal @DataPrepperPlugin and the SPI path.

Resolves #6760

Description

[Describe what this change achieves]

Issues Resolved

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has a documentation issue. Please link to it in this PR.
- New functionality has javadoc added
Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

dlvenable

Thanks @kiran536 for this contribution! I think this is the right direction, but also have some comments.

dlvenable · 2026-06-16T15:11:14Z

+
+        // 2. Register a mock plugin service in OSGi registry (mirrors LegacyPluginBundleActivator)
+        final Dictionary<String, Object> props = new Hashtable<>();
+        props.put(OsgiPluginRegistry.PLUGIN_NAME_PROPERTY, "test_echo");


The existing plugin framework tests already include some examples of loading test plugins. We should avoid creating test plugins that are public.

I suspect you took this approach to be able to load it from a library. One idea is that you can move echo processor under the data-prepper-test project. These will not automatically deploy into Data Prepper during a release.

dlvenable · 2026-06-16T15:12:28Z

+        if (dataFile != null) {
+            this.adaptedBundlesDir = dataFile;
+        } else {
+            this.adaptedBundlesDir = new File(System.getProperty("java.io.tmpdir"), "dp-osgi-adapted");


We should avoid using a temporary directory and stick with only running under the data-prepper directory.

dlvenable · 2026-06-16T15:14:21Z

+ * plugin code changes. Generates OSGi MANIFEST.MF headers from existing
+ * plugin metadata.
+ */
+public class BundleAdapter {


Rather than adapting at runtime, we should support this within the plugins themselves. Would an OSGi manifest cause any issues with how it works now? I'd think if no OSGi code runs there is no impact.

We should probably have a Gradle plugin that prepares Data Prepper plugins for OSGi. The existing Data Prepper plugins could use this Gradle plugin. And we could release this Gradle plugin for authors outside of the Data Prepper repository.

dlvenable · 2026-06-16T15:16:08Z

+ * The resolved version is converted to valid OSGi format (major.minor.micro[.qualifier]).
+ * Any {@code -SNAPSHOT} suffix is stripped or converted to an OSGi qualifier.
+ */
+final class DataPrepperApiVersion {


We already have DataPrepperVersion. Can we continue to use that? Why do we have a different one?

dlvenable · 2026-06-16T15:18:21Z

+            "org.opensearch.dataprepper.expression",
+            "org.opensearch.dataprepper.logging",
+            "org.opensearch.dataprepper.metrics",
+            "org.opensearch.dataprepper.model.acknowledgements",


This list is going to be hard to maintain. Can we implement this such that any org.opensearch.dataprepper.model is considered shared?

Better yet, could we say any package/class in org.opensearch.dataprepper is shared unless it is org.opensearch.dataprepper.plugins?

Long term, we should probably have a stronger package structure, but this is the current pattern we have.

dlvenable · 2026-06-16T15:19:05Z

+    private static Map<String, String> createDefaultConfig() {
+        final Map<String, String> config = new HashMap<>();
+        final long pid = ProcessHandle.current().pid();
+        final String cacheDir = System.getProperty("java.io.tmpdir") + File.separator


Let's try to use directories under the data-prepper directory.

We don't have a useful API, but you can see an example of how we do it now.

data-prepper/data-prepper-plugins/geoip-processor/src/main/java/org/opensearch/dataprepper/plugins/geoip/extension/MaxMindConfig.java

Line 51 in 4f3ab96

private String databaseDestination = System.getProperty("data-prepper.dir") + File.separator + "data" + File.separator + "geoip";

dlvenable · 2026-06-16T15:22:16Z

+@Named
+public class OsgiFrameworkRunner {
+    private static final Logger LOG = LoggerFactory.getLogger(OsgiFrameworkRunner.class);
+    static final String PLUGIN_FRAMEWORK_PROPERTY = "plugin.framework";


Are these OSGi properties? We should start any Data Prepper properties with data-prepper. for clarity and consistency.

dlvenable · 2026-06-16T16:11:21Z

+ * 3. Discovers it via OsgiPluginRegistry
+ * 4. Validates hot-load install/uninstall cycle
+ */
+class OsgiEndToEndTest {


Please rename this to OsgiIT to follow existing patterns.

Follow the approach taken in DefaultPluginFactoryIT to get a DefaultPluginFactory. Test loading plugins through that mechanism.

We should also have a way to be sure that it was loaded through OSGi instead of through the current mechanism.

Overall - we need to verify that this actually loads as part of an integration test even when the existing code is on the path.

dlvenable · 2026-06-16T16:15:20Z

+            "org.opensearch.dataprepper.typeconverter"
+    );
+
+    static final String SLF4J_PACKAGE = "org.slf4j";


I think we should also include Jakarta validation here as this is a key component between the two.

I also wonder about Jackson since we use Jackson annotations. Perhaps just the Jackson annotation packages would be appropriate (not the whole Jackson classpath).

github-actions · 2026-06-16T16:36:40Z

⚠️ License Header Violations Found

The following newly added files are missing required license headers:

data-prepper-plugins/echo-processor/build.gradle
data-prepper-plugins/echo-processor/src/main/resources/META-INF/data-prepper.plugins.properties
plugin-framework-osgi/build.gradle

Please add the appropriate license header to each file and push your changes.

See the license header requirements: https://github.com/opensearch-project/data-prepper/blob/main/CONTRIBUTING.md#license-headers

@PostConstruct

Add an opt-in plugin-loading mode backed by an embedded Apache Felix OSGi framework, enabled with -Ddata-prepper.plugin.framework=osgi. When the property is unset, Data Prepper uses the existing classpath loader. In that legacy mode plugin discovery is functionally identical to before: the same ServiceLoader providers are found in the same order, the OSGi framework never starts, no OSGi code path executes, and only a couple of inert Spring beans are constructed. Plugins are prepared for OSGi at build time rather than adapted at runtime. A new standalone, publishable Gradle plugin module data-prepper-gradle-plugins/osgi-plugin (plugin id org.opensearch.dataprepper.osgi) bakes an OSGi manifest into the plugin JAR at build time using bnd (biz.aQute.bnd). The module is publishable so plugin authors outside this repository can apply it; in-repo plugins opt in by applying the plugin id. There is no runtime adaptation and no BundleAdapter. At startup in OSGi mode StaticBundleLoader installs the pre-built bundle JARs directly, resolves and starts them with fail-fast behavior, a startup summary log, and Micrometer metrics. A JAR that lacks an OSGi manifest (no Bundle-SymbolicName) fails fast with an actionable message pointing the author to the Gradle plugin. The plugin-framework-osgi module embeds the Felix lifecycle (FelixPluginManager), bootstraps it in OSGi mode through a Spring-managed runner (OsgiFrameworkRunner), and exposes a PluginProvider backed by the OSGi service registry (OsgiPluginRegistry). Baked plugins declare their @DataPrepperPlugin packages through the DataPrepper-Plugin-Classes manifest header, which the Gradle plugin generates from META-INF/data-prepper.plugins.properties, and LegacyPluginBundleActivator registers those classes as OSGi services. Data Prepper API packages are exported to bundles with semver version attributes. The exported package set is generated at build time by a Gradle task that writes META-INF/osgi-shared-packages.properties listing every org.opensearch.dataprepper.* package except org.opensearch.dataprepper.plugins.*, so there is no hand-maintained list. The version comes from the existing data-prepper-api DataPrepperVersion, formatted into the OSGi three-part string by a small package-private toOsgiVersion helper; the separate DataPrepperApiVersion class was removed. This is consistent with the data-prepper-api backward-compatibility contract from opensearch-project#6607, so a plugin built on 2.15 resolves on a 2.16 host. The shared set also includes jakarta.validation, jakarta.validation.constraints, and com.fasterxml.jackson.annotation (annotations only: jackson-databind and jackson-core are intentionally not exported, preserving isolation), plus org.slf4j and the framework package org.opensearch.dataprepper.plugin.osgi so bundles can load LegacyPluginBundleActivator from the system bundle. Felix framework storage resolves under System.getProperty("data-prepper.dir")/data/osgi/ (felix-cache-<pid>), matching the MaxMindConfig idiom, with a java.io.tmpdir fallback when the property is unset such as in tests. It does not use the system temp directory by default. BundleResolutionErrorTranslator turns Felix resolution failures into readable diagnostics. BundleHealthCheck verifies framework, bundle, and classloader isolation. BundleClassLoaderScope manages the thread context classloader at the plugin invocation boundary so ServiceLoader and SPI lookups resolve against the bundle classloader. PluginHealthProbe is a seam for future functional health checks. The OsgiFrameworkRunner bean is always instantiated by Spring but returns early in @PostConstruct unless data-prepper.plugin.framework=osgi. The Felix framework JAR ships on the runtime classpath as a small (<1MB) library with zero transitive dependencies and is not exercised in legacy mode. Scope is limited to classloader isolation and deploy-time validation. Bundles install once at startup and stop at shutdown, the same lifecycle as classpath plugins. There is no production hot-reload; PluginHotLoader is test-scoped only. Data Prepper-owned system properties use the data-prepper. namespace: data-prepper.plugin.framework and data-prepper.plugin.bundles.dir. The genuine OSGi and Felix properties (org.osgi.*, felix.*) are unchanged. Testing: an integration test OsgiIT in a new integrationTest source set under plugin-framework-osgi builds a DefaultPluginFactory like DefaultPluginFactoryIT, loads a test plugin through it, and proves the load happened through OSGi rather than the classpath. It asserts that FrameworkUtil.getBundle is non-null, the plugin classloader is a BundleReference, the classloader is not the application classloader, and the application classloader throws ClassNotFoundException for the plugin class because the plugin is supplied only as a bundle JAR and never on the application classpath. The Gradle plugin has unit and functional (GradleRunner) tests. Wiring changes: settings.gradle adds the plugin-framework-osgi module, the new data-prepper-gradle-plugins/osgi-plugin (consumed via includeBuild in pluginManagement), the data-prepper-test/osgi-test-plugin subproject, a Felix version-catalog entry, and the bnd plugin. build-resources.gradle adds plugin-framework-osgi to coreProjects. data-prepper-core declares a runtimeOnly dependency on the new module. PluginProviderLoader is made public with a registerProvider() hook and caches the framework-mode flag at construction. DefaultPluginFactory evaluates plugin providers lazily so OSGi-registered providers are visible. Tests verify these changes, and legacy behavior is unchanged. The example and test plugin is data-prepper-test/osgi-test-plugin, not a released data-prepper-plugins module. It demonstrates a minimal @DataPrepperPlugin built as an OSGi bundle through the new Gradle plugin. Resolves opensearch-project#6760 Signed-off-by: Kiran Kumar Veeravelly <veeravkk@amazon.com>

kiran536 requested review from KarstenSchnitter, Zhangxunmt, dinujoh, divbok, dlvenable, graytaylor0, kkondaka, oeyh, san81, sb2k16, srikanthjg and srikanthpadakanti as code owners June 10, 2026 09:44

kiran536 force-pushed the feat/osgi-plugin-framework branch 2 times, most recently from 4c0085d to 56efa9c Compare June 15, 2026 03:00

dlvenable reviewed Jun 16, 2026

View reviewed changes

kiran536 force-pushed the feat/osgi-plugin-framework branch 2 times, most recently from 3d1ff30 to 1f5a383 Compare June 17, 2026 12:23

kiran536 force-pushed the feat/osgi-plugin-framework branch from 1f5a383 to 196ddd0 Compare June 17, 2026 12:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(osgi): add opt-in Apache Felix plugin framework#6917

feat(osgi): add opt-in Apache Felix plugin framework#6917
kiran536 wants to merge 1 commit into
opensearch-project:mainfrom
kiran536:feat/osgi-plugin-framework

kiran536 commented Jun 10, 2026

Uh oh!

dlvenable left a comment

Uh oh!

dlvenable Jun 16, 2026

Uh oh!

dlvenable Jun 16, 2026

Uh oh!

dlvenable Jun 16, 2026

Uh oh!

dlvenable Jun 16, 2026

Uh oh!

dlvenable Jun 16, 2026

Uh oh!

dlvenable Jun 16, 2026

Uh oh!

dlvenable Jun 16, 2026

Uh oh!

dlvenable Jun 16, 2026

Uh oh!

dlvenable Jun 16, 2026

Uh oh!

github-actions Bot commented Jun 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

kiran536 commented Jun 10, 2026

Description

Issues Resolved

Check List

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 16, 2026

⚠️ License Header Violations Found

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants