[Feature] Apache Felix OSGi Plugin Framework Integration

[kiran536]

[RFC] Apache Felix OSGi Plugin Framework Integration

Description

This RFC proposes integrating Apache Felix as the OSGi runtime for Data Prepper's
plugin system. It builds on the goals outlined in
#321 (Plugin Redesign)
and #1543 (Run plugins in their own classloader),
delivering classloader isolation, dynamic plugin loading, and a modular
architecture while maintaining full backward compatibility with existing plugins.

What is the problem?

Data Prepper's current plugin architecture has several limitations:

1. No classloader isolation. All plugins share the application classpath. If
   two plugins depend on different versions of the same library (e.g., Jackson
   2.14 vs 2.17), one will shadow the other, causing runtime failures.

2. No dynamic loading. Plugins must be present at startup. There is no way
   to install, update, or remove a plugin without restarting the entire Data
   Prepper process.

3. Monolithic distribution. As noted in [RFC] Plugin Redesign #321, all plugins are embedded in a
   single distribution. Users running in air-gapped environments must build
   custom distributions to include only the plugins they need.

4. No lifecycle management. Plugins are instantiated and garbage-collected
   but have no formal start/stop lifecycle beyond what the pipeline provides.
   There is no framework-level health monitoring of plugin state.

5. Dependency conflicts at scale. With 100+ plugin modules, the flat
   classpath makes it increasingly difficult to manage transitive dependency
   versions without conflicts.

What are you proposing?

Introduce an embedded Apache Felix OSGi framework as an opt-in plugin
loading mode alongside the existing classpath-based loading. The two modes are
controlled by a single Java system property (set via -D JVM flag):

-Dplugin.framework=osgi    # Enable OSGi mode
-Dplugin.framework=legacy  # Default, existing behavior

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                     DualModePluginManager                       │
│              (feature flag: plugin.framework=legacy|osgi)       │
├────────────────────────────┬────────────────────────────────────┤
│       LEGACY MODE          │           OSGI MODE               │
│                            │                                    │
│  ClasspathPluginProvider   │  FelixPluginManager               │
│    └─ Reflections scan     │    └─ Embedded Felix Framework    │
│                            │       └─ OSGi Service Registry   │
│  ClasspathExtension        │                                    │
│  ClassProvider             │  OsgiPluginProvider               │
│    └─ Reflections scan     │    └─ Queries OSGi registry      │
│                            │                                    │
│                            │  OsgiExtensionClassProvider       │
│                            │    └─ Resolves extensions via OSGi│
│                            │                                    │
│                            │  SpringOsgiBridge                 │
│                            │    └─ BeanFactory as OSGi service │
│                            │                                    │
│                            │  PluginHotLoader                  │
│                            │    └─ Runtime install/uninstall   │
│                            │                                    │
│                            │  BundleHealthCheck                │
│                            │    └─ Status + isolation verify   │
├────────────────────────────┴────────────────────────────────────┤
│                    SHARED (both modes)                           │
│  DefaultPluginFactory → PluginCreator → ComponentPlugin         │
│                         ArgumentsContext                        │
│  PluginConfigurationConverter, VariableExpander                 │
│  PluginBeanFactoryProvider (isolated Spring contexts)           │
│  ExtensionLoader → ExtensionsApplier                           │
└─────────────────────────────────────────────────────────────────┘

Key Design Decisions

1. Controlled Classloader Delegation

Felix is configured with FRAMEWORK_BUNDLE_PARENT = FRAMEWORK so each plugin
bundle gets its own classloader. Shared packages are explicitly exported from
the system bundle:

• org.opensearch.dataprepper.model.* — Plugin API (14 sub-packages)
• org.opensearch.dataprepper.metrics — Metrics API
• org.opensearch.dataprepper.plugin — Plugin SPI
• org.springframework.beans.factory + org.springframework.context — Spring DI
• org.slf4j + org.slf4j.event — Logging

These are explicit sub-package entries (not wildcards); adding a new API package requires updating the export list.

Boot delegation covers JDK internals (sun.*, com.sun.*, javax.*, org.xml.*, org.w3c.*, jdk.*). This
means plugins can bundle their own versions of libraries like Jackson or Guava
without conflicting with other plugins.

2. Legacy Plugin Bridge

Existing plugins require zero code changes. LegacyPluginBundleActivator reads
META-INF/data-prepper.plugins.properties, scans for @DataPrepperPlugin
annotated classes using Reflections, and registers them as OSGi services.
LegacyExtensionBundleActivator does the same for ExtensionPlugin subclasses.

3. OSGi Service Convention

Plugin descriptors are registered as String services with properties pluginName, pluginType, and pluginClass. Plugin classes are resolved via Bundle.loadClass() to ensure correct classloader delegation.

4. Spring DI Integration

SpringOsgiBridge registers the Spring BeanFactory as an OSGi service. Plugin
bundles can look up this service to resolve Spring-managed beans. The existing
PluginBeanFactoryProvider isolated-context hierarchy (core → shared → plugin)
is preserved.

5. Dynamic Plugin Loading

PluginHotLoader supports runtime install(location) / uninstall(location)
with:

• Rollback on start failure (broken bundles are automatically uninstalled)
• BundleListener for event tracking
• uninstallAll() for bulk cleanup during shutdown
• Idempotent operations (re-installing an active bundle is a no-op)

6. Resilient Lifecycle

• FelixPluginManager.stop() uses waitForStop(30s) with timeout detection
• DualModePluginManager.shutdown() tears down in reverse order, catching
  errors per component to ensure all components get a chance to clean up
• All providers guard against framework-not-running state and return empty
  results rather than throwing

7. Health Monitoring

BundleHealthCheck provides:

• getBundleStatuses() — id, symbolicName, version, state for each bundle
• verifyClassloaderIsolation() — confirms each bundle has its own classloader
• isHealthy() — detects bundles stuck in INSTALLED (unresolved) state

OSGi Bundle Convention Plugin

A Gradle convention plugin (data-prepper.osgi-bundle) generates OSGi manifest
headers for any plugin module:

plugins {
    id 'data-prepper.osgi-bundle'
}

This adds Bundle-ManifestVersion, Bundle-SymbolicName, Bundle-Version,
Export-Package, and Import-Package headers to the JAR manifest.

What are your assumptions or prerequisites?

• Apache Felix Framework 7.0.5+ (Java 11+ compatible)
• OSGi R7 (Core 7.0) specification (Felix 7.x implements R7, not R8; the compile-time dependency is osgi.core:8.0.0 for API completeness but only the R7 subset is used at runtime)
• Gradle 8.x for the convention plugin
• The existing directory structure from [RFC] Directory Structure for Data Prepper #305 is in place
• Spring Framework 5.3.x remains the DI framework (no Spring-DM or Blueprint)

Backward Compatibility

This proposal is fully backward compatible:

• Default mode is legacy — zero behavior change
• No public API changes to data-prepper-api
• @DataPrepperPlugin, @DataPrepperPluginConstructor,
  @DataPrepperExtensionPlugin annotations unchanged
• PluginProvider interface unchanged (new addPluginProvider() method on
  PluginProviderLoader is additive)
• ExtensionClassProvider interface unchanged (new
  addExtensionPluginClasses() method on ClasspathExtensionClassProvider is
  additive)
• Verified: OsiDataPrepperInternalPlugins and OsiDataPrepperOtherAmazonPlugins
  (34+ plugins) build and test with zero changes

Alternatives Considered

• JPMS (Java Modules): Built into the JDK but lacks dynamic loading and a service registry.
• Custom classloaders (Elasticsearch model): Simpler but no standardized lifecycle or service discovery.
• Fat/shadow JARs: Build-time isolation only; no dynamic loading and produces larger artifacts.

OSGi was chosen for its combination of classloader isolation, service registry, dynamic lifecycle management, and standardized module metadata.

OSGi Glossary

• Bundle: A JAR with OSGi manifest headers; the unit of deployment and isolation.
• Service Registry: Framework-managed registry where bundles publish and discover services.
• BundleActivator: Callback interface (start/stop) invoked when a bundle is activated or stopped.
• Boot Delegation: Packages delegated directly to the parent classloader (e.g., sun.*, javax.*).
• System Bundle: Bundle 0; represents the framework itself and exports shared packages to all bundles.

Risks and Mitigations

Risk	Mitigation
Felix adds startup latency	Lazy bundle activation; benchmark before/after
Classloader conflicts with shared deps	Explicit system package exports for API packages
Reflections library fails inside OSGi classloader	LegacyPluginBundleActivator runs Reflections within bundle classloader scope
Spring DI incompatible with bundle isolation	SpringOsgiBridge exposes BeanFactory as OSGi service
Operational complexity	Feature flag defaults to legacy; OSGi is opt-in

Related Issues

• [RFC] Plugin Redesign #321 — [RFC] Plugin Redesign
• Run plugins in their own classloader #1543 — Run plugins in their own classloader
• [RFC] Directory Structure for Data Prepper #305 — Directory Structure for Data Prepper

[dlvenable]

@kiran536 , Thank you for this proposal! This lines up with a number of goals that we have for the project.

I think the key goals that you listed are:

1. Classpath isolation - As the project has grown this has become a growing issue. One instance of this is the difficulty of updating Armeria in Update Armeria to 1.32.6 #6343.
2. Dynamic loading - Eventually I'd like to see the ability to load plugins using some for of registry. But I'm not sure that this is a critical need without something like that.

Long-term plugin types

In terms of long-term plugins support I'd believe we should aim to have these classes of plugins:

Core plugins

• These would be part of the base distribution always
• Maintained by Data Prepper maintainers
• Loads in the same classpath as Data Prepper core
• Provided as libraries in Data Prepper
• Examples: add_entries, drop_events, rename_keys

Embedded plugins

• Plugins that are part of the base distribution
• Maintained by Data Prepper maintainers
• Loads in different classpath using a different classloader
• Provided as uber-jars with dependencies
• Could be removed dynamically if desired
• Examples: s3 sink/source, aws_lambda processor, kakfa sink/source/buffer

External plugins

• Plugins that can be loaded dynamically
• May be maintained by the OpenSearch project. May be public open-source. May be private plugins
• Maintained by the community.
• Loads in different classpath using a different classloader
• Provided as uber-jars with dependencies
• Could be add dynamically

Here is a sketch of how the data might reside in Data Prepper

data-prepper/
  lib/    # These load as normal libraries in the same classloader
    mutate-entries.jar  # Has rename_keys
    drop-events.jar
    data-prepper-core.jar
    ...
  plugins/   # These are uber-jars with dependencies embedded
    s3-sink-2.16.0.jar
    kafka-source-2.16.0.jar
    aws-lambda-processor-2.16.0

How do you see Apache Felix supporting these?

Classloader

When plugins with a different classloader run they will get the current version of data-prepper-api loaded, correct? I think we probably want that since this is the bridge between core and a plugin.

Opt-in

Why do you propose making this opt-in? What benefit would a Data Prepper user have to not having this always enabled?

I'd think that we make this the only way to load plugins in Data Prepper. We still have the SPI approach (PluginProviderLoader) available for anybody who wans to drastically change the behavior of a custom Data Prepper instance. They could have the legacy approach (I'm not sure why) or a plugin loader that is specialized for their needs.

Compatibility

Does Apache Felix have any support for detecting version incompatibilities? Or if the plugin fails to load, how will it handle that error? Can we provide clear errors on this?

Implementation

As part of this change, we should update at least one plugin that Data Prepper currently provides to an Embedded Plugin. Do you have any suggestions on which of these to do first?

[KarstenSchnitter]

@kiran536 thank you for writing this RFC, and @dlvenable thank you for the first review.

I think your five problems separate into two classes:

1. Problems that need to be addressed as soon as possible since there are already issues caused by them.
2. Problems where a solution would bring a benefit in mid- to long-term time range.

ClassLoader isolation and dependency management are currently pressing topics, e.g., netty and jackson dependencies. These are the issues that must be addressed. Plugin lifecycle, dynamic reloading and modular distribution are problems that have a little more time. As of now, hot-reload of configuration is unsupported in Data Prepper (c.f. #973 and #5767). Without that feature dynamic loading of plugins is of little value.

I agree with @dlvenable, that separating the plugins into a core and an extended group makes a lot of sense. I am unsure whether an external group is necessary. The difference would be, that core plugins always come with Data Prepper using the core class loader and requiring little dependencies. In contrast, extended plugins would make use of the new approach to have a distinct class loader separate their dependencies from the core and other plugins. This would allow to keep large dependencies like the AWS SDK out of the Data Prepper core.

I propose to start with shaded dependencies but as bundles of plugins. This is similar to @dlvenable proposal and does not exclude the later introduction of OSGi as this would basically only require adding an OSGi manifest to the shaded jars. Let me elaborate on the plugin bundles:

Data Prepper has a lot of very similar plugins with the same dependencies. It makes sense to bundle, for example, all OpenTelemetry source plugins together. Maybe even the sink. Similar for Kafka plugins, S3, SQS, ... This avoids rebundling potentially large dependencies too many times. The approach also allows a step-wise introduction of the plugin loading model without too much effort.

There is one more issue I see with the introduction of OSGi in general: It might break Data Prepper's delivery guarantees. There are two issues here:

1. Plugin health: Apache Felix has no direct health check on plugins. It is possible that failures occur without being properly reported. In that case Data Prepper might be running with a non-functional plugin but passing health checks.

2. E2E-Acknowledgements: Apache Felix has no notion of the Data Prepper acknowledgements. It might stop and restart a plugin at any point in time. For the acknowledgement model of Data Prepper this might look like a plugin failure as in-flight data does not get acknowledged. I cannot say for sure, what the impact on a source (or buffer) plugin might be.

Both issues require more investigation on how to address them. They are inherent to the dynamic plugin loading ability. Supporting this feature requires addressing these issues.

[srikanthpadakanti]

@kiran536 Thank you for this RFC. Two questions on the design:

When an embedded plugin loads in its own classloader, how does it interact with the pipeline buffer? The buffer sits between source and processor. If they are in different classloaders, Event objects crossing the buffer need to resolve from a shared parent classloader. Is that covered by the system package export list or does it need explicit handling?

What happens to PluginMetrics? Today it is injected via constructor through Spring DI. In the OSGi model, does each bundle get its own PluginMetrics instance from the service registry or does it still come through SpringOsgiBridge? If it still comes through Spring, then every plugin bundle has a hard dependency on the bridge, which contradicts the isolation goal.

[kiran536]

Thanks @dlvenable , @KarstenSchnitter (hello from a fellow Physics Major!), @srikanthpadakanti for your thoughtful questions. OSGi/Felix gives us optionality (not mandate) of introducing new way to build, manage and deploy plugins in addition to providing solid classpath isolation. OSGI/Felix adds little overhead (memory or compute wise) while providing many benefits. We would like to exclude HotReloading from the scope of this PFR and focus solely on modularization and classpath isolation.

IIUC the core question all three of you are asking is: does the isolation problem justify adopting Felix, or are simpler alternatives good enough? I chose Felix because it solves classloading more completely than the alternatives, and I want to add more detials about what "more completely" means. Shaded JARs relocate packages at build time, which breaks reflection and serialization when rewritten package names no longer match what code expects at runtime. SPI service files can be relocated by Shadow, but custom service discovery mechanisms and Class.forName calls are not automatically handled. Custom URLClassLoaders, something like Elasticsearch model, provide real per-plugin isolation and work at scale and ES has supported 100s of plugins this way. But custom classloaders have no deploy-time validation. You discover a missing class or a version mismatch when a code path executes, which in a data pipeline can be hours into a production run processing records that matter. Felix's resolver validates Import-Package and Export-Package declarations before the pipeline starts. A missing or incompatible dependency keeps the bundle in INSTALLED state with a diagnostic message. This requires version attributes on exports and Import-Package ranges on bundles, which will be added as part of initial implementation. For a system where silent version conflicts cause data loss, that fail-fast behavior is not a nice-to-have. It is the reason I chose Felix over building a custom classloader scheme that would need its own resolver to achieve the same guarantee.

Felix adds build complexity: OSGi manifest generation, tooling, a new conceptual layer for contributors who have never seen bundle headers. Debugging classloader issues under OSGi produces stack traces that are harder to read than flat-classpath errors. We use mature spec with bundle manifests, the resolver, and per-bundle classloaders. I would also like to add that the OSGi Alliance transferred its assets to the Eclipse Foundation's OSGi Working Group in 2020, and active specification evolution has slowed. Felix 7.0.5 shipped in June 2022 but the broader Felix umbrella projects are actually quite active building new tooling, add-on features etc. The spec is very mature and stable with no breaking changes underneath

To the questions around whether Felix detects version incompatibilities, and whether we can produce clear error messages. This will be covered in initial implementation. We will add version attributes to FRAMEWORK_SYSTEMPACKAGES_EXTRA exports and generate Import-Package ranges via the Gradle OSGi plugin so Felix's resolver can enforce version checks. We will build a BundleResolutionErrorTranslator that converts Felix's internal resolution failure format into messages like "Plugin X requires Jackson 2.17+, but 2.15 is available from the host." I will add a startup summary log that reports all bundle successes and failures after loading. These are proposed components, not existing code. The proof point is migrating one real plugin, the Armeria/Netty plugin from issue #6343 is the leading candidate, and demonstrating three things: the plugin runs with its own dependency versions without affecting the host classpath, an intentionally incompatible Import-Package declaration produces a clear resolver error at deploy time, and existing pipeline behavior has zero regressions.

To the questions around Felix having no awareness of Data Prepper's end-to-end acknowledgement model, that a bundle can be ACTIVE in OSGi while internally broken, and that dynamic loading has little value without dynamic pipeline configuration. You are right on all three counts. Here is how the phased approach addresses each one directly. The initial implementation excludes hot-reload from production entirely (i.e disabled). PluginHotLoader will not be on the production classpath. It will exist only in a test-scoped module. In Initial phase, bundles install once at startup and stop once at shutdown, the same lifecycle as today's classpath plugins. There is no mid-flight bundle stop, so the acknowledgement orphaning scenario you described cannot occur. That said, I agree with your observation that even static-mode activation failures need proper handling.The initial implementation will tighten this: activation failures will propagate to the pipeline startup path so the pipeline either fails fast or skips the failed plugin with a clear error (only in optional plugin cases). In a future phase we will build full pipeline drain coordination via a BundleListener on STOPPING events, functional health probes that detect internally-broken plugins beyond OSGi's structural ACTIVE state, and acknowledgement-aware shutdown through a proposed PluginLifecycleCoordinator that waits for in-flight AcknowledgementSet instances before stopping a bundle. These require data-prepper-core changes and some amount of effort. We will not ship hot-reload in production until they are in place.

All Data Prepper API types (Event, JacksonEvent, Record, Buffer, AcknowledgementSet, PluginMetrics) are exported from the host classloader via FRAMEWORK_SYSTEMPACKAGES_EXTRA. Every plugin bundle imports these from the system bundle, so when a source writes a Record to the buffer and a processor reads it, both sides reference the same class loaded once by the host. No serialization boundary exists. The classloader boundary wraps only plugin internals: their private copies of Jackson, Netty, HTTP clients. A plugin can bundle Jackson 2.15 while the host runs Jackson 2.17 because plugin-internal Jackson types never cross the bundle boundary. PluginMetrics injection works the same way it does today: the host creates a PluginMetrics instance and passes it via @DataPrepperPluginConstructor. No ServiceTracker, no Declarative Services, no SpringOsgiBridge dependency from the plugin side.

Every new plugin increases transitive dependency conflict risk. The next conflict could involve Jackson, or gRPC, or the AWS SDK, and the cost of resolving these manually through version pinning and shade rules grows with every plugin we add. The initial implementation can help us stop that growth. Felix gives us deploy-time validation today and a path to richer plugin lifecycle management when we are ready for it.

[kiran536]

I also wanted to add a bit more details on the individual questions:

1. How does Felix support the three PFR problems (isolation, dynamic loading, monolithic distribution)?

Felix addresses each directly. For isolation, each bundle gets its own classloader, verified by BundleHealthCheck.verifyClassloaderIsolation(). For dynamic loading, PluginHotLoader.install() installs and starts bundles at runtime with atomic rollback. For monolithic distribution, the OSGi bundle model produces separate JARs with their own manifests, and LegacyPluginBundleActivator reads plugin properties per bundle. Felix provides formal lifecycle states (INSTALLED, RESOLVED, STARTING, ACTIVE, STOPPING, UNINSTALLED). We can fully tailor both the build time artifacts and run time environments. I see a future where we can tailor the DataPrepper instance per the need of each pipeline including only the minimum required plugins

2. Will plugins get the current version of data-prepper-api from the host classloader?

Yes. FelixPluginManager.start() exports ~20 Data Prepper API packages via FRAMEWORK_SYSTEMPACKAGES_EXTRA. All plugins resolve these from the host classloader. We will add version attributes, so a plugin compiled against a different API version would not silently load the host's version without warning.

3. Why opt-in instead of always-on?

DualModePluginManager reads -Dplugin.framework=osgi. This is to prevent regression risk for existing users. Once battle-tested and all unknowns figured out , always-on becomes viable.

4. **What benefit does a user get from NOT enabling OSGi?

This is just till we battle test the new model. There really are no benefits from not enabling it

5. **Does Felix detect version incompatibilities?

Yes, OSGi has built-in version range resolution. Felix refuses to resolve bundles whose Import-Package ranges do not match exports.

6. **How does it handle plugin load failures?

PluginHotLoader.install() uses atomic rollback: install, start, on failure uninstall broken bundle and rethrow BundleException. Double-fault on uninstall is logged but does not mask the original error. We will make sure the whole DataPrepper fails on these conditions

7. **Can we provide clear error messages?

Yes. The foundation exists: per-bundle state logging, BundleHealthCheck.getBundleStatuses(), unresolved bundle detection. We will implement the following: (1) a structured error translator to convert OSGi-internal format to human-readable messages like "Plugin X requires Y version 2.0+, but 1.5 is available," (2) a startup summary reporting all successes and failures after loading, and (3) health endpoint integration exposing BundleHealthCheck via Data Prepper's health API.

8. OSGi may break Data Prepper's delivery guarantees.

Correct. DualModePluginManager.shutdown() does reverse-order teardown and PluginHotLoader.uninstall() calls bundle.stop() before bundle.uninstall(). Gap: no coordination between OSGi bundle lifecycle events and pipeline drain/flush. The bundleChanged() listener only does debug logging. Needs a BundleListener that intercepts STOPPING events and coordinates with pipeline shutdown (buffer.drain(), wait for in-flight records). Phase 1 minimum: propagate activation failures to the pipeline startup path.

9. No functional health check on plugins.

Great callout. BundleHealthCheck checks structural health (bundle resolved, classloader isolated, bundle states). Yes, it is possible bundle can be ACTIVE in OSGi while internally broken (throwing exceptions,). OSGi has no visibility into functional health. Needs application-level health probes: a PluginHealthProbe interface, metrics-based detection, or Data Prepper health endpoint integration. We will build deep health checks

10. E2E Acknowledgements not respected by OSGi lifecycle.

Correct. OSGi exports the acknowledgement API so plugins can use it, but the lifecycle does not respect it. Bundle stop mid-flight leads to orphaned AcknowledgementSet, source never gets completion callback, and the result is duplicates or data loss. Before stopping a bundle: stop source from producing, wait for in-flight AcknowledgementSet to complete or timeout, then stop bundle. Requires a PluginLifecycleCoordinator before enabling Hotreloading

11. These issues require deeper investigation and are inherent to dynamic loading.

Agreed. These are some architectural gaps, and can be supported in the solution. In initial implementation (current PFR) classloader isolation only, no hot-reload in production. In future phases we will have pipeline-aware lifecycle coordination. and production hot-reload with delivery guarantees. The PFR will document hot-reload exclusion from production as a known limitation.

[dlvenable]

@kiran536 ,

Thank you for the detailed responses.

Overall, I think this is the right direction for the product.

One high-level question: Are there alternatives to OSGi? Or alternative implementations to Felix to consider? You did say this is a mature spec and still maintained. This is good. But I'm not sure what other projects may be providing similar offerings.

The initial implementation excludes hot-reload from production entirely (i.e disabled). PluginHotLoader will not be on the production classpath. It will exist only in a test-scoped module. In Initial phase, bundles install once at startup and stop once at shutdown, the same lifecycle as today's classpath plugins.

I agree with this approach.

And on this topic...

Before stopping a bundle: stop source from producing, wait for in-flight AcknowledgementSet to complete or timeout, then stop bundle. Requires a PluginLifecycleCoordinator before enabling Hotreloading

Can we just not stop bundles until the whole Data Prepper app is stopping? This seems the parallel to the not doing the hot-loader initially.

All Data Prepper API types (Event, JacksonEvent, Record, Buffer, AcknowledgementSet, PluginMetrics) are exported from the host classloader via FRAMEWORK_SYSTEMPACKAGES_EXTRA.

This is good. FYI, in #6607, I added a backward compatibility check on changes to data-prepper-api. The idea here is that we don't want to break existing plugins. So I think that this will allow a plugin built on, say 2.15, to run against 2.16. This should be the case even when the data-prepper-api is provided from the host classloader.

DualModePluginManager reads -Dplugin.framework=osgi. This is to prevent regression risk for existing users. Once battle-tested and all unknowns figured out , always-on becomes viable.

OK. This is good. We will just use this for the short-term to make it experimental. I like this. But I have a concern with it - this means that we cannot start to migrate existing plugins in the package to the new model.

We will make sure the whole DataPrepper fails on these conditions

Yes, I think this is the right approach. At least initially.

---

@KarstenSchnitter , regarding this comment:

I agree with @dlvenable, that separating the plugins into a core and an extended group makes a lot of sense. I am unsure whether an external group is necessary.

I think the extended and external groups are probably technically the same thing. They get loaded with their own classloader. The main difference relates to who maintains the plugin and where. We use a number of plugins internally at Amazon already and these could be considered part of the "external group."

[kiran536]

One high-level question: Are there alternatives to OSGi? Or alternative implementations to Felix to consider? You did say this is a mature spec and still maintained. This is good. But I'm not sure what other projects may be providing similar offerings.

@dlvenable , the other alternatives I considered were

1. Custom class loaders: No Framework, just what Java supports out of the box. This is more run time thing rather than build time solution. There is no built-in mechanism for type-space consistency across plugins that share API types referencing third-party types, and we will end up building ad-hoc tooling for manifest generation and dependency validation.
2. JPMS (Java Platform Module System): Again something from Java without having to depend on a framework. JPMS does not support multiple versions of the same module in a single JVM (a hard requirement for plugin isolation), module resolution happens at JVM startup with no per-plugin classloader isolation, and library ecosystem support for JPMS remains incomplete.
3. Gradle module isolation. Gradle's module metadata and dependency constraints can enforce version alignment at build time. This catches conflicts before deployment. However, it operates at build time only and provides no runtime isolation. Two plugins built separately can still conflict when loaded into the same JVM.

We are choosing Felix because of the following advatages
The Felix resolver checks every bundle's dependency graph at deploy time and refuses to start any bundle whose dependencies conflict. A JAR that would silently produce a ClassCastException hours into a pipeline run is instead rejected before the pipeline starts, with a diagnostic naming the exact incompatible package. Custom classloaders and shaded JARs discover these same conflicts at runtime, in code paths that testing never exercised. The resolver's uses directive extends this further: it detects when two plugins load different versions of the same class that would meet in the same call chain, catching type-space conflicts that custom classloaders surface only as inscrutable ClassCastExceptions at runtime.

The same module layer that isolates plugins today also provides dynamic lifecycle management, a service registry, and hot-deploy of updated bundles. We pay the adoption cost once and gain an expanding set of capabilities over time, rather than hand-building each new piece of plugin infrastructure from scratch.

The runtime cost is negligible: Felix is a single JAR under 1MB with zero transitive dependencies, and classloading uses delegation through a pre-computed package-to-classloader map. The real overhead is cognitive, and we mitigate it with tooling that Felix provides bnd (which computes manifests automatically from bytecode), a Gradle OSGi plugin, and an error translator that converts resolver failures into plain-language messages. The overall code changes are relatively minimal

Can we just not stop bundles until the whole Data Prepper app is stopping? This seems the parallel to the not doing the hot-loader initially.

Yes, we can consider as one of the options when we support hot reloading in future. It will depend on how flexible we want to make the plugin upgrades/launches seemless

The idea here is that we don't want to break existing plugins. So I think that this will allow a plugin built on, say 2.15, to run against 2.16. This should be the case even when the data-prepper-api is provided from the host classloader.

Absolutely, backward compatibility is P0. And the semver semantics will be honored. Again if the updates all happen by stopping DataPrepper, this will not be a major issue with the system modules/packages.

They get loaded with their own classloader. The main difference relates to who maintains the plugin and where. We use a number of plugins internally at Amazon already and these could be considered part of the "external group."

I was wondering if we can give Amazon as an example. as we are already doing this. The super ambitious goal is to let customer bring in their our plugins in the managed/serverless environments

[KarstenSchnitter]

I agree with the sentiment, that custom plugin class loaders do not solve all dependency problems. SPI class loading is a problem especially when used by dependencies:

Let's assume there is a shaded jar for the plugin, that tries to load some implementations via SPI. It will find registered classes from its parent classloader but cannot instantiate them causing an error that crashes the entire application. This can happen at any point when the service loader is activated so it is not automatically detected at application start-up.

For shaded jars mitigating this issues means more effort in creating the shaded jar. Alternatively using the JPMS can be of help, when the classes and interfaces in question belong to a specific module (not UNNAMED). Modularizing the entire Data Prepper code base would be a huge change and be at least as disruptive as introducing Apache Felix.

What I would like to understand is, how Apache Felix would deal with the SPI loading example from above. How would it behave and what mitigations would be required?

I also think, that we need a clearer picture on the lifecycle integration between Data Prepper and Felix. There is the start-up phase, health checks, acknowledgements, error handling and controlled shutdown including sending of in-flight data to be addressed. Possibly this list is still incomplete. All of this could be demonstrated in a poc but some upfront design would be appreciated.

Eventually, I would also appreciate a list of plugins, that should be migrated to this approach. These should perhaps be plugins with big and unique dependencies or optional use-cases.

[dlvenable]

Eventually, I would also appreciate a list of plugins, that should be migrated to this approach. These should perhaps be plugins with big and unique dependencies or optional use-cases.

I think in the long-term anything that is not just mutating events should take this approach.

Initially, here are some recommendations for plugins that could start taking this approach:

• Iceberg source (marked as experimental)
• Iceberg sink (currently in review in Add Iceberg sink plugin #6734)
• Splunk HEC source (proposed in [Feature] Splunk HEC Source for Data Prepper  #6823)