[Feature Request]: Add support for staging specific jars for managed transforms

[chamikaramj]

What would you like to happen?

Currently, during managed transforms expansion via the expansion service container, we stage all service jars in that container (all jars in the classpath).

This should be updated to only stage specific jars used by the transform being expanded.

This can be done by extending the functionality offered by the expansion service config config for managed transforms.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Abacn]

Hi @chamikaramj, is this still a release blocker for Beam 2.69.0 ?

[chamikaramj]

I think it's good to get this in to reduce staging size during managed transforms expansion. I should be able to get a PR out in a bit.

[chamikaramj]

Opened #36515.