Expose physical plan optimizer

[timsaucer]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

I want to add custom physical plan optimizer rules, written in rust, to datafusion-python. Now that we have the ability to share execution plans across the FFI boundary we should be able to expose this interface.

Describe the solution you'd like

• Implement FFI upstream datafusion
• Expose with_physical_optimizer_rules and/or with_physical_optimizer_rule

Describe alternatives you've considered

None

Additional context

This will support things like https://github.com/datafusion-contrib/datafusion-distributed in datafusion-python

[timsaucer]

Upon deeper evaluation, this seems non-trivial since the optimizer won't understand what kinds of plans its getting. I suspect we need to go a route of serializing/deserializing the plans.

[milenkovicm]

I guess this is the same issue like ballista support

[renato2099]

couldn't substrate.io be considered here as well ?