fix: skips projection pruning for whole subtree

[Acfboy]

Which issue does this PR close?

• Closes Bug: unnecessary columns projected and redundant filters pushed down #18816 .

Rationale for this change

In UserDefinedLogicalNodeCore, the default implementation of necessary_children_exprs returns None, which signals to the optimizer that it cannot determine which columns are required from the child.

The optimizer takes a conservative approach and skips projection pruning for that node, leading to complex and redundant plans in the subtree. However, it would make more sense to assume all columns are required and let the optimizer proceed, rather than giving up on the entire subtree entirely.

What changes are included in this PR?

LogicalPlan::Extension(extension) => {
    if let Some(necessary_children_indices) =
        extension.node.necessary_children_exprs(indices.indices())
    {
        ...
    } else {
        // Requirements from parent cannot be routed down to user defined logical plan safely
        // Assume it requires all input exprs here
        plan.inputs()
            .into_iter()
            .map(RequiredIndices::new_for_all_exprs)
            .collect()
    }
}

instead of

datafusion/datafusion/optimizer/src/optimize_projections/mod.rs

Lines 331 to 337 in b6d46a6

	LogicalPlan::Extension(extension) => {
	let Some(necessary_children_indices) =
	extension.node.necessary_children_exprs(indices.indices())
	else {
	// Requirements from parent cannot be routed down to user defined logical plan safely
	return Ok(Transformed::no(plan));
	};

Are these changes tested?

Yes.

In addition to unit tests, I've also added a complete end-to-end integration test that reproduces the full scenario in the issue. This might seem redundant, bloated, or even unnecessary. Please let me know if I should remove these tests.

An existing test is modified, but I think the newer behavior is expected.

Are there any user-facing changes?

Yes. But I think the new implementation is the expected behavior.

[neilconway]

Thanks -- lgtm!

[Acfboy]

Just updated the branch to resolve the out-of-date status. Ready for a final look when you have a chance @alamb . Thanks!

[alamb]

Thanks @neilconway and @Acfboy

[alamb]

FYI @niebayes