support pass extensions for plan logging and profiling per optimization pass#74
Open
yliang412 wants to merge 11 commits into
Open
support pass extensions for plan logging and profiling per optimization pass#74yliang412 wants to merge 11 commits into
yliang412 wants to merge 11 commits into
Conversation
Signed-off-by: Yuchen Liang <yuchenl3@andrew.cmu.edu>
Signed-off-by: Yuchen Liang <yuchenl3@andrew.cmu.edu>
Signed-off-by: Yuchen Liang <yuchenl3@andrew.cmu.edu>
Signed-off-by: Yuchen Liang <yuchenl3@andrew.cmu.edu>
Signed-off-by: Yuchen Liang <yuchenl3@andrew.cmu.edu>
2050de9 to
84e3764
Compare
Signed-off-by: Yuchen Liang <yuchenl3@andrew.cmu.edu>
Signed-off-by: Yuchen Liang <yuchenl3@andrew.cmu.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The optd rewrite and optimization phases were hard-coded in
Cascades::optimize, which made them difficult to observe or instrument.EXPLAINonly showed a limited before/after view, and there was no systematic way to collect per-phase plan snapshots or timing metrics.Summary of changes
PlanPass/PassExtension/PassManagerframework inoptd-core.PlanPassimplementations.PassManagerso the memo-based cascades phase can use the same extension hooks.CascadesthroughPassManager, including reverse-orderafter_passcallbacks so extensions behave like proper around hooks.ExplainPassExtensionin the DataFusion planner to record optd plans after each named phase.physical_plan after optd-cascadestoEXPLAINoutput and removed the redundantphysical_plan after optd-finalizedrow.logical_plan after optd-*andphysical_plan after optd-*sections.SET optd.profile_passes = true.optd/core/src/rules/profile.rsso tracing uses the defaultoptd_core::rules::profiletarget.tracingwith structured fields for pass name, changed flag, elapsed time, total time, and invocation count.Example: Profiling Tracing Output
RUST_LOG=optd_core::rules::profile=info cargo run --bin planner_test_apply -- tpch::q3 -- --exact Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.18s Running `target/debug/planner_test_apply 'tpch::q3' -- --exact` 2026-04-28T16:55:39.305964Z INFO optd_core::rules::profile: optimizer pass profile pass="decorrelation" changed=false elapsed_ms=0.0179 total_ms=0.0179 invocations=1 2026-04-28T16:55:39.306993Z INFO optd_core::rules::profile: optimizer pass profile pass="simplification" changed=true elapsed_ms=0.473 total_ms=0.473 invocations=1 2026-04-28T16:55:39.309540Z INFO optd_core::rules::profile: optimizer pass profile pass="cascades" changed=true elapsed_ms=2.25 total_ms=2.25 invocations=1(The log actually get print twice, because we plan the query twice, once for planning, once for execution)
Future work