Skip to content

Fix potential overflow when we print verbose physical plan#15

Merged
zhuqi-lucas merged 4 commits into
branch-50from
branch-50-upgrade
Sep 27, 2025
Merged

Fix potential overflow when we print verbose physical plan#15
zhuqi-lucas merged 4 commits into
branch-50from
branch-50-upgrade

Conversation

@zhuqi-lucas
Copy link
Copy Markdown
Collaborator

@zhuqi-lucas zhuqi-lucas commented Sep 26, 2025

Here is the PR which is in apache datafusion:
apache#17383

This PR fix it to make debug log not using pure physical plan printing.

It make our e2e testing failed with:

2025-09-26T06:12:05.638390Z DEBUG datafusion::physical_planner: Optimized physical plan:
CoalesceBatchesExec: target_batch_size=8192
  HashJoinExec: mode=CollectLeft, join_type=LeftAnti, on=[(c_customer_sk@0, c_customer_sk@0), (c_current_cdemo_sk@1, c_current_cdemo_sk@1), (c_current_hdemo_sk@2, c_current_hdemo_sk@2), (c_current_addr_sk@3, c_current_addr_sk@3), (c_first_shipto_date_sk@4, c_first_shipto_date_sk@4), (c_first_sales_date_sk@5, c_first_sales_date_sk@5), (c_birth_day@6, c_birth_day@6), (c_birth_year@7, c_birth_year@7), (c_customer_id@8, c_customer_id@8), (c_salutation@9, c_salutation@9), (c_first_name@10, c_first_name@10), (c_last_name@11, c_last_name@11), (c_preferred_cust_flag@12, c_preferred_cust_flag@12), (c_birth_country@13, c_birth_country@13), (c_email_address@14, c_email_address@14), (c_last_review_date@15, c_last_review_date@15), (c_birth_month@16, c_birth_month@16)]
    ProjectionExec: expr=[c_customer_sk@0 as c_customer_sk, c_current_cdemo_sk@1 as c_current_cdemo_sk, c_current_hdemo_sk@2 as c_current_hdemo_sk, c_current_addr_sk@3 as c_current_addr_sk, c_first_shipto_date_sk@4 as c_first_shipto_date_sk, c_first_sales_date_sk@5 as c_first_sales_date_sk, c_birth_day@6 as c_birth_day, c_birth_year@8 as c_birth_year, c_customer_id@9 as c_customer_id, c_salutation@10 as c_salutation, c_first_name@11 as c_first_name, c_last_name@12 as c_last_name, c_preferred_cust_flag@13 as c_preferred_cust_flag, c_birth_country@14 as c_birth_country, c_email_address@15 as c_email_address, c_last_review_date@16 as c_last_review_date, lpad(CAST(c_birth_month@7 AS Utf8), 2, 0) as c_birth_month]
      DataSourceExec: file_groups={1 group: [[test/delta_encoding_required_column.parquet]]}, projection=[c_customer_sk, c_current_cdemo_sk, c_current_hdemo_sk, c_current_addr_sk, c_first_shipto_date_sk, c_first_sales_date_sk, c_birth_day, c_birth_month, c_birth_year, c_customer_id, c_salutation, c_first_name, c_last_name, c_preferred_cust_flag, c_birth_country, c_email_address, c_last_review_date], file_type=parquet
    CooperativeExec
      FileScanExec: table = atlas.test.delta_encoding_required_column_partitioned, scan_direction=Native, page_size=1024, partitions=[0, 1]



thread 'tokio-runtime-worker' has overflowed its stack
fatal runtime error: stack overflow, aborting

@github-actions github-actions Bot added the core label Sep 26, 2025
@zhuqi-lucas
Copy link
Copy Markdown
Collaborator Author

I am testing a better way to keep debug but using displayable for details, if it works, it should be better.

@zhuqi-lucas zhuqi-lucas changed the title change physical plan details loglevel from debug to trace for potential overflow Fix potential overflow when we print verbose physical plan Sep 27, 2025
@zhuqi-lucas zhuqi-lucas merged commit 7b53d3e into branch-50 Sep 27, 2025
57 of 59 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants