Skip to content

test: Add datafusion.format.* configs test coverage #21355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alamb merged 1 commit into from
Apr 7, 2026
+198 −0
Merged

test: Add datafusion.format.* configs test coverage #21355

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
198 changes: 198 additions & 0 deletions datafusion/sqllogictest/test_files/set_variable.slt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -379,6 +379,204 @@ RESET datafusion.execution.batches_size
statement error DataFusion error: Invalid or Unsupported Configuration: Config field is a scalar usize and does not have nested field "bar"
RESET datafusion.execution.batch_size.bar

#############################################
## Test datafusion.format.* configurations ##
#############################################
query T
SELECT name FROM information_schema.df_settings WHERE name LIKE 'datafusion.format.%' ORDER BY name
Copy link
Copy Markdown
Contributor

@alamb alamb Apr 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to tests showing that they can be set/removed, how about also adding tests that run a function that is affected by them (e.g. select a date and then show that changing datafusion.format.date changes how they are displayed)?

Copy link
Copy Markdown
Member Author

@erenavsarogullari erenavsarogullari Apr 5, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alamb Thanks for the review and yes, we also need to add more functional coverage as follows:
Expected Behavior:

statement ok
SET datafusion.format.date_format = '%d-%m-%Y'

query D
SELECT DATE '2024-03-15'
----
15-03-2024

statement ok
RESET datafusion.format.date_format

query D
SELECT DATE '2024-03-15'
----
2024-03-15

However, currently, sqllogictest framework does not support custom format updates and it runs the test with default format settings and the results as follows:
Current Behavior:

statement ok
SET datafusion.format.date_format = '%d-%m-%Y'

query D
SELECT DATE '2024-03-15'
----
Returns 2024-03-15 instead of 15-03-2024

The reason is that and sqllogictest framework needs to be extended in order to uptake format settings by SessionContext instead default format settings: https://github.com/apache/datafusion/blob/main/datafusion/sqllogictest/src/engines/datafusion_engine/normalize.rs#L239. I need to have a look on this. Does it make sense to be merged current PR and extend sqllogictest framework and adding remaining functional tests by follow-up PR? If so, i can create follow-up issue.

Copy link
Copy Markdown
Contributor

@alamb alamb Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to have a look on this. Does it make sense to be merged current PR and extend sqllogictest framework and adding remaining functional tests by follow-up PR? If so, i can create follow-up issue.

Yes that is an excellent idea. I will do so. Thank you

Copy link
Copy Markdown
Member Author

@erenavsarogullari erenavsarogullari Apr 7, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#21447 has been created to follow-up.

----
datafusion.format.date_format
datafusion.format.datetime_format
datafusion.format.duration_format
datafusion.format.null
datafusion.format.safe
datafusion.format.time_format
datafusion.format.timestamp_format
datafusion.format.timestamp_tz_format
datafusion.format.types_info

# date_format: SET / SHOW / RESET / SHOW
statement ok
SET datafusion.format.date_format = '%d-%m-%Y'

query TT
SHOW datafusion.format.date_format
----
datafusion.format.date_format %d-%m-%Y

statement ok
RESET datafusion.format.date_format

query TT
SHOW datafusion.format.date_format
----
datafusion.format.date_format %Y-%m-%d

# datetime_format
statement ok
SET datafusion.format.datetime_format = '%Y/%m/%d %H:%M:%S'

query TT
SHOW datafusion.format.datetime_format
----
datafusion.format.datetime_format %Y/%m/%d %H:%M:%S

statement ok
RESET datafusion.format.datetime_format

query TT
SHOW datafusion.format.datetime_format
----
datafusion.format.datetime_format %Y-%m-%dT%H:%M:%S%.f

# timestamp_format
statement ok
SET datafusion.format.timestamp_format = '%FT%H:%M:%S'

query TT
SHOW datafusion.format.timestamp_format
----
datafusion.format.timestamp_format %FT%H:%M:%S

statement ok
RESET datafusion.format.timestamp_format

query TT
SHOW datafusion.format.timestamp_format
----
datafusion.format.timestamp_format %Y-%m-%dT%H:%M:%S%.f

# timestamp_tz_format (default NULL)
statement ok
SET datafusion.format.timestamp_tz_format = '%Y-%m-%d %H:%M:%S %z'

query TT
SHOW datafusion.format.timestamp_tz_format
----
datafusion.format.timestamp_tz_format %Y-%m-%d %H:%M:%S %z

statement ok
RESET datafusion.format.timestamp_tz_format

query TT
SHOW datafusion.format.timestamp_tz_format
----
datafusion.format.timestamp_tz_format NULL

# time_format
statement ok
SET datafusion.format.time_format = '%H-%M-%S'

query TT
SHOW datafusion.format.time_format
----
datafusion.format.time_format %H-%M-%S

statement ok
RESET datafusion.format.time_format

query TT
SHOW datafusion.format.time_format
----
datafusion.format.time_format %H:%M:%S%.f

# duration_format: values are normalized to lowercase; ISO8601 and pretty are valid
statement ok
SET datafusion.format.duration_format = ISO8601

query TT
SHOW datafusion.format.duration_format
----
datafusion.format.duration_format iso8601

statement ok
SET datafusion.format.duration_format to 'PRETTY'

query TT
SHOW datafusion.format.duration_format
----
datafusion.format.duration_format pretty

statement ok
RESET datafusion.format.duration_format

query TT
SHOW datafusion.format.duration_format
----
datafusion.format.duration_format pretty

# null display string
statement ok
SET datafusion.format.null = 'NuLL'

query TT
SHOW datafusion.format.null
----
datafusion.format.null NuLL

statement ok
RESET datafusion.format.null

query TT
SHOW datafusion.format.null
----
datafusion.format.null (empty)

# safe
statement ok
SET datafusion.format.safe = false

query TT
SHOW datafusion.format.safe
----
datafusion.format.safe false

statement ok
RESET datafusion.format.safe

query TT
SHOW datafusion.format.safe
----
datafusion.format.safe true

# types_info
statement ok
SET datafusion.format.types_info to true

query TT
SHOW datafusion.format.types_info
----
datafusion.format.types_info true

statement ok
RESET datafusion.format.types_info

query TT
SHOW datafusion.format.types_info
----
datafusion.format.types_info false

# Case-insensitive variable name
statement ok
SET datafusion.FORMAT.DATE_FORMAT = '%m/%d/%Y'

query TT
SHOW datafusion.format.date_format
----
datafusion.format.date_format %m/%d/%Y

statement ok
RESET datafusion.format.date_format

query TT
SHOW datafusion.format.date_format
----
datafusion.format.date_format %Y-%m-%d

# Invalid format option name
statement error DataFusion error: Invalid or Unsupported Configuration: Config value "unknown_option" not found on FormatOptions
SET datafusion.format.unknown_option = true

############
## Test runtime configuration variables
############
Expand Down
Loading