Skip to content

Commit d2278a9

Browse files
cj-zhukovmartin-g
andauthored
Check sqllogictests for any dangling config settings (#17914) (#20838)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes ##17914. ## Rationale for this change In a previous PR #20474, I added a bash script that parsed the `SLT` files and checked whether any DataFusion configuration options were modified without being reset. While that approach worked, it relied on external scripting and additional parsing logic. This PR introduces a simpler and more direct solution implemented in Rust. At the end of each `SLT` test file execution, the current configuration is compared with the default configuration using a `Drop` implementation. If any configuration values were modified and not restored, a warning is printed. This approach is easier to maintain and keeps the validation logic within the Rust codebase rather than relying on an external bash script. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? - Capture the default DataFusion configuration when the `SLT` runner is initialized. - Implement `Drop` for the DataFusion SLT engine. - When an `SLT` file finishes executing, compare the current configuration with the default configuration. - If differences are detected, print a warning showing which configuration options were modified. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? This behavior is exercised by the existing `SLT` test suite. The configuration check runs automatically when each `SLT` file completes execution. <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? No. This change only affects internal `SLT` test infrastructure and does not modify any public APIs. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
1 parent b7e4213 commit d2278a9

File tree

1 file changed

+58
-5
lines changed
  • datafusion/sqllogictest/src/engines/datafusion_engine

1 file changed

+58
-5
lines changed

datafusion/sqllogictest/src/engines/datafusion_engine/runner.rs

Lines changed: 58 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
// specific language governing permissions and limitations
1616
// under the License.
1717

18+
use std::collections::HashMap;
1819
use std::sync::Arc;
1920
use std::{path::PathBuf, time::Duration};
2021

@@ -38,29 +39,38 @@ pub struct DataFusion {
3839
relative_path: PathBuf,
3940
pb: ProgressBar,
4041
currently_executing_sql_tracker: CurrentlyExecutingSqlTracker,
42+
default_config: HashMap<String, Option<String>>,
4143
}
4244

4345
impl DataFusion {
4446
pub fn new(ctx: SessionContext, relative_path: PathBuf, pb: ProgressBar) -> Self {
47+
let default_config = ctx
48+
.state()
49+
.config()
50+
.options()
51+
.entries()
52+
.iter()
53+
.map(|e| (e.key.clone(), e.value.clone()))
54+
.collect();
55+
4556
Self {
4657
ctx,
4758
relative_path,
4859
pb,
4960
currently_executing_sql_tracker: CurrentlyExecutingSqlTracker::default(),
61+
default_config,
5062
}
5163
}
5264

5365
/// Add a tracker that will track the currently executed SQL statement.
5466
///
5567
/// This is useful for logging and debugging purposes.
5668
pub fn with_currently_executing_sql_tracker(
57-
self,
69+
mut self,
5870
currently_executing_sql_tracker: CurrentlyExecutingSqlTracker,
5971
) -> Self {
60-
Self {
61-
currently_executing_sql_tracker,
62-
..self
63-
}
72+
self.currently_executing_sql_tracker = currently_executing_sql_tracker;
73+
self
6474
}
6575

6676
fn update_slow_count(&self) {
@@ -135,6 +145,49 @@ impl sqllogictest::AsyncDB for DataFusion {
135145
async fn shutdown(&mut self) {}
136146
}
137147

148+
impl Drop for DataFusion {
149+
fn drop(&mut self) {
150+
let mut changed = false;
151+
152+
for e in self.ctx.state().config().options().entries() {
153+
let default_entry = self.default_config.remove(&e.key);
154+
155+
if let Some(default_entry) = default_entry
156+
&& default_entry.as_ref() != e.value.as_ref()
157+
{
158+
if !changed {
159+
changed = true;
160+
self.pb.println(format!(
161+
"SLT file {} left modified configuration",
162+
self.relative_path.display()
163+
));
164+
}
165+
166+
let default = default_entry.as_deref().unwrap_or("NULL");
167+
let current = e.value.as_deref().unwrap_or("NULL");
168+
169+
self.pb
170+
.println(format!(" {}: {} -> {}", e.key, default, current));
171+
}
172+
}
173+
174+
// Any remaining entries were present initially but removed during execution
175+
for (key, value) in &self.default_config {
176+
if !changed {
177+
changed = true;
178+
self.pb.println(format!(
179+
"SLT file {} left modified configuration",
180+
self.relative_path.display()
181+
));
182+
}
183+
184+
let default = value.as_deref().unwrap_or("NULL");
185+
186+
self.pb.println(format!(" {key}: {default} -> NULL"));
187+
}
188+
}
189+
}
190+
138191
async fn run_query(
139192
ctx: &SessionContext,
140193
is_spark_path: bool,

0 commit comments

Comments
 (0)