SNOW-2331021: Wrap internal Series/DF creation#3768
Conversation
sfc-gh-jkew
left a comment
There was a problem hiding this comment.
Why did you need to rebuild the csv? Not complaining I just want to make sure that is not a regular occurance.
Since this test was fixing a hybrid-specific bug, I wanted to see if the number of failures was reduced. It may help if we sort the DF on test name before exporting the CSV so the diff is less messy. |
Ahh, I see now in the description: " few hundred test failures" that's awesome. |
Just to be clear, I didn't check which tests were fixed, I just compared the output of I just now sorted and dropped the position index of CSV, so diffs should be smaller in the future. |
Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes SNOW-2331021
Fill out the following pre-review checklist:
Please describe how your code solves the related issue.
Many internal functions create an instance of
pd.Seriesorpd.DataFrame, then retrieve a query compiler from the object. With hybrid execution enabled, this query compiler is sometimes an instance ofNativeQueryCompiler, which may break invariants when aSnowflakeQueryCompileris expected instead. The new helper functionnew_snow_seriesandnew_snow_dfdisable hybrid mode when creating a Series/DF for internal use.This PR replaces constructor calls within the SFQC and indexing_overrides, which were responsible for the bug in the original ticket. Other constructor calls remain, but I have not yet triaged them.
This removes a few hundred test failures in
modin_hybrid_integ_results.csv.