Build index scans when tuples are compared for equality or used in set membership.#3541
Open
nicktobey wants to merge 2 commits into
Open
Build index scans when tuples are compared for equality or used in set membership.#3541nicktobey wants to merge 2 commits into
nicktobey wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes dolthub/dolt#11003
Currently, Dolt will not optimize queries if index columns are used inside tuples. This can create problems for multi-column indexes, in situations where it is natural to represent the entire key as a tuple.
The most likely situation is one where the user wants to delete a specific set of records, and writes a query that looks like this:
Currently, Dolt will never optimize this query to use an index. As written, the query will do a full table scan. Alternatively, the user can execute an individual delete statement for each row, but for a table with N rows, this will cause Dolt to write O(log N) garbage chunks for each row deleted. For large datasets, neither option is efficient.
Another workaround would be for the user to rewrite the query into an equivalent query that can be optimized. Such a query might look like this:
However, there's a compelling reason to not require the user to do this: prepared statements.
For example, the python SQL driver allows using a prepared statement like this:
No equivalent exists for the workaround query: the user would need to manually construct a different query string based on the number of rows to remove, and clients dynamically constructing SQL query strings is typically a bad practice. Thus, it makes sense to support optimizing the original query.
There are two different ways to address this: either improving the analyzer so that it can analyze tuple expressions, or rewriting the filter internally to match the workaround query's filter.
This PR is an attempt to implement the latter, by allowing a single node in the costed index scan tree (the iScanLeaf struct) to describe a filter on multiple index columns.
In hindsight, rewriting the filter might have been easier, and would also better handle cases where only some of the columns in the tuple match the index keys. But such an approach could have its own pitfalls.