normalize "in" predicate as disjunction of "=="

[lr4d]

Description:

The following two statements are equivalent:

• x in [1,2,3]
• (x == 1) or (x == 2) or (x == 3)

This approach simplifies the function by just calling itself instead of iterating over a for loop and essentially running the code block under if op == "==": for each value of an "in" predicate

• Closes #xxxx
• Changelog entry

[fjetter]

Does the any statement abort evaluation as soon as it finds the first value which evaluates to True or does it continue the evaluation? aborting early could vastly improve best case scenarios but I'm not sure about how the stdlib implementation deals with this (https://docs.python.org/3/library/functions.html#any doesn't tell us anything about it)

Edit:

Yes it does

In [4]: class Boom:
   ...:     def __bool__(self):
   ...:         raise Exception()

In [7]: any([b])
---------------------------------------------------------------------------
Exception
....

In [8]: any([True, b])
Out[8]: True

[fjetter]

eventually we might want to consider passing the filters directly to pyarrow since they implemented this by now as well. I would expect them to deal with these things much faster than we are in python.

[fjetter]

do we have any kind of benchmarks? What I'm a bit concerned about is that we interact with the parquet reader now for every element in the list. I don't know how expensive this is

[lr4d]

What I'm a bit concerned about is that we interact with the parquet reader now for every element in the list. I don't know how expensive this is

Very valid concern. We can isolate the operation evaluation logic in a separate function and call that directly so that we don't need to bother about this

[fjetter]

cc @mlondschien this might interest you

xref #325

[fjetter]

Very valid concern. We can isolate the operation evaluation logic in a separate function and call that directly so that we don't need to bother about this

All depends on how complex the end state will be since your intention is to simplify things. I think this would be ok, though

[mlondschien]

Thanks @fjetter for pinging me. This does not seem to explode the complexity exponentially as we're normalizing for a single column only. So I don't think the concerns mentioned in #325 would apply here.

[lr4d]

eventually we might want to consider passing the filters directly to pyarrow since they implemented this by now as well. I would expect them to deal with these things much faster than we are in python.

Good point. Do you know how "mature" this logic for pyarrow is atm? It might make more sense to invest in using their functionality directly rather than this kind of work