Would this package be useful in DataFramesMeta?

[pdeffebach]

I just finished a big PR in DataFramesMeta to try and reduce latency.

Given an expression of the form

f(:x, :y)

we make an anonymous function

(x, y) -> f(x, y)

This carries a compilation cost of creating the anonymous function. So my PR made an optimization where if it saw f(:x, :y) it would just return f, since it's the same as the anonymous function above.

But more other expressions are still problematic. Consider

:x + 1

This will always have to turn into

x -> x + 1

for purposes of the src => fun => dest syntax in DataFrames.

It looks like this package will let me cash :(x -> x + 1) so that the anonymous function of that form is only compiled once, and later calls are taken from a lookup. Is that correct?

If so, DataFramesMeta seems like a good application of this package.

[ChrisRackauckas]

Yeah, it would just hash as the same function.

But do you want to compile so much? It seems like it would make it a Symbolics expression and then use substitute, you could get away without ever compiling just by interpreting the expression. If the runtime cost isn't a major factor then using interpreting could be the right approach.

[pdeffebach]

This is great news. I will file an issue at DataFramesMeta to remind myself to add this eventually.

We definitely need to compile, unfortunately. DataFramesMeta is really just an easy way of making calls like transform(df, :x => f => :y). So we need f to exist and be a function.

And yeah, runtime cost is a major factor since DataFrames has untyped columns. So we need function barriers for performance.