Support loading datasets as `polars.LazyFrame`

[damonbayer]

It would be nice to be able to load data as a LazyFrame (e.g. with polars.scan_parquet and polars.scan_csv), which can be much more efficient than loading the entire data ahead of time.

[ryanraaschCDC]

We already have an option to return a lazy dataframe but are you specifically wanting it with the scan function? Here is the current implementation:

cfa-dataops/cfa/dataops/catalog.py

Line 393 in 0addb08

pl_lazy: bool = False,

[damonbayer]

Yes

Calling read_csv().lazy() is an antipattern as this forces Polars to materialize a full csv file and therefore cannot push any optimizations into the reader. Therefore always prefer scan_csv if you want to work with LazyFrames.

From https://docs.pola.rs/api/python/stable/reference/api/polars.read_csv.html#polars.read_csv

[ryanraaschCDC]

Gotcha. This was setup by someone else. I'll implement the right way to create a lazyframe.