chore: Force pyiceberg reader for polars iceberg scan

blarghmatey · blarghmatey · commit e4db5bc8f06d · 2026-05-22T09:01:01.000-04:00
After the upgrade to 1.4 there have been persistent issues with Dagster runs hanging
open. The 1.4 release introduced the polars native Iceberg reader which seems to be the
culprit. This forces use of pyiceberg as the reader in an effort to address the problem.
diff --git a/packages/ol-orchestrate-lib/src/ol_orchestrate/lib/glue_helper.py b/packages/ol-orchestrate-lib/src/ol_orchestrate/lib/glue_helper.py
@@ -128,4 +128,6 @@ def get_dbt_model_as_dataframe(database_name: str, table_name: str) -> pl.LazyFr
     )
     table = glue.load_table(f"{database_name}.{table_name}")
 
-    return pl.scan_iceberg(table, storage_options=polars_s3_storage_options)
+    return pl.scan_iceberg(
+        table, reader_override="pyiceberg", storage_options=polars_s3_storage_options
+    )

Original file line number	Diff line number	Diff line change
`@@ -128,4 +128,6 @@ def get_dbt_model_as_dataframe(database_name: str, table_name: str) -> pl.LazyFr`
`128`	`128`	`)`
`129`	`129`	`table = glue.load_table(f"{database_name}.{table_name}")`
`130`	`130`
`131`		`- return pl.scan_iceberg(table, storage_options=polars_s3_storage_options)`
	`131`	`+ return pl.scan_iceberg(`
	`132`	`+ table, reader_override="pyiceberg", storage_options=polars_s3_storage_options`
	`133`	`+ )`