Ability to import dlt signatures without a current SparkSession

[marijncv]

I've been happily using the dlt-with-debug library, but I'm running into an issue when importing the dlt signatures without an active SparkSession. I'm trying:

import dlt_with_debug.dlt_signatures as dlt

But the __init__.py calls the v2.py file which in turn tries to get the pipeline_id from the current SparkSession. But this fails when there is no active SparkSession.

I think my usecase can be achieved with an extra check in

dlt-with-debug/dlt_with_debug/v2.py

Line 8 in db7f9b3

pipeline_id = spark.conf.get("pipelines.id", None)

that first confirms there is an actual SparkSession before calling .conf.get() on it.

@souvik-databricks curious what you think! And I'd be happy to contribute via PR

[souvik-databricks]

Hi @marijncv Thank you for using the project and I saw your PR for the fix. Will review it and get it in by the end of this week.

[javamonkey79]

I am having the same issue, could you please merge the PR?

[javamonkey79]

@marijncv since @souvik-databricks is MIA, could you bumpversion in your fork?