Info
| Info |
Value |
| Operating System |
rhel7 and windows |
| Geni Version |
0.0.38 |
| JDK |
11.0.10 |
| Spark Version |
3.1.2 |
Problem / Steps to reproduce
If zero-one.geni.core has been required, it creates a default spark session which impacts the behavior of calling g/create-spark-session
Specifically, geni.core loads geni.spark-context which loads geni.defaults, which creates a spark session in an atom which should probably be a delay
(def s (g/create-spark-session {:app-name “foo”}))
(g/spark-conf s)
; => {…:spark.app.name Geni app…}
; which is the wrong nameif requiring zero-one.geni.spark directly instead (as g), the spark session is correctly configured.
The incorrect behaviour takes effect if core is required at any point before creating the session, so it is a bit problematic. As above, maybe replacing the default with a delay will be sufficient to avoid this.
Thanks for your work on this library!
Info
Problem / Steps to reproduce
If
zero-one.geni.corehas been required, it creates a default spark session which impacts the behavior of callingg/create-spark-sessionSpecifically,
geni.coreloadsgeni.spark-contextwhich loadsgeni.defaults, which creates a spark session in anatomwhich should probably be adelayif requiring
zero-one.geni.sparkdirectly instead (asg), the spark session is correctly configured.The incorrect behaviour takes effect if core is required at any point before creating the session, so it is a bit problematic. As above, maybe replacing the default with a delay will be sufficient to avoid this.
Thanks for your work on this library!