Heap histogram doesn't work in Spark UI

[paulpaul1076]

Hello, I saw that there's a new option in Spark UI, called heap histogram.

But when I click on it I get this:

In executor logs I get this:

java.io.IOException: Cannot run program "/usr/lib/jvm/java-11-openjdk-11.0.21.0.9-2.el8.x86_64/bin/jmap": error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1128) ~[?:?]
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) ~[?:?]
    at org.apache.spark.util.Utils$.getHeapHistogram(Utils.scala:2147) ~[spark-core_2.12-3.5.0.jar:3.5.0]
    at org.apache.spark.storage.BlockManagerStorageEndpoint$$anonfun$receiveAndReply$1.applyOrElse(BlockManagerStorageEndpoint.scala:82) ~[spark-core_2.12-3.5.0.jar:3.5.0]
    at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:103) [spark-core_2.12-3.5.0.jar:3.5.0]
    at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213) [spark-core_2.12-3.5.0.jar:3.5.0]
    at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100) [spark-core_2.12-3.5.0.jar:3.5.0]
    at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75) [spark-core_2.12-3.5.0.jar:3.5.0]
    at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41) [spark-core_2.12-3.5.0.jar:3.5.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
    at java.lang.ProcessImpl.<init>(ProcessImpl.java:340) ~[?:?]
    at java.lang.ProcessImpl.start(ProcessImpl.java:271) ~[?:?]
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1107) ~[?:?]
    ... 11 more

How do we get this to work?

I guess in order for this to work you have to rebundle a different version of the JDK that includes jmap?

[razvan]

Hey, yes you are correct.

It seems that UBI 8 only includes jmap in the devel version of the openjdk package.

To build your own image with jmap included you could do something like this:

FROM docker.stackable.tech/stackable/spark-k8s:3.5.0-stackable23.11.0

USER root

RUN microdnf update && \
    microdnf install java-11-openjdk-devel && \
    microdnf clean all

To use your own Spark image, add the custom property to the application resource like this:

apiVersion: spark.stackable.tech/v1alpha1
kind: SparkApplication
spec:
  sparkImage:
    custom: <your image name here>
    productVersion: "3.5.0"
    pullPolicy: IfNotPresent
...

Hope this helps.

[paulpaul1076]

@sbernauer @lfrancke thank you very much! @sbernauer I see that this image has the "-dev" suffix, is it stable enough to be used for critical things? Or does it include some stuff that may be less stable than the non-dev image?

[lfrancke]

Well.... It is our nightly image so it can change at any time and we would not officially recommend using it in production yet. But we haven't introduced anything breaking yet and if it works for you I'd be pretty confident that you can use it. I recommend pinning a hash as the tag itself is floating.
Our next release is planned for March.

I'm not entirely sure for how long we keep nightly images around right now but I'm sure one of the others can answer that.

[paulpaul1076]

I ran this image and got some weird exception when expiring iceberg table snapshots:

org.apache.spark.SparkException: EOFException occurred while reading the port number from pyspark.daemon's stdout.

Haven't had the time to understand why it happens, will try to do it tomorrow and also try the stable image with python installed. I just need to install python with microdnf, right?

[razvan]

Hey, Python is already installed in the Stackable image and there is a link to it that is set explicitely.
The problem seems to be that microdnf update also upated python it's self and removed that link.
It should be enough to restore it.

[paulpaul1076]

Thanks, now the dev image works, actually, seems like there was some other problem that resolved itself.