Disable verbose containerdebug logging in Spark Kubernetes operator

[rbreejen]

Affected Stackable version

v26.3.0

Affected Apache Spark-on-Kubernetes version

No response

Current and expected behavior

Hi all,
Is there any way to disable containerdebug when running Stackable Spark on Kubernetes?

By default, containerdebug appears to always be enabled, and it generates a large amount of unnecessary and verbose logging! Currently it seems like the spark-k8s-operator hardcodes the variable _STACKABLE_PRE_HOOK to containerdebug --output={VOLUME_MOUNT_PATH_LOG}/containerdebug-state.json --loop &.

We have tried overriding the variable _STACKABLE_PRE_HOOK by:

• spec.env with _STACKABLE_PRE_HOOK=""
• job.podOverrides.spec.containers[].env with _STACKABLE_PRE_HOOK=""
• envOverrides at role level or role group level.

However, none of these approaches appear to disable containerdebug. Is there a supported configuration option to turn it off, or is disabling it currently not possible?

Any help would be appreciated!

Possible solution

No response

Additional context

No response

Environment

No response

Would you like to work on fixing this bug?

None

[Techassi]

I'm afraid there is currently no way to disable containerdebug entirely, but that is definitely be something which we should support. I will bring this up to the team.

I'm curious, why do you feel like the output is too verbose? It should only be logged on container startup and then every 30 minutes.

[lfrancke]

Just to add more context: I do understand that it can be annoying - and potentially allowing it to be disabled is definitely on the table.
The reason we added it was due to support. We have many support cases where it is helpful to get some information from within the actual running Pod which can't be collected anymore after the fact. Especially network diagnostics.

So, our intentions were good and we did not mean to annoy you :)

As @Techassi said though: It'd be good to understand what exactly it is that you consider verbose/unnecessary.

[rbreejen]

Thanks for the context. I understand the support benefits of containerdebug, but I would prefer it to be opt-in rather than enabled by default. In production environments, diagnostic functionality is usually enabled explicitly when needed to keep logging consistent and avoid unnecessary noise. We also appear to be running an older containerdebug version, as we still see output every minute instead of every 30 minutes. Even with the reduced interval, this can add up significantly when running many Spark applications.

We have just found a workaround by overriding the containerdebug executable, but a supported option to disable it completely would still be highly appreciated.

[lfrancke]

Ah! You wrote that you're on SDP 26.3 :)
But we fixed that already in SDP 25.7, so you must be on a pretty old release and I encourage you to upgrade.
We'll release our latest SDP 26.7 version this or next week.

So, your point is valid but it should be much much better if you update to a newer version.
If you actually are on a newer version and still see this then let us know, that'd be a real bug.

[rbreejen]

Correct, the SDP operator is 26.3. We are currently using an older custom Spark image that is still based on Spark 3.5.1, which we plan to upgrade soon.

Hopefully, the implementation of this ticket will allow us to remove the containerdebug workaround.