Skip to content

Removing Wrong Spark Spans for Inactive Databricks Clusters #10651

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gh-worker-dd-mergequeue-cf854d merged 18 commits into from
Mar 5, 2026
Merged

Removing Wrong Spark Spans for Inactive Databricks Clusters #10651

Changes from 15 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
1189586
updating to get rid of wrong spark spans
larakulkarni1 Feb 20, 2026
5aeb249
fixed format
larakulkarni1 Feb 20, 2026
fb1450e
updating logs to test
larakulkarni1 Feb 23, 2026
1d3d147
fixing an error + adding right logs
larakulkarni1 Feb 24, 2026
f0be2cf
ran spotless
larakulkarni1 Feb 24, 2026
de2d1b6
added a log
larakulkarni1 Feb 24, 2026
9714fb0
added logs
larakulkarni1 Feb 27, 2026
7646838
added logs
larakulkarni1 Feb 27, 2026
5177a4f
Merge branch 'master' into lara.kulkarni/updatingTheSparkSPans
larakulkarni1 Mar 2, 2026
2efadac
spotless cleaning
larakulkarni1 Mar 2, 2026
06d44cd
removed logs
larakulkarni1 Mar 3, 2026
5057b9a
edits logs and added statments
larakulkarni1 Mar 3, 2026
20f26df
added spotless check fixes
larakulkarni1 Mar 3, 2026
98d3adb
added space back in
larakulkarni1 Mar 3, 2026
f2afee0
spotless check
larakulkarni1 Mar 3, 2026
8479570
updated logic
larakulkarni1 Mar 4, 2026
3be324d
Merge branch 'master' into lara.kulkarni/updatingTheSparkSPans
larakulkarni1 Mar 4, 2026
a82739e
Merge branch 'master' into lara.kulkarni/updatingTheSparkSPans
larakulkarni1 Mar 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion ...ommon/src/main/java/datadog/trace/instrumentation/spark/AbstractDatadogSparkListener.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,7 @@ public synchronized void onApplicationStart(SparkListenerApplicationStart applic
}

private void initApplicationSpanIfNotInitialized() {
if (applicationSpan != null) {
if (applicationSpan != null || isRunningOnDatabricks) {
return;
}

Expand Down Expand Up @@ -336,6 +336,13 @@ public synchronized void finishApplication(
}
initApplicationSpanIfNotInitialized();

if (applicationSpan == null) {
// On Databricks or streaming environments, the application span is not created.
// Flush any remaining traces and return.
tracer.flush();
return;
}
Copy link
Copy Markdown
Contributor

@charlesmyu charlesmyu Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking more closely at this code - it seems like we already try to handle Databricks jobs up above by checking for applicationSpan == null && jobCount > 0 and returning before we ever initialize the application span. Would it make sense to unify our updated logic into there?

This might also explain why we weren't seeing this issue previously - most customers don't start all purpose clusters and run nothing on them before they spin down

Copy link
Copy Markdown
Contributor Author

@larakulkarni1 larakulkarni1 Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about adding in the check in initApplicationSpanIfNotInitialized() because it's used in multiple methods (they have guards currently) but on the off chance it's used again in the future, adding the check in initApplicationSpanIfNotInitialized would prevent a DBX cluster having a spark span if the new code doesn't have a guard.

Copy link
Copy Markdown
Contributor

@charlesmyu charlesmyu Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For posterity, we discussed in person and we agreed that in that case we would prefer updating all callers to reflect the check being moved into initApplicationSpanIfNotInitialized. For now, though, we follow the existing pattern and update the check where it is called in finishApplication.


if (throwable != null) {
applicationSpan.addThrowable(throwable);
} else if (exitCode != 0) {
Expand Down