Skip to content

test(integration): extend loader-hang workaround to Node 22.0.0#9422

Closed
tlhunter wants to merge 1 commit into
masterfrom
tlhunter/fix-node22-loader-ssi-hang
Closed

test(integration): extend loader-hang workaround to Node 22.0.0#9422
tlhunter wants to merge 1 commit into
masterfrom
tlhunter/fix-node22-loader-ssi-hang

Conversation

@tlhunter

Copy link
Copy Markdown
Member

What does this PR do?

Extends the existing setShouldKill workaround in integration-tests/init.spec.js (describe('initialize.mjs', ...)) to also cover Node 22.0.0, so short-lived --loader-based children that print their expected output but never exit get force-killed after a 1s grace period, same as is already done for Node 20.0.0.

Motivation

The integration-guardrails (22.0.0) CI job intermittently fails with Error: Timeout of 30000ms exceeded inside initialize.mjs > as --loader > ... > when dd-trace in the app dir > with DD_INJECTION_ENABLED, with a different sub-test timing out on different runs (ruling out a bug in one specific assertion). GitHub Actions logs an orphaned node process ("Terminate orphan process") at job cleanup, confirming a child process is genuinely hanging rather than a slow assertion.

This is pre-existing on master, unrelated to PR #8863 — confirmed by checking multiple recent master CI runs where this exact job/file/timeout signature recurs.

Root cause: initialize.mjs calls Module.register('./loader-hook.mjs', import.meta.url) when used with --loader. Node's module.register() runs its hooks on a dedicated internal worker thread. I reproduced the hang locally by installing Node 22.0.0 via nvm and looping ./node_modules/.bin/mocha --timeout 30000 --grep "as --loader" integration-tests/init.spec.js: the child scripts (init/trace.mjs, init/instrument.mjs) print their expected output and call process.exit(), but the process never exits — ps aux showed the leaked node process still alive with 12 threads (the hooks worker + libuv pool) after the test's own timeout/kill. runAndCheckOutput's proc.once('exit', ...) promise in integration-tests/helpers/index.js therefore never resolves, and mocha's 30s test timeout fires.

Node 20.0.0 and 22.0.0 are each the first release of their major line to ship module.register's off-thread hooks worker, and both shipped with known bugs where the hooks worker thread isn't reliably torn down around process.exit() (see nodejs/node#52706, #53097, #53182, and the "propagate process.exit from the loader thread to the main thread" fix) — bugs that were fixed in later patch releases of each line. This matches the comment already present for the Node 20.0.0 case in this file, just one major line later.

Fix: rather than papering over the symptom with a longer timeout (which wouldn't actually resolve — the child never exits at all, so no timeout is "long enough"), this extends the existing, purpose-built setShouldKill grace-period-kill mechanism to Node 22.0.0 as well, matching the established pattern and comment style for the Node 20.0.0 case.

Additional Notes

  • This is a Node.js runtime bug in the ESM loader hooks worker, not a leak in dd-trace's own source — the same class of bug is already worked around for Node 20.0.0 in this file, so this PR follows the same approach rather than introducing a new mechanism.
  • Verified locally: installed Node 22.0.0 via nvm, reproduced the hang and the orphaned process before this change, then confirmed no 30s timeouts and no orphaned processes across repeated runs of integration-tests/init.spec.js after applying the fix.
  • No production code paths are touched; this only affects the integration test harness.

🤖 Generated with Claude Code

integration-tests/init.spec.js intermittently timed out (30000ms) in the
initialize.mjs > as --loader > ... > with DD_INJECTION_ENABLED describe
block, with GitHub Actions logging an orphaned node process at job
cleanup ("Terminate orphan process").

Reproduced locally on Node 22.0.0: after the spawned child prints its
expected output and calls process.exit(), the process leaves behind a
live loader hooks worker thread and never exits, so
runAndCheckOutput's proc.once('exit', ...) promise never resolves.
This is the same class of bug already worked around for Node 20.0.0
(the first minor of the 20.x line to ship module.register's off-thread
hooks worker): both 20.0.0 and 22.0.0 are the first patch release of
their major line to ship this worker, and both shipped with known
process.exit()/exit-propagation bugs in it (nodejs/node#52706, #53097,
#53182) that were fixed in later patch releases.

Extend setShouldKill's existing grace-period-then-kill workaround to
also cover Node 22.0.0, matching the Node 20.0.0 case.
@dd-octo-sts

dd-octo-sts Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 6.77 MB
Deduped: 7.43 MB
No deduping: 7.43 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.1 | 122.62 kB | 438.86 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@datadog-official

datadog-official Bot commented Jul 17, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog retried 1 test - 1 passed on retry View in Datadog

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 98.34% (+0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 3c587fd | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Jul 17, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-07-17 23:29:53

Comparing candidate commit 3c587fd in PR branch tlhunter/fix-node22-loader-ssi-hang with baseline commit 57d5f96 in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2310 metrics, 48 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:appsec-appsec-enabled-24

  • unstable execution_time [-166.683ms; +228.168ms] or [-6.368%; +8.717%]

scenario:appsec-appsec-enabled-26

  • unstable execution_time [-229379.848µs; +230081.682µs] or [-8.970%; +8.997%]

scenario:appsec-appsec-enabled-with-attacks-24

  • unstable execution_time [-155.990ms; +161.263ms] or [-5.051%; +5.222%]

scenario:appsec-appsec-enabled-with-attacks-26

  • unstable execution_time [-191891.821µs; +190549.055µs] or [-6.601%; +6.555%]

scenario:appsec-control-20

  • unstable execution_time [-123317.652µs; +124981.786µs] or [-7.491%; +7.592%]

scenario:appsec-control-24

  • unstable execution_time [-110914.303µs; +111533.403µs] or [-8.969%; +9.019%]

scenario:appsec-control-26

  • unstable execution_time [-130.196ms; +106.638ms] or [-10.602%; +8.684%]

scenario:appsec-iast-no-vulnerability-control-20

  • unstable execution_time [-13257.931µs; +14129.349µs] or [-5.167%; +5.507%]

scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-20

  • unstable execution_time [-16.358ms; +22.882ms] or [-6.192%; +8.661%]

scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-20

  • unstable execution_time [-11.311ms; +22.826ms] or [-4.418%; +8.916%]

scenario:appsec-iast-with-vulnerability-control-20

  • unstable execution_time [-27.397ms; +32.069ms] or [-4.949%; +5.793%]

scenario:appsec-iast-with-vulnerability-iast-enabled-default-config-20

  • unstable execution_time [-24.455ms; +31.311ms] or [-4.441%; +5.686%]

scenario:child_process-file-args-24

  • unstable execution_time [-23.720ms; +27.934ms] or [-4.950%; +5.830%]

scenario:debugger-line-probe-with-snapshot-default-24

  • unstable cpu_user_time [-1979.399ms; +3102.675ms] or [-23.944%; +37.532%]
  • unstable execution_time [-1979.512ms; +3129.644ms] or [-22.075%; +34.902%]
  • unstable instructions [-16.8G instructions; +26.7G instructions] or [-24.941%; +39.494%]
  • unstable max_rss_usage [-8.008MB; +12.941MB] or [-5.107%; +8.253%]
  • unstable throughput [-841.300op/s; +535.126op/s] or [-22.917%; +14.577%]

scenario:debugger-line-probe-with-snapshot-default-26

  • unstable cpu_user_time [-6.032s; +2.951s] or [-45.104%; +22.067%]
  • unstable execution_time [-6.154s; +3.030s] or [-43.518%; +21.427%]
  • unstable instructions [-53.1G instructions; +25.9G instructions] or [-46.636%; +22.735%]
  • unstable max_rss_usage [-17.804MB; +8.957MB] or [-10.368%; +5.216%]
  • unstable throughput [-591.857op/s; +1213.670op/s] or [-23.871%; +48.951%]

scenario:debugger-line-probe-with-snapshot-minimal-24

  • unstable cpu_user_time [-1710.617ms; +557.056ms] or [-20.719%; +6.747%]
  • unstable execution_time [-1720.814ms; +592.149ms] or [-19.225%; +6.616%]
  • unstable instructions [-14.6G instructions; +4.8G instructions] or [-21.691%; +7.094%]
  • unstable throughput [-168.683op/s; +465.488op/s] or [-4.586%; +12.654%]

scenario:debugger-line-probe-with-snapshot-minimal-26

  • unstable cpu_user_time [-5.476s; +2.363s] or [-46.140%; +19.909%]
  • unstable execution_time [-5.530s; +2.386s] or [-43.851%; +18.924%]
  • unstable instructions [-48.3G instructions; +21.0G instructions] or [-48.187%; +20.923%]
  • unstable max_rss_usage [-16.449MB; +6.909MB] or [-9.791%; +4.113%]
  • unstable throughput [-473.231op/s; +1076.034op/s] or [-17.043%; +38.753%]

scenario:debugger-line-probe-without-snapshot-24

  • unstable cpu_user_time [-1710.600ms; +578.244ms] or [-20.753%; +7.015%]
  • unstable execution_time [-1734.036ms; +585.538ms] or [-19.353%; +6.535%]
  • unstable instructions [-14.6G instructions; +4.8G instructions] or [-21.614%; +7.137%]
  • unstable throughput [-164.796op/s; +467.667op/s] or [-4.484%; +12.725%]

scenario:debugger-line-probe-without-snapshot-26

  • unstable cpu_user_time [-2608.141ms; +4160.040ms] or [-27.443%; +43.772%]
  • unstable execution_time [-2624.595ms; +4200.912ms] or [-25.642%; +41.042%]
  • unstable instructions [-23.1G instructions; +37.0G instructions] or [-29.064%; +46.480%]
  • unstable max_rss_usage [-7.434MB; +14.115MB] or [-4.670%; +8.866%]
  • unstable throughput [-828.745op/s; +515.342op/s] or [-25.579%; +15.906%]

scenario:dogstatsd-with-tags-20

  • unstable cpu_user_time [-377.301ms; +303.951ms] or [-7.925%; +6.384%]
  • unstable execution_time [-380.658ms; +303.755ms] or [-7.867%; +6.278%]
  • unstable throughput [-105516.779op/s; +137133.421op/s] or [-6.086%; +7.909%]

scenario:plugin-claude-agent-sdk-compact-stream-scan-24

  • unstable cpu_usage_percentage [-7.289%; +3.193%]

scenario:plugin-pg-service-26

  • unstable cpu_usage_percentage [-6.429%; +7.726%]
  • unstable execution_time [-102.308ms; +73.250ms] or [-11.152%; +7.985%]
  • unstable throughput [-434307.636op/s; +532880.322op/s] or [-6.482%; +7.953%]

@codecov

codecov Bot commented Jul 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.34%. Comparing base (57d5f96) to head (3c587fd).

Additional details and impacted files
@@           Coverage Diff            @@
##           master    #9422    +/-   ##
========================================
  Coverage   98.34%   98.34%            
========================================
  Files         924      924            
  Lines      123056   123056            
  Branches    10924    10609   -315     
========================================
+ Hits       121016   121024     +8     
+ Misses       2040     2032     -8     
Flag Coverage Δ
aiguard 58.22% <ø> (-0.03%) ⬇️
aiguard-integration 57.08% <ø> (ø)
apm-bucket-0 58.48% <ø> (-0.03%) ⬇️
apm-bucket-1 64.59% <ø> (-0.03%) ⬇️
apm-bucket-2 63.64% <ø> (-0.03%) ⬇️
apm-bucket-3 60.93% <ø> (-0.03%) ⬇️
apm-capabilities-tracing 62.31% <ø> (ø)
apm-integrations-aerospike 57.61% <ø> (-0.03%) ⬇️
apm-integrations-confluentinc-kafka-javascript 62.50% <ø> (-0.03%) ⬇️
apm-integrations-couchbase 57.89% <ø> (-0.02%) ⬇️
apm-integrations-http 63.63% <ø> (-0.03%) ⬇️
apm-integrations-kafkajs 63.09% <ø> (-0.03%) ⬇️
apm-integrations-next 60.02% <ø> (-0.03%) ⬇️
apm-integrations-prisma 59.45% <ø> (-0.03%) ⬇️
appsec 73.92% <ø> (-0.02%) ⬇️
appsec-express_fastify_graphql 71.53% <ø> (-0.02%) ⬇️
appsec-integration 51.90% <ø> (+0.14%) ⬆️
appsec-kafka_ldapjs_lodash 64.82% <ø> (-0.03%) ⬇️
appsec-mongodb-core_mongoose_mysql 68.55% <ø> (-0.02%) ⬇️
appsec-next 58.19% <ø> (-0.03%) ⬇️
appsec-node-serialize_passport_postgres 68.24% <ø> (-0.02%) ⬇️
appsec-sourcing_stripe_template 66.54% <ø> (-0.02%) ⬇️
debugger 65.88% <ø> (-0.01%) ⬇️
instrumentations-bucket-0 52.39% <ø> (-0.03%) ⬇️
instrumentations-bucket-1 61.12% <ø> (-0.03%) ⬇️
instrumentations-bucket-10 62.98% <ø> (-0.03%) ⬇️
instrumentations-bucket-11 52.40% <ø> (-0.03%) ⬇️
instrumentations-bucket-12 52.85% <ø> (-0.03%) ⬇️
instrumentations-bucket-13 52.35% <ø> (-0.03%) ⬇️
instrumentations-bucket-2 54.32% <ø> (-0.03%) ⬇️
instrumentations-bucket-3 60.09% <ø> (-0.03%) ⬇️
instrumentations-bucket-4 52.98% <ø> (-0.03%) ⬇️
instrumentations-bucket-5 58.32% <ø> (-0.03%) ⬇️
instrumentations-bucket-6 61.68% <ø> (-0.03%) ⬇️
instrumentations-bucket-7 59.22% <ø> (-0.03%) ⬇️
instrumentations-bucket-8 60.49% <ø> (-0.03%) ⬇️
instrumentations-bucket-9 62.35% <ø> (-0.03%) ⬇️
instrumentations-instrumentation-couchbase 51.91% <ø> (-0.03%) ⬇️
instrumentations-integration-esbuild 34.05% <ø> (ø)
llmobs-ai_anthropic_bedrock 63.32% <ø> (-0.03%) ⬇️
llmobs-bucket-1 62.61% <ø> (-0.02%) ⬇️
llmobs-openai 63.29% <ø> (-0.03%) ⬇️
llmobs-sdk 65.41% <ø> (-0.03%) ⬇️
llmobs-vertex-ai 59.97% <ø> (-0.03%) ⬇️
master-coverage 98.34% <ø> (?)
openfeature 54.70% <ø> (+<0.01%) ⬆️
openfeature-unit 53.43% <ø> (-0.03%) ⬇️
platform-core_esbuild_instrumentations-misc 40.67% <ø> (-0.02%) ⬇️
platform-integration 62.31% <ø> (ø)
platform-shimmer_unit-guardrails_webpack 39.34% <ø> (-0.02%) ⬇️
plugins-bucket-0 57.76% <ø> (-0.03%) ⬇️
plugins-bucket-1 55.17% <ø> (ø)
plugins-bucket-11 63.09% <ø> (+0.34%) ⬆️
plugins-bucket-17 ?
plugins-bucket-18 62.90% <ø> (-0.55%) ⬇️
plugins-bucket-19 60.96% <ø> (-1.60%) ⬇️
plugins-bucket-20 62.90% <ø> (-2.40%) ⬇️
plugins-bucket-4 59.39% <ø> (-0.03%) ⬇️
plugins-bullmq_cassandra_cookie 62.68% <ø> (-0.03%) ⬇️
plugins-cookie-parser_crypto_dd-trace-api 57.56% <ø> (-0.03%) ⬇️
plugins-fetch_fs_generic-pool 59.62% <ø> (-0.03%) ⬇️
plugins-google-cloud-pubsub_grpc_handlebars 65.67% <ø> (-0.03%) ⬇️
plugins-hapi_hono_ioredis 61.10% <ø> (-0.03%) ⬇️
plugins-jest_knex_langgraph 56.36% <ø> (?)
plugins-jest_langgraph_ldapjs ?
plugins-ldapjs_light-my-request_limitd-client 59.34% <ø> (?)
plugins-light-my-request_limitd-client_lodash ?
plugins-lodash_mariadb_memcached 58.85% <ø> (?)
plugins-mariadb_memcached_mercurius ?
plugins-moleculer_mongodb_mongodb-core 62.84% <ø> (?)
plugins-mongodb_mongodb-core_mongoose ?
plugins-mongoose_multer_mysql 59.85% <ø> (?)
plugins-multer_mysql_mysql2 ?
plugins-mysql2_nats_node-serialize 61.63% <ø> (?)
plugins-nats_node-serialize_opensearch ?
plugins-opensearch_passport-http_pino 60.41% <ø> (?)
plugins-passport-http_pino_postgres ?
plugins-postgres_process_pug 59.13% <ø> (?)
plugins-process_pug_redis ?
plugins-redis_router_sequelize 62.97% <ø> (?)
plugins-test-and-upstream-rhea_undici_url 62.54% <ø> (?)
plugins-undici_url_valkey ?
plugins-valkey_vm_winston 58.83% <ø> (?)
plugins-vm_winston_ws ?
plugins-ws 60.46% <ø> (?)
profiling 63.07% <ø> (-0.03%) ⬇️
serverless-aws-sdk-aws-sdk 55.71% <ø> (-0.03%) ⬇️
serverless-aws-sdk-bedrockruntime 55.43% <ø> (-0.03%) ⬇️
serverless-aws-sdk-client 57.17% <ø> (-0.03%) ⬇️
serverless-aws-sdk-dynamodb 56.37% <ø> (-0.03%) ⬇️
serverless-aws-sdk-eventbridge 49.93% <ø> (-0.03%) ⬇️
serverless-aws-sdk-kinesis 60.15% <ø> (-0.03%) ⬇️
serverless-aws-sdk-lambda 58.14% <ø> (-0.03%) ⬇️
serverless-aws-sdk-s3 56.30% <ø> (-0.03%) ⬇️
serverless-aws-sdk-serverless-peer-service 60.56% <ø> (-0.03%) ⬇️
serverless-aws-sdk-sns 60.96% <ø> (-0.03%) ⬇️
serverless-aws-sdk-sqs 61.40% <ø> (-0.03%) ⬇️
serverless-aws-sdk-stepfunctions 56.29% <ø> (-0.03%) ⬇️
serverless-aws-sdk-util 52.19% <ø> (-0.03%) ⬇️
serverless-bucket-0 55.22% <ø> (-0.01%) ⬇️
serverless-bucket-1 60.09% <ø> (-0.03%) ⬇️
test-optimization-cucumber 73.04% <ø> (+0.02%) ⬆️
test-optimization-cypress 66.42% <ø> (+0.08%) ⬆️
test-optimization-jest 74.44% <ø> (+<0.01%) ⬆️
test-optimization-mocha 74.73% <ø> (+0.04%) ⬆️
test-optimization-playwright-playwright-atr 61.43% <ø> (-0.01%) ⬇️
test-optimization-playwright-playwright-efd 61.62% <ø> (ø)
test-optimization-playwright-playwright-final-status 61.59% <ø> (-0.01%) ⬇️
test-optimization-playwright-playwright-impacted-tests 61.53% <ø> (+0.38%) ⬆️
test-optimization-playwright-playwright-reporting 61.21% <ø> (-0.01%) ⬇️
test-optimization-playwright-playwright-test-management 62.15% <ø> (+<0.01%) ⬆️
test-optimization-playwright-playwright-test-span 61.34% <ø> (-0.06%) ⬇️
test-optimization-selenium 60.72% <ø> (-0.16%) ⬇️
test-optimization-testopt 59.21% <ø> (+0.08%) ⬆️
test-optimization-vitest 71.38% <ø> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@BridgeAR BridgeAR left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will actually not resolve the flakiness.

We need to deactivate the maglev compiler and or deactivate code compilation. I landed a fix in iitm that should resolve it (deactivating the code compilation)

@tlhunter tlhunter closed this Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants