[BUG]: orchestrion + civisibility deadlock on parent-t.Parallel + subtest-t.Parallel persists in v2.8.1 (despite #4554 in v2.7.1)

[intel352]

Tracer Version(s)

dd-trace-go v2.8.1 (via datadog/test-visibility-github-action@v2.9.0), orchestrion v1.10.0

Go Version(s)

go1.25.10 linux/amd64 (GitHub-hosted ubuntu-latest runner)

Bug Report

The CI Visibility wrapper deadlocks in waitParallel against the parent-t.Parallel + subtest-t.Parallel pattern, which is the same shape that PR #4554 ("fix orchestrion deadlock with parallel subtests") shipped a fix for in v2.7.1. We are on v2.8.1 — after the fix — but still observe the deadlock in CI.

Stack trace (Linux runner, go test -p 4 -short ./... over a large package). Every blocked goroutine has the same shape: a subtest's call to t.Parallel() parks in testing.(*testState).waitParallel and never returns. The wrapper above it on the stack is gotesting.instrumentTestingTFunc.func1.1:

=== RUN   TestJobsHealthReadinessCheck
=== PAUSE TestJobsHealthReadinessCheck
=== RUN   TestJobsHealthLivenessCheck
=== PAUSE TestJobsHealthLivenessCheck
...
TestJobsHealthLivenessCheck (9m59s)
TestJobsHealthReadinessCheck (9m59s)
FAIL    github.com/.../internal/modules    600.439s

goroutine 96 [chan receive, 9 minutes]:
testing.(*testState).waitParallel(0xc000eddd60)
        /opt/hostedtoolcache/go/1.25.10/x64/src/testing/testing.go:2116 +0xaa
testing.(*T).Parallel(0xc0013601c0)
        /opt/hostedtoolcache/go/1.25.10/x64/src/testing/testing.go:1709 +0x259
TestJobsHealthReadinessCheck.func2(0xc0013601c0)
        .../jobs_health_test.go:329 +0x27
github.com/DataDog/dd-trace-go/v2/internal/civisibility/integrations/gotesting.instrumentTestingTFunc.func1.1(0xc0013601c0)
        /home/runner/go/pkg/mod/github.com/!data!dog/dd-trace-go/v2@v2.8.1/internal/civisibility/integrations/gotesting/instrumentation_orchestrion.go:335 +0x495
github.com/DataDog/dd-trace-go/v2/internal/civisibility/integrations/gotesting.instrumentTestingTFunc.func1(0xc0013601c0)
        /home/runner/go/pkg/mod/github.com/!data!dog/dd-trace-go/v2@v2.8.1/internal/civisibility/integrations/gotesting/instrumentation_orchestrion.go:339 +0x64f
testing.tRunner(0xc0013601c0, 0xc00158a8d0)
        /opt/hostedtoolcache/go/1.25.10/x64/src/testing/testing.go:1934 +0xea
created by testing.(*T).Run in goroutine 206
        /opt/hostedtoolcache/go/1.25.10/x64/src/testing/testing.go:1997 +0x47d

The real failing test (parent + subtest both call t.Parallel()):

func TestJobsHealthReadinessCheck(t *testing.T) {
    t.Parallel()

    t.Run("returns ready when health checker succeeds", func(t *testing.T) {
        t.Parallel()
        // ... in-process httptest.NewRequest against a chi.Router ...
    })

    t.Run("returns not_ready when health checker fails", func(t *testing.T) {
        t.Parallel()  // ← deadlocks here
        // ...
    })
}

Observed conditions

The CI deadlock is deterministic on Linux ubuntu-latest for a full package (~50 tests, multiple parents using parent-t.Parallel). Test binary hits the 10-minute go test timeout. Reproduces both with and without -race.

Reproduction Code

I tried to build a minimal reproducer. It does NOT deadlock on macOS / darwin/arm64 (go1.26.0, orchestrion v1.10.0, dd-trace-go v2.8.1) — passes consistently across 5+ runs. The deadlock seems to require either the Linux scheduler or a heavier per-package test count to saturate testState.maxParallel. Including the attempt here in case it helps narrow it down:

go.mod:

module example.com/orchestrion-parallel-deadlock-repro

go 1.26

require (
    github.com/DataDog/dd-trace-go/orchestrion/all/v2 v2.8.1
    github.com/DataDog/orchestrion v1.10.0
    github.com/stretchr/testify v1.10.0
)

parallel_test.go:

package repro

import (
    "net/http"
    "net/http/httptest"
    "testing"

    "github.com/stretchr/testify/assert"
)

func runOneSubtest(t *testing.T) {
    t.Parallel()
    srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
    }))
    t.Cleanup(srv.Close)
    resp, err := http.Get(srv.URL)
    if err != nil {
        t.Fatalf("get: %v", err)
    }
    resp.Body.Close()
    assert.Equal(t, http.StatusOK, resp.StatusCode)
}

func TestParentA(t *testing.T) {
    t.Parallel()
    for _, name := range []string{"a1", "a2", "a3", "a4"} {
        t.Run(name, runOneSubtest)
    }
}

func TestParentB(t *testing.T) {
    t.Parallel()
    for _, name := range []string{"b1", "b2", "b3", "b4"} {
        t.Run(name, runOneSubtest)
    }
}

func TestParentC(t *testing.T) {
    t.Parallel()
    for _, name := range []string{"c1", "c2", "c3", "c4"} {
        t.Run(name, runOneSubtest)
    }
}

func TestParentD(t *testing.T) {
    t.Parallel()
    for _, name := range []string{"d1", "d2", "d3", "d4"} {
        t.Run(name, runOneSubtest)
    }
}

Run with:

DD_CIVISIBILITY_ENABLED=true \
DD_API_KEY=fake DD_SITE=datadoghq.com \
DD_CIVISIBILITY_AGENTLESS_ENABLED=true \
  go test -toolexec='orchestrion toolexec' \
  -v -timeout=60s -parallel=2 ./...

Question

Two possibilities I see:

1. The fix from PR fix(internal/civisibility): fix orchestrion deadlock with parallel subtests #4554 covers the trivial-case shape (clone path with meta.originalTest != nil), but instrumentation_orchestrion.go:217-220 creates fresh metadata with originalTest = nil when no additional-feature wrapper is above the test. In that case instrumentTestingParallel(t) returns false and the call falls through to stdlib Parallel(). On a saturated maxParallel semaphore, the parent's wrapped state never releases the slot.
2. A regression introduced between v2.7.1 and v2.8.1 re-introduces the deadlock under load.

Happy to provide additional CI artifacts (full stack dump with all ~100+ goroutines, pprof profile, etc.) if useful for triage. The current mitigation is to disable the action on the affected workflows.

[intel352]

Update — confirmed local non-reproducibility on macOS / arm64 with the exact CI versions.

Pulled the failing project locally and ran the exact CI test command with the same versions the GitHub action installs:

• darwin/arm64, go1.26.0
• orchestrion v1.10.0 binary + github.com/DataDog/orchestrion v1.10.0 in go.mod
• github.com/DataDog/dd-trace-go/v2 v2.8.1
• github.com/DataDog/dd-trace-go/orchestrion/all/v2 v2.8.1

Command:

SKIP_BDD_TESTS=true \
DD_CIVISIBILITY_ENABLED=true DD_API_KEY=fake DD_SITE=datadoghq.com \
DD_CIVISIBILITY_AGENTLESS_ENABLED=true GOMAXPROCS=4 \
  go test -toolexec='orchestrion toolexec' \
    -p 4 -short -run='Test' -timeout=12m -v ./internal/modules/...

Result on macOS: PASS in 1.6 seconds. The exact same package's tests (including TestJobsHealthReadinessCheck and TestJobsHealthLivenessCheck with the parent-t.Parallel + subtest-t.Parallel pattern in jobs_health_test.go) complete cleanly. Dd-tracer v2.8.1 banner is visible in the output, confirming instrumentation is active.

Result on CI (GitHub-hosted ubuntu-latest, go1.25.10): deterministic 10-minute deadlock at instrumentTestingTFunc.func1.1 → user-test → t.Parallel() → waitParallel.

I also tried the project's own pre-fix pin (orchestrion v1.5.0 + dd-trace-go v2.3.0) locally on Mac — also PASS in 2 seconds. So the deadlock only manifests on Linux + GitHub Actions runner, with both v2.3.0 (pre-fix) and v2.8.1 (post-fix).

This narrows the trigger to one of:

1. Linux scheduler/maxParallel interaction that differs from macOS at the testing.(*testState).waitParallel level
2. GitHub-hosted runner CPU/IO characteristics (the ubuntu-latest runner has 4 cores, smaller than typical dev machines)
3. Something the datadog/test-visibility-github-action@v2.9.0 install script does that isn't replicated by go install ...@latest + orchestrion pin locally

I can keep trying to reproduce inside a Linux Docker container if it would help triage, but wanted to surface this asymmetry early in case it narrows the search for the regression vs. the v2.7.1 fix.

[intel352]

Update 2 — confirmed no-repro even in a 4-CPU Linux container with the exact CI versions.

Ran the same test against the same chimera package inside a golang:1.25 Docker container, constrained to 4 CPUs / 8 GB RAM to mirror the GitHub-hosted ubuntu-latest runner:

docker run --rm --cpus=4 --memory=8g -v "$(pwd):/workspace" \
  -v /tmp/dd-orchestrion-linux-repro.sh:/repro.sh:ro \
  -w /workspace golang:1.25 bash /repro.sh

Inside the container the script:

1. go install github.com/DataDog/orchestrion@v1.10.0
2. go get github.com/DataDog/orchestrion@v1.10.0 github.com/DataDog/dd-trace-go/v2@v2.8.1 github.com/DataDog/dd-trace-go/orchestrion/all/v2@v2.8.1
3. go mod tidy
4. GOMAXPROCS=4 go test -toolexec='orchestrion toolexec' -p 4 -short -run='Test' -timeout=10m -v ./internal/modules/...
5. with DD_CIVISIBILITY_ENABLED=true, DD_API_KEY=fake, DD_SITE=datadoghq.com, DD_CIVISIBILITY_AGENTLESS_ENABLED=true, DD_CIVISIBILITY_AUTO_INSTRUMENTATION_PROVIDER=github

Result: PASS in 0.982s. Zero waitParallel markers in output.

Setting	Local Docker (passes)	CI (deadlocks)
OS	linux/amd64 (golang:1.25)	linux/amd64 (ubuntu-24.04)
Go	go1.25.x	go1.25.10
CPUs	4	4
dd-trace-go	v2.8.1	v2.8.1
orchestrion	v1.10.0	~v1.10.0 (per action)
DD_CIVISIBILITY env	same	same
Test command	same	same

So the deadlock is NOT platform (linux vs darwin), arch (amd64 vs arm64), CPU-count (4), or version. The trigger appears to be something specific to either the GitHub-hosted runner environment or what the datadog/test-visibility-github-action@v2.9.0 install script does beyond go install + go get.

If anyone on the test-optimization team has access to a failing ubuntu-latest runner, the relevant data points to capture mid-deadlock would be:

• Goroutine dump (SIGQUIT against the test PID) — to see whether instrumentTestingParallel ever runs and what meta.originalTest looks like for the parked subtests
• Environment dump (env inside the test step) — to find any DD_* or GOFLAGS variable the action sets that a hand-rolled go install + go get doesn't
• The exact contents of the install script INSTALLATION_SCRIPT_URL resolves to at the time of the failing run

Happy to provide a stripped-down chimera reproduction repo if useful for upstream CI to attach to a runner of their own and capture artifacts.

[intel352]

Update 3 — still not reproducible after matching CI's GOFLAGS mechanism + 16 GB RAM.

Correction on prior comment

I claimed "they have access to a failing runner" — that was sloppy. To be clear: I have the failing CI artifacts and can capture whatever data is useful from a re-run. Adjusting the ask: if the team can describe what artifacts would narrow this down (full goroutine dump, specific env-var capture, sysctl values, etc.) I'll happily collect them from a fresh failing run and attach.

Confirmed CI install-script mechanism is GOFLAGS, not a -toolexec flag

From install_test_visibility_v14.sh:982-983 (checksum 73824cfac695...):

# Append orchestrion to the GOFLAGS variable to enable Test Optimization.
echo "GOFLAGS=${GOFLAGS} '-toolexec=orchestrion toolexec'"

The script's stdout is >> $GITHUB_ENV (per the action's Download and run configuration script step), so the resulting workflow-step environment has:

GOFLAGS=' '-toolexec=orchestrion toolexec''

…with the literal single quotes preserved (confirmed in the failing job's logs):

unit-fast  Run Unit Tests (Fast)  GOFLAGS:  '-toolexec=orchestrion toolexec'

This means every go invocation in subsequent steps uses orchestrion as a toolexec (including go list, go vet, internal go-tool subprocesses), not just the explicit go test command. My earlier local repros passed -toolexec as a CLI flag, which only applies to the single go test invocation. I reran with export GOFLAGS=" '-toolexec=orchestrion toolexec'" (matching the CI value exactly) to fix this asymmetry.

Updated Docker setup, still no repro

docker run --rm \
  --cpus=4 --memory=16g \
  -v "$(pwd):/workspace" -v /tmp/repro.sh:/repro.sh:ro \
  -w /workspace golang:1.25 bash /repro.sh

Inside the container:

• go install github.com/DataDog/orchestrion@v1.10.0
• go get github.com/DataDog/orchestrion@v1.10.0 github.com/DataDog/dd-trace-go/v2@v2.8.1 github.com/DataDog/dd-trace-go/orchestrion/all/v2@v2.8.1
• export GOFLAGS=" '-toolexec=orchestrion toolexec'" (literal CI value)
• GOMAXPROCS=4 go test -p 4 -short -run='Test' -timeout=10m -v ./internal/modules/...
• with DD_CIVISIBILITY_ENABLED=true DD_API_KEY=fake DD_SITE=datadoghq.com DD_CIVISIBILITY_AGENTLESS_ENABLED=true DD_CIVISIBILITY_AUTO_INSTRUMENTATION_PROVIDER=github

Result: PASS in 0.819 s. Zero waitParallel markers.

What's left as differences from CI

Setting	Local Docker (passes)	CI (deadlocks)
Image	golang:1.25 (Debian 12)	ubuntu-24.04
CPUs	4	4
RAM	16 GB	16 GB
Go	go1.25.x	go1.25.10
dd-trace-go	v2.8.1	v2.8.1
orchestrion	v1.10.0	v1.10.0
GOFLAGS	literal CI value	literal CI value
Test command	identical	identical
actions/setup-go@v5 shim	no	yes
actions/runner process supervisor	no	yes
Hosted-runner kernel/sysctl	no	yes

The most plausible remaining triggers:

• Something actions/setup-go@v5 does that bare go install doesn't (e.g. specific GOMODCACHE symlinks, env defaults, or GOROOT placement)
• The actions/runner agent's process tree affecting os.Exec/signal handling that orchestrion's toolexec depends on
• Kernel scheduler differences between Debian-12 (golang:1.25) and Ubuntu-24.04 hosted runners

If you point me at a specific artifact to capture from a deliberately-failing run (env dump, /proc/<pid>/stack, full goroutine pprof, runner-agent debug logs, kernel sysctl -a, ulimit settings, etc.), I can run it once and attach the output here.

[intel352]

Update 4 — narrowed: the deadlock is in the additional-feature-wrapper code path, which my local repros never exercise.

I was passing DD_API_KEY=fake throughout. With a fake key, the settings fetch returns 401 and applyAdditionalFeaturesToTestFunc short-circuits — there's no clone, no wrapper, no deadlock-prone code path. From v2.8.1 source:

// internal/civisibility/integrations/gotesting/instrumentation.go:200-212
func applyAdditionalFeaturesToTestFunc(f func(*testing.T), testInfo *commonInfo, parentExecMeta *testExecutionMetadata) func(*testing.T) {
    settings := integrations.GetSettings()

    _ = integrations.GetKnownTests()

    // If none of the additional features are enabled, return the original function.
    if !settings.TestManagement.Enabled && !settings.EarlyFlakeDetection.Enabled && !settings.FlakyTestRetriesEnabled {
        return f
    }
    ...

In our failing CI run, the dd-trace-go banner shows Datadog Tracer v2.8.1 and the auth IS succeeding (we don't see the same 401 / API key is missing cluster that local-fake-key runs produce). So at least one of TestManagement, EarlyFlakeDetection, or FlakyTestRetriesEnabled is true in the failing run, the clone-wrapper IS active, and the t.Parallel() from inside the wrapped clone hits the waitParallel deadlock.

That also explains why PR #4554's fix in instrumentTestingParallel (which forwards Parallel() to originalTest only when meta.originalTest != nil) didn't fully cover us: the clone path sets originalTest on the parent clone, but the call stack we deadlock at is a subtest's t.Parallel() on its own clone — and runSubtest (instrumentation_orchestrion.go:217-220) creates fresh metadata for that subtest clone with originalTest = nil:

execMeta := getTestMetadata(currentT)
if execMeta == nil {
    // Create fresh metadata when additional-feature wrappers were not executed above us.
    execMeta = createTestMetadata(currentT, nil)   // ← originalTest = nil
    defer deleteTestMetadata(currentT)
}

So instrumentTestingParallel(currentT) on that subtest sees meta.originalTest == nil and returns false, falling through to the stdlib Parallel(), which then parks the subtest in waitParallel on the cloned *testing.T's state machine.

Hypothesis to verify

The fix from PR #4554 covers the parent test parallel-forwarding case, but the subtest-on-a-clone case still falls through to stdlib Parallel() with originalTest = nil. When the parent test's wrapper is itself parked (which it is — the wrapper does wrappedFunc(t) and waits), no goroutine ever releases the parent's parallel-slot bookkeeping for the cloned subtests to inherit.

If correct, the fix would be to propagate the parent's originalTest (or equivalent) down to the freshly-created subtest meta at instrumentation_orchestrion.go:218 instead of passing nil.

How CTL can confirm before a DataDog patch

In our failing workflow, set one of these env vars at the test step to disable the wrapper:

• DD_TEST_MANAGEMENT_ENABLED=false
• DD_CIVISIBILITY_EARLY_FLAKE_DETECTION_ENABLED=false
• DD_CIVISIBILITY_FLAKY_RETRY_ENABLED=false

If disabling all three makes the deadlock go away, this narrows it to the wrapper path conclusively. Will try and report back.

Sorry for the false-no-repro signal earlier — the fake API key was the missing variable.

[intel352]

Update 5 — wrapper-path hypothesis empirically disproven. The deadlock survives EFD + Auto-Test-Retries being toggled off server-side.

I had Datadog Test Optimization project settings toggled to:

• Auto Test Retries: OFF
• Early Flake Detection: OFF
• Test Impact Analysis: OFF (already)
• Test Management: not enabled
• Code Coverage: OFF (note "doesn't support dd-trace-go")
• PR Comments: ON

…then re-enabled the action on the three failing workflows and pushed.

Result: the deadlock still fires with the same instrumentTestingTFunc.func1.1 → user-test → t.Parallel() → waitParallel shape, in MORE packages this time:

• unit-fast: timed out at 600s, FAIL github.com/.../backend/cmd/facade 600.317s, 75 waitParallel markers in the dump (different package than the original internal/modules failure)
• race: same 5-min timeout, internal/modules, identical pattern
• Slow Integration Tests (-p 1) and bdd (-run TestBDDFeatures) still pass — both avoid the parallel-subtest pattern

So my hypothesis ("clone wrapper is the trigger; toggling those features should short-circuit applyAdditionalFeaturesToTestFunc and remove the deadlock") was wrong. Either:

1. Server-side settings haven't propagated to the SDK's GetSettings() response yet (would explain everything if there's a cache TTL longer than a few minutes). Is there a documented propagation delay for these settings, or a way to force-refresh / bust the cache from the SDK side?
2. There's a non-wrapper code path that also triggers the deadlock. The fact that the failure moved to a different package (cmd/facade) on the second run also suggests the trigger is wider than I thought — it's whatever the standard instrumentTestingTFunc wrapper does on every t.Run, not just the applyAdditionalFeaturesToTestFunc clone path.

The cmd/facade test that newly deadlocked has the same parent-t.Parallel() + subtest-t.Parallel() shape that the jobs_health_test.go test had. So at minimum we know the bug isn't specific to any one test in chimera — it fires across multiple packages with this shape.

We've reverted to the disabled-on-failing-workflows state for now (PR #588 in CrisisTextLine/chimera) and will wait for upstream. Happy to capture any additional artifacts from a fresh failing run (settings response body, full goroutine pprof, env dump, etc.) if it'd help triage.

Sorry for the noise on the hypothesis — should have waited until empirical confirmation before posting.

[tonyredondo]

Thanks @intel352 for the very detailed investigation here. The amount of narrowing you did in the follow-up comments was extremely useful.

We re-reviewed the issue and your updates, and we believe this should be fixed by #4735, which is included in v2.9.0-rc.3.

Your diagnosis of the failing shape was right: the problematic pattern is the parent t.Parallel() + subtest t.Parallel() case, where the subtest ends up blocked in testing.(*testState).waitParallel. As you also called out, this was not specific to one test or package; the later cmd/facade failure with the same stack shape pointed to a more general scheduler-accounting problem around that test structure.

The missing piece was that our Datadog-managed test execution was not fully mirroring Go's own testing.tRunner behavior when a wrapped/cloned parent test owned parallel subtests. Go's test runner releases the parent test's parallel slot before unblocking and waiting for its parallel subtests. In the affected path, the wrapper could wait for the subtests while the parent clone had not released the parallel slot those subtests needed. That creates exactly the cycle shown in your dumps:

instrumentTestingTFunc.func1.1 -> user subtest -> t.Parallel() -> testing.(*testState).waitParallel

The fix in #4735 addresses the scheduler behavior at the shared cloned-test/subtest coordination point rather than only one feature-specific path. Concretely changes the parallel-subtest completion flow so that Datadog-managed test clones now:

• detect when the current cloned test owns parallel subtests,
• release the parent test state's parallel slot before closing the subtest barrier,
• wait for each child subtest to signal completion after they have been unblocked,
• reacquire the scheduler slot only when required for a non-parallel parent,
• account for cases where Parallel() was forwarded from the clone to the original *testing.T.

That should break the deadlock cycle for the parent/subtest shape you reported, including runs where no additional feature wrapper is active and runs where Auto Test Retries / Early Flake Detection own the cloned execution.

If you are willing, could you try v2.9.0-rc.3 in the affected workflow and let us know whether the deadlock is gone? It would be very helpful to confirm the mitigation against your real GitHub Actions environment.

[intel352]

Confirming v2.9.0-rc.3 fixes the deadlock for us — thanks @tonyredondo!

Setup we ran on the same three failing GitHub Actions workflows:

• Bumped datadog/test-visibility-github-action from v2.9.0 to v2.10.0 (commit SHA cdf298ef8277c8c3bec321499084ce5d9e4ff1e8).
• Added go-tracer-version: v2.9.0-rc.3 to each Configure Datadog Test Optimization step so orchestrion's toolexec-injected imports resolve through the bundled tracer that carries fix(ci-visibility): harden review edge cases #4735, rather than the action's default "latest stable" (which was still landing on the v2.8.x line that hit the deadlock).
• Also pinned github.com/DataDog/dd-trace-go/v2 and github.com/DataDog/dd-trace-go/orchestrion/all/v2 to v2.9.0-rc.3 in our backend's go.mod to keep the consumer-side module graph aligned with what orchestrion injects (a go.mod-only pin alone wasn't enough — confirmed in an earlier CI run where the runtime still loaded dd-trace-go/v2@v2.8.2 from the toolexec-injected path despite go.mod being pinned).

Results on the three previously-failing workflows (instrumentation enabled, instrumentation pinned to v2.9.0-rc.3):

Workflow	Before (v2.8.1/v2.8.2)	After (v2.9.0-rc.3)
Backend Tests (unit-fast)	10m timeout (deadlocked in internal/modules)	pass — 7m54s
Backend Race Detector (-race, -tags=integration)	22m17s fail (same deadlock, surfaced faster under race)	pass — 8m8s
Smoke Tests (go test against docker-compose stack)	failed (instrumented-runtime / service-startup interference)	pass — 10m15s

No t.Parallel() → waitParallel → instrumentTestingTFunc frames in any of the post-fix runs. The parent + subtest t.Parallel() shape that originally reproduced (internal/modules/jobs_health_test.go, internal/modules/routes_test.go) now runs cleanly with Auto Test Retries / Early Flake Detection still enabled server-side.

Happy to leave the v2.9.0-rc.3 pin in place until v2.9.0 GA ships — let me know if you want any additional dumps from the post-fix runs while the artifacts are still around.

[intel352]

@tonyredondo Is there any chance of this lib having a negative impact on production code?

I'm just thinking about the bug that was encountered, I'm unaware if it was exposing a behavior that is truly unique to testing, or whether a similar scenario could have been encountered in production and resulted in failure due to the orchestrion lib?