Skip to content

Isolate integration tests per run to fix concurrent-run flakiness#378

Merged
AndreyVMarkelov merged 1 commit into
mainfrom
fix/integration-test-folder-isolation
Jul 7, 2026
Merged

Isolate integration tests per run to fix concurrent-run flakiness#378
AndreyVMarkelov merged 1 commit into
mainfrom
fix/integration-test-folder-isolation

Conversation

@AndreyVMarkelov

@AndreyVMarkelov AndreyVMarkelov commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Problem

The integration tests all shared a single hard-coded folder (/Testing/Dropbox.Api.Tests) on one shared Dropbox account. Each test's Initialize asserted the folder was empty and Cleanup emptied it, so overlapping CI runs (e.g. several open PRs building at once) raced on the folder's contents.

This surfaced as intermittent, non-reproducible failures such as:

  • Assert.AreEqual failed. Expected:<0>. Actual:<1>. in Initialize
  • entry-count mismatches (Expected:<3>. Actual:<2>)
  • path_lookup/not_found during Cleanup

max-parallel: 1 only serializes the OS matrix within a single workflow run — it does nothing across separate workflow runs, so concurrent PR builds collided.

Fix

Give each test run its own unique subfolder /Testing/Dropbox.Api.Tests/<guid>, created in ClassInitialize and removed in a new ClassCleanup. Concurrent runs can no longer see or delete each other's files.

Validation (local)

  • Sequential runs (5×) and single runs: 20/20 pass
  • Two concurrent full suites: previously both failed deterministically on the folder race; after the fix that race is gone (a run that would have failed now passes).
  • dotnet build clean; dotnet format --verify-no-changes --severity warn passes.

@codecov

codecov Bot commented Jul 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 53.81%. Comparing base (f2a7feb) to head (4bd1ac9).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #378   +/-   ##
=======================================
  Coverage   53.81%   53.81%           
=======================================
  Files          16       16           
  Lines        1403     1403           
  Branches      134      134           
=======================================
  Hits          755      755           
  Misses        608      608           
  Partials       40       40           
Flag Coverage Δ
integration 50.32% <ø> (ø)
unit 10.54% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@AndreyVMarkelov AndreyVMarkelov force-pushed the fix/integration-test-folder-isolation branch from 69b4bbf to 8aeae31 Compare July 7, 2026 20:22
The integration tests all shared a single hard-coded folder
(/Testing/Dropbox.Api.Tests) on one shared Dropbox account. Each test's
Initialize asserted the folder was empty and Cleanup emptied it, so
overlapping CI runs (e.g. several open PRs building at once) raced on
the folder's contents. This surfaced as intermittent failures like
"Assert.AreEqual failed. Expected:<0>. Actual:<1>", count mismatches,
and "path_lookup/not_found" during cleanup.

Give each test run its own unique subfolder
(/Testing/Dropbox.Api.Tests/<guid>) created in ClassInitialize and
removed in a new ClassCleanup, so concurrent runs can no longer see or
delete each other's files.
@AndreyVMarkelov AndreyVMarkelov force-pushed the fix/integration-test-folder-isolation branch from 8aeae31 to 4bd1ac9 Compare July 7, 2026 20:24
@AndreyVMarkelov AndreyVMarkelov merged commit 2943b98 into main Jul 7, 2026
13 checks passed
@AndreyVMarkelov AndreyVMarkelov deleted the fix/integration-test-folder-isolation branch July 7, 2026 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants