Skip to content

Run tests concurrently in CI#845

Merged
zhixiangli merged 2 commits into
fsspec:mainfrom
Yonghui-Lee:ci-tests
May 14, 2026
Merged

Run tests concurrently in CI#845
zhixiangli merged 2 commits into
fsspec:mainfrom
Yonghui-Lee:ci-tests

Conversation

@Yonghui-Lee
Copy link
Copy Markdown
Collaborator

@Yonghui-Lee Yonghui-Lee commented May 13, 2026

This PR enables concurrent test execution in CI using pytest-xdist to speed up the build process. It also addresses several test failures that were exposed by running against the emulator in a parallel environment.

Key Changes

CI Infrastructure & Parallelization

  • Enabled pytest-xdist: Added -n auto to pytest calls in .github/workflows/ci.yml and added pytest-xdist to environment_gcsfs.yaml and pyproject.toml.
  • Isolated Worker Buckets: Modified gcsfs/tests/settings.py to append the xdist worker ID (e.g., _gw0) to bucket names. This ensures that parallel workers do not conflict by operating on the same buckets.
  • Global Emulator Setup: Moved the GCS emulator startup to a dedicated step in ci.yml and set STORAGE_EMULATOR_HOST globally. This ensures all workers share a single emulator instance rather than trying to spin up conflicting docker containers.

Test Fixes & Cleanups

  • Skipped test_sign on Emulator: Skipped this test entirely when running against the emulator. The emulator does not support signed URL validation.
  • Skipped test_credentials_from_raw_token on Emulator: Skipped this test on the emulator because the emulator accepts any token, causing the test to fail when it expects an invalid credentials error.
  • Consolidated Dummy Credentials: Deleted gcsfs/tests/fake-secret.json which was malformed (invalid PEM format) and caused DefaultCredentialsError in extended tests. Updated CI to use fake-service-account-credentials.json which contains a validly formatted dummy key.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 13, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.19%. Comparing base (991faba) to head (36e6fe2).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #845      +/-   ##
==========================================
- Coverage   88.52%   88.19%   -0.34%     
==========================================
  Files          15       15              
  Lines        2989     2989              
==========================================
- Hits         2646     2636      -10     
- Misses        343      353      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread gcsfs/tests/test_credentials.py
Comment thread gcsfs/tests/settings.py Outdated
Comment thread gcsfs/tests/test_core.py
@zhixiangli zhixiangli merged commit e768973 into fsspec:main May 14, 2026
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants