Skip to content

fix: guard sort() with shuffle check so --shuffle is not silently ignored#5638

Closed
kapil971390 wants to merge 1856 commits into
codeceptjs:masterfrom
kapil971390:fix/shuffle-overwritten-by-sort
Closed

fix: guard sort() with shuffle check so --shuffle is not silently ignored#5638
kapil971390 wants to merge 1856 commits into
codeceptjs:masterfrom
kapil971390:fix/shuffle-overwritten-by-sort

Conversation

@kapil971390

Copy link
Copy Markdown
Contributor

Fixes #5605

What

run() calls this.testFiles.sort() unconditionally after loadTests() has already applied shuffle() when --shuffle is set. The sort overwrites the randomised order every time, making --shuffle silently have no effect.

// loadTests() — line 217
if (this.opts.shuffle) {
  this.testFiles = shuffle(this.testFiles)  // ✅ shuffled
}

// run() — line 293 (added by #5438)
this.testFiles.sort()  // ❌ always runs — shuffle is lost

Fix

Guard the sort with !this.opts.shuffle:

// Sort alphabetically for consistent order, but preserve shuffle when active.
if (!this.opts.shuffle) {
  this.testFiles.sort()
}

Normal runs continue to get alphabetical order. --shuffle runs now keep their randomised order.

Why this happened

The sort() was introduced in #5438 to fix worker suite distribution — a correct fix for that issue, but it introduced this regression for --shuffle users.

Change

Single guard condition added to lib/codecept.js — no other behaviour changed.

dependabot Bot and others added 30 commits October 14, 2025 17:45
* implemented aria selectors for PW/WebDriver/Puppeteer

* added aria elements

* Implemented by role selector and aria locators with tests

* fixed aria selectors for WebDriverIO

* added tests, reverted runok

* 4.0.0-beta.1

* fixed aria tests

* fixed WD tests

* improved output

* fixed webdriver types

---------

Co-authored-by: DavertMik <davert@testomat.io>
Dependabot couldn't find the original pull request head commit, 25a49b7.

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…(4.x) (codeceptjs#5191)

* fix(utils): remove incorrect `async` from `emptyFolder`

* fix(utils): resolve command injection vulnerability in `emptyFolder`

---------

Co-authored-by: kobenguyent <7845001+kobenguyent@users.noreply.github.com>
Bumps [tsd](https://github.com/tsdjs/tsd) from 0.32.0 to 0.33.0.
- [Release notes](https://github.com/tsdjs/tsd/releases)
- [Commits](tsdjs/tsd@v0.32.0...v0.33.0)

---
updated-dependencies:
- dependency-name: tsd
  dependency-version: 0.33.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [envinfo](https://github.com/tabrindle/envinfo) from 7.14.0 to 7.19.0.
- [Release notes](https://github.com/tabrindle/envinfo/releases)
- [Changelog](https://github.com/tabrindle/envinfo/blob/main/CHANGELOG.md)
- [Commits](tabrindle/envinfo@v7.14.0...v7.19.0)

---
updated-dependencies:
- dependency-name: envinfo
  dependency-version: 7.19.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 4 to 6.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](actions/setup-node@v4...v6)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps-dev): bump typescript from 5.8.3 to 5.9.3

Bumps [typescript](https://github.com/microsoft/TypeScript) from 5.8.3 to 5.9.3.
- [Release notes](https://github.com/microsoft/TypeScript/releases)
- [Changelog](https://github.com/microsoft/TypeScript/blob/main/azure-pipelines.release-publish.yml)
- [Commits](microsoft/TypeScript@v5.8.3...v5.9.3)

---
updated-dependencies:
- dependency-name: typescript
  dependency-version: 5.9.3
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update typedoc version to ^0.28.14

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kobenguyent <7845001+kobenguyent@users.noreply.github.com>
* refactor: migrate Appium helper to ESM

- Convert require() to import statements
- Change webdriverio from dynamic require to static import
- Import webdriverio as namespace (import * as webdriverio)
- Import all dependencies at the top of the file:
  - ElementNotFound from './errors/ElementNotFound.js'
  - dontSeeElementError from './errors/ElementAssertion.js'
- Remove inline require statements from methods:
  - dontSeeElement()
  - seeElement()
  - waitForVisible()
  - waitForInvisible()
- Use Locator.build() instead of new Locator() for ESM compatibility
- Maintain export default Appium at the end
- All imports use .js extensions for ESM compliance

Verified:
- Module loads successfully
- No require() statements remaining
- Tests run without errors

* enable appium tests

* fix: initialize chai.should() in Appium tests

The test was failing with 'Cannot read properties of undefined (reading "be")'
because chai's should assertion style was not initialized.

Added chai.should() call after imports to enable should-style assertions
throughout the test file.

Fixes test: device lock : #seeDeviceIsLocked, #seeDeviceIsUnlocked
* Fix: preserve global timeout with BeforeSuite hook

* test: clarify feature name for global timeout with BeforeSuite

* chore: remove unused plugins field from beforeSuiteTimeout config
Bumps [ts-morph](https://github.com/dsherret/ts-morph) from 26.0.0 to 27.0.2.
- [Release notes](https://github.com/dsherret/ts-morph/releases)
- [Commits](dsherret/ts-morph@26.0.0...27.0.2)

---
updated-dependencies:
- dependency-name: ts-morph
  dependency-version: 27.0.2
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [semver](https://github.com/npm/node-semver) from 7.7.2 to 7.7.3.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/main/CHANGELOG.md)
- [Commits](npm/node-semver@v7.7.2...v7.7.3)

---
updated-dependencies:
- dependency-name: semver
  dependency-version: 7.7.3
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 5 to 6.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](actions/setup-node@v5...v6)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
- Added transpilation of .ts files using TypeScript compiler
- Creates temporary .mjs file to load transpiled code
- Cleans up temporary files after loading
- Fixes 'Unknown file extension .ts' error when using steps_file.ts
- Recursively transpile TypeScript files and their dependencies
- Handle .js imports that reference .ts source files
- Replace import paths in transpiled code to point to temp .mjs files
- Clean up all temporary files on completion or error
- Add comprehensive tests for TypeScript support with cross-file imports

Fixes issue where steps_file.ts importing other .ts files would fail with
'Cannot find module' error. Now properly transpiles all dependencies and
updates import paths to reference the transpiled temporary files.
dependabot Bot and others added 28 commits May 26, 2026 05:39
…js#5580)

Gherkin beforeEach/afterEach pass Mocha hook Context into setup/teardown,
but asyncWrapper read suite.ctx.currentTest (undefined on Context), so
event.test.before received a placeholder test (title "...", empty tags).

Also forward Mocha's done callback instead of no-op () => {}, so
event.test.before completes before scenario Background Before hooks run.

Co-authored-by: Daniil Krapiunitski <daniil.krapiunitski@emplifi.io>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ceptjs#5582)

Adds three layers of coverage for the bug fixed in 770749e (gherkin
beforeEach/afterEach emitting event.test.before with a placeholder test
when the hook Context lacked .ctx.currentTest):

* Unit (test/unit/bdd_test.js): six tests in a new "Gherkin hook events"
  block that drive the codeceptjs.before/after hooks with a mock Mocha
  Context and assert the real scenario title and tags reach
  event.test.before, event.test.after, the Before/After step-definition
  hooks, and that done is forwarded.
* Runner (test/runner/bdd_test.js + test/data/sandbox): a capture plugin
  and codecept.bdd.events.js config that serialize event payloads to a
  temp JSON file so a spawned runner asserts the real @important/@very
  tags reach listeners end-to-end.
* Acceptance (test/acceptance/gherkin/before_hook.feature + steps.js):
  a new Background+tagged feature whose Then step verifies both the BDD
  Before() hook and a direct event.test.before listener captured the
  real scenario.

Also flips setup/teardown to prefer suite.ctx.currentTest over
suite.currentTest, restoring the regular Scenario path while keeping the
BDD fallback.

Co-authored-by: DavertMik <davert@testomat.io>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…odeceptjs#5583)

* fix(appium): resolve "Unsupported helper type: unknown" in fillField

Appium extends WebDriver but WebElement._detectHelperType only checked
constructor.name === 'WebDriver', so Appium helpers fell through to
'unknown'. fillRichEditor (added in codeceptjs#5527) wraps elements in WebElement
and calls evaluate(), which then threw on every fillField/appendField
call from Appium.

- WebElement now walks the prototype chain so any subclass of
  WebDriver/Playwright/Puppeteer is detected correctly.
- WebDriver.fillField skips fillRichEditor when isWeb === false, since
  rich-editor detection needs a DOM that doesn't exist in Appium native.
- Appium constructor: appiumV2 defaulted via `|| true` (always truthy);
  the v1-deprecation banner then fired on every default-config user.
  Fixed both: explicit `appiumV2: false` is now honored, and the banner
  only prints when the user opts into v1.
- Added regression tests for prototype-chain detection.
- Enabled the Android Appium workflow on pull_request to 4.x so the fix
  can be verified end-to-end via Sauce Labs from the PR.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(appium): raise connectionRetryTimeout to 5min, enable test retries

The Sauce Labs Android emulator cold-start regularly exceeds
webdriverio's 2-minute connectionRetryTimeout default, producing
UND_ERR_HEADERS_TIMEOUT on `POST /wd/hub/session` and failing the
suite before any test runs. Bump the default for the Appium helper
to 5 minutes (mobile cloud grids are slow by nature).

Also re-enable Mocha's `this.retries(1)` in the Appium test suite —
already attempted but commented out — so a single transient Sauce
Labs hiccup no longer reds the whole job.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(appium): silence "Method is not implemented" log spam from isDisplayed

webdriverio v9 implements element.isDisplayed() by injecting a JS
visibility check via POST /execute/sync. Appium's native context has no
JS engine, so every call lands a "Method is not implemented" error in
the wdio logger before the existing _isDisplayedSafe catch swallows it.

Tests pass but logs are noisy with these false-positive errors during
seeElement, click, grabTextFrom, grabAttributeFrom, etc.

Override isDisplayed at the element level: short-circuit to true while
in native context so the failing request is never issued. Matches the
semantics _isDisplayedSafe already applies (treats "not implemented" as
displayed since we found the element). In web/webview context, defer
to the original implementation.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: DavertMik <davert@testomat.io>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…ceptjs#5596)

`codecept init` installs tsx and sets `require: ['tsx/cjs']`, but the
generated tsconfig.json still carried the legacy ts-node setup:

- a `"ts-node": { files: true }` block, though ts-node is never installed
  and is marked "not recommended" in loaderCheck.js
- `"module": "commonjs"`, wrong for an ESM ("type": "module") project

Generate a tsconfig that matches the tsx/ESM setup init already configures:
drop the ts-node block, use `module: ESNext` with `moduleResolution: bundler`
(so extensionless imports resolve as tsx/docs expect), and bump target/lib
to ES2022.

Co-authored-by: DavertMik <davert@testomat.io>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
codeceptjs#5437)

* fix: support Playwright 1.58+ output format in `codeceptjs info`

Playwright 1.58 changed the output format of `npx playwright install --dry-run`:
- Old format: "browser: chromium version 143.0.7499.4"
- New format: "Chrome for Testing 145.0.7632.6 (playwright chromium v1208)"

Updated the regex to handle both formats while excluding chromium-headless-shell.

Fixes codeceptjs#5422

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test: add unit tests for parsePlaywrightBrowsers regex

Extract parsePlaywrightBrowsers function and add unit tests to verify
both old (Playwright < 1.58) and new (1.58+) output formats are parsed
correctly, and that chromium-headless-shell is excluded.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
…ng (4.x) (codeceptjs#5438)

* fix: run-workers --by suite parallelization broken by test file sorting (codeceptjs#5412)

The sorting of test files in loadTests() (added in codeceptjs#5386) broke the
--by suite parallelization. When files were sorted before worker
distribution, all workers could receive the same tests instead of
different suites being distributed to different workers.

Fix: Move testFiles.sort() from loadTests() to run(). This ensures:
- Worker distribution uses original (unsorted) file order for consistent
  distribution across workers
- Test execution still uses alphabetical order (sorted in run())

Added unit test to verify files are not sorted after loadTests().

Fixes codeceptjs#5412

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: adapt tests for CI environment

- alphabetical_order_test: avoid calling codecept.run() which hangs
  in CI due to container.started() and event listener setup; test
  loadTests() directly instead
- worker_test: revert custom config assertions to original 4.x
  relaxed values (exact counts are filesystem-dependent)
- worker_test: simplify distribution test to only verify suite
  distribution without assuming glob order

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test: remove fragile worker distribution test

The test depends on mocha.loadFiles() with full container globals
(Feature, Scenario) which aren't available in unit test context on CI.
The core fix (sort moved from loadTests to run) is already verified
by alphabetical_order_test.js.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Revert "test: remove fragile worker distribution test"

This reverts commit 5c03e7a.

* test: fix worker distribution test for CI compatibility

Compare loadTests() output against fresh globSync() to verify files
are not sorted, instead of assuming a specific non-alphabetical order.
This works regardless of filesystem glob order (ext4 vs others).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* support windows

* fix path issues on test

* fix path issues on test

* fix path issues on test

* fix path issues on test

* fix path issues on test

* fix path issues on test

* resolve load config file"

* DRY

* DRY

* DRY

* DRY

* DRY

* fix: uts

* fix: uts

* fix: uts

* relax conditions

* relax conditions

* debug

* debug

* fix uts

* fix uts

* fix uts

* fix uts

* fix uts

* fix uts
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Mickael <mickael.desmousseaux@rapid4cloud.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…eptjs#5615)

* fix(init): missing file ext of step file when using typescript

* add more tests

* add proper verification
…odeceptjs#5621)

@codeceptjs/detox-helper was listed under optionalDependencies, which
npm/pnpm install by default. It hard-depends on detox and react-native,
so every codeceptjs install pulled in those heavy packages (and the
dtrace-provider build script) even though core never requires the Detox
helper — it is a user-configured external helper.

Move it to devDependencies (still needed for in-repo helper docs
generation). Users who need the Detox helper install it separately.

Co-authored-by: DavertMik <davert@testomat.io>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…s#5622)

Adds browser-free characterization tests that pin down the error and
settle-guarantees of the promise composition core, so it can be fixed or
refactored safely. No lib/ code is changed — current behavior (including
known-bad behavior) is frozen.

- recorder_test.js: errHandler/catch routing, catchWithoutStop terminal
  vs normal errors, ignoreErr, nested-session id semantics, unbalanced
  session restore, and task timeout (success + failure).
- session_composition_test.js (new): session()/within()/retryTo()/hopeThat()
  composition with a fake helper registered through the real container, a
  settles()/drain() harness that turns deadlocks into fast named failures,
  and hermetic per-test isolation of the recorder singleton.
- mocha/asyncWrapper_test.js: test() lifecycle (queued-step failure, sync
  throw, test.throws pass) and a rejecting injected() before-hook.

Characterized divergences (session-id leak on error, within skipping
_withinEnd, retryTo retrying past rejection, recorder.retries leaking across
runs, stopped-recorder hang) are documented for the follow-up fix plan.

Co-authored-by: DavertMik <davert@testomat.io>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ore (codeceptjs#5630)

Every pause() call registered two permanent listeners on the global event
dispatcher (step.after, test.finished) and never removed them. Repeated
pauses — now the normal case because the MCP server drives pause()
programmatically via setPauseHandler/pauseNow — accumulated listeners, fired
finish() multiple times, and ran an unconditional recorder.session.restore('pause')
on every test finish even when no pause session was open, unbalancing the
recorder's session stack (the hang class blocking 4.0).

- Convert the two anonymous listeners into named handlers (onStepAfter,
  onTestFinished) and register them through an idempotent helper that removes
  any prior registration first, so repeated pause()/pauseNow() keep exactly
  one of each. onTestFinished removes both listeners when the test finishes.
- Track an open-pause flag and only restore the 'pause' session when one is
  actually open (set on session.start in pauseSession, cleared at all three
  restore sites).
- pauseNow now performs the same idempotent registration as pause().

setPauseHandler/pauseNow signatures and resolve semantics are unchanged
(bin/mcp-server.js untouched). 4 regression tests cover idempotent
registration, listener removal on finish, the no-double-restore guard, and the
MCP pauseNow lifecycle; reverting the fix fails the listener tests.

Co-authored-by: DavertMik <davert@testomat.io>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…4.x (codeceptjs#5624, codeceptjs#5633) (codeceptjs#5637)

* fix(mocha): fail fast when test hook import chain rejects

In 4.x the per-test setup()/teardown() hooks load ./test.js via a dynamic
import() with no .catch(). If that import rejects, or the .then body throws
(e.g. enhanceMochaTest on an undefined test, a listener throwing), done() is
never called and the mocha hook hangs forever — the silent-hang failure mode
the 4.0 release is most concerned with.

This mirrors the existing suiteSetup()/suiteTeardown() shape exactly: append a
.catch(err => doneFn(err)) to both import chains. No recorder.errHandler is
added (the per-test handler owns the single errFn slot). makeDoneCallableOnce
already guards against a double done() call.

Adds 3 regression tests: setup() and teardown() with a throwing then-body call
done with the error within 1s instead of hanging, plus a happy-path check.
Reverting the fix makes the two error-path tests hang (verified).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit ac59a40)

* fix(core): restore sessions and propagate errors on session/within/retryTo error paths

Fixes four of the latent error-path divergences characterized in codeceptjs#5622, by
making the error paths symmetric with the success paths. The characterization
assertions are updated to the corrected behavior in the same commit.

- within() async error (lib/effects.js): the catch now calls finishHelpers()
  (so helpers' _withinEnd runs on error, not just success) and returns
  recorder.promise() (so the error propagates through within() instead of being
  detached onto a trailing task that a caller awaiting within() never sees).

- session() async error (lib/session.js): schedules a recorder task that runs
  recorder.session.restore so the recorder session is restored on error (it
  was only restored on success, leaking the session id). The existing
  restoreVars/listener cleanup is kept as-is — switching to finalize()'s real
  restoreVars() closes the browser context under BROWSER_RESTART=session.

- session() sync error (lib/session.js): the finally recorder.catch now calls
  recorder.session.restore before re-throwing (the only place that runs on a
  rejected chain).

- retryTo() (lib/effects.js): a thrown callback no longer reject()s the outer
  promise prematurely — it routes through recorder.throw so the retry logic
  owns the outcome (retry, or reject once maxTries is exhausted). A callback
  that throws then succeeds on a later attempt now resolves instead of
  rejecting. tries now starts at 1 on the first attempt (was 2); the retry
  count is preserved (tries < maxTries).

Verified: unit 748/0, runner 273/0 (incl. all retryFailedStep/rerun tests),
acceptance within/session/els green under both BROWSER_RESTART=browser and
=session.

Not changed here (would be unsafe or out of scope, see PR):
recorder.retries clearing (retryFailedStep depends on it), the stopped-recorder
no-op contract, and nested cross-level restore ordering.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit dca7626)

---------

Co-authored-by: DavertMik <davert@testomat.io>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ently ignored

sort() added in codeceptjs#5438 was unconditional — it ran after loadTests() applied
shuffle(), overwriting the randomised order every time. Guard the sort
with !this.opts.shuffle so alphabetical order is used for normal runs
and the shuffled order is preserved when --shuffle is requested.

Fixes codeceptjs#5605
@kapil971390

Copy link
Copy Markdown
Contributor Author

Closing — PR included unintended commits due to fork sync issue. Will reopen with a clean branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.