fix: guard sort() with shuffle check so --shuffle is not silently ignored#5638
Closed
kapil971390 wants to merge 1856 commits into
Closed
fix: guard sort() with shuffle check so --shuffle is not silently ignored#5638kapil971390 wants to merge 1856 commits into
kapil971390 wants to merge 1856 commits into
Conversation
* implemented aria selectors for PW/WebDriver/Puppeteer * added aria elements * Implemented by role selector and aria locators with tests * fixed aria selectors for WebDriverIO * added tests, reverted runok * 4.0.0-beta.1 * fixed aria tests * fixed WD tests * improved output * fixed webdriver types --------- Co-authored-by: DavertMik <davert@testomat.io>
Dependabot couldn't find the original pull request head commit, 25a49b7. Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…(4.x) (codeceptjs#5191) * fix(utils): remove incorrect `async` from `emptyFolder` * fix(utils): resolve command injection vulnerability in `emptyFolder` --------- Co-authored-by: kobenguyent <7845001+kobenguyent@users.noreply.github.com>
Bumps [tsd](https://github.com/tsdjs/tsd) from 0.32.0 to 0.33.0. - [Release notes](https://github.com/tsdjs/tsd/releases) - [Commits](tsdjs/tsd@v0.32.0...v0.33.0) --- updated-dependencies: - dependency-name: tsd dependency-version: 0.33.0 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [envinfo](https://github.com/tabrindle/envinfo) from 7.14.0 to 7.19.0. - [Release notes](https://github.com/tabrindle/envinfo/releases) - [Changelog](https://github.com/tabrindle/envinfo/blob/main/CHANGELOG.md) - [Commits](tabrindle/envinfo@v7.14.0...v7.19.0) --- updated-dependencies: - dependency-name: envinfo dependency-version: 7.19.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 4 to 6. - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@v4...v6) --- updated-dependencies: - dependency-name: actions/setup-node dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps-dev): bump typescript from 5.8.3 to 5.9.3 Bumps [typescript](https://github.com/microsoft/TypeScript) from 5.8.3 to 5.9.3. - [Release notes](https://github.com/microsoft/TypeScript/releases) - [Changelog](https://github.com/microsoft/TypeScript/blob/main/azure-pipelines.release-publish.yml) - [Commits](microsoft/TypeScript@v5.8.3...v5.9.3) --- updated-dependencies: - dependency-name: typescript dependency-version: 5.9.3 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Update typedoc version to ^0.28.14 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: kobenguyent <7845001+kobenguyent@users.noreply.github.com>
* refactor: migrate Appium helper to ESM - Convert require() to import statements - Change webdriverio from dynamic require to static import - Import webdriverio as namespace (import * as webdriverio) - Import all dependencies at the top of the file: - ElementNotFound from './errors/ElementNotFound.js' - dontSeeElementError from './errors/ElementAssertion.js' - Remove inline require statements from methods: - dontSeeElement() - seeElement() - waitForVisible() - waitForInvisible() - Use Locator.build() instead of new Locator() for ESM compatibility - Maintain export default Appium at the end - All imports use .js extensions for ESM compliance Verified: - Module loads successfully - No require() statements remaining - Tests run without errors * enable appium tests * fix: initialize chai.should() in Appium tests The test was failing with 'Cannot read properties of undefined (reading "be")' because chai's should assertion style was not initialized. Added chai.should() call after imports to enable should-style assertions throughout the test file. Fixes test: device lock : #seeDeviceIsLocked, #seeDeviceIsUnlocked
* Fix: preserve global timeout with BeforeSuite hook * test: clarify feature name for global timeout with BeforeSuite * chore: remove unused plugins field from beforeSuiteTimeout config
Bumps [ts-morph](https://github.com/dsherret/ts-morph) from 26.0.0 to 27.0.2. - [Release notes](https://github.com/dsherret/ts-morph/releases) - [Commits](dsherret/ts-morph@26.0.0...27.0.2) --- updated-dependencies: - dependency-name: ts-morph dependency-version: 27.0.2 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [semver](https://github.com/npm/node-semver) from 7.7.2 to 7.7.3. - [Release notes](https://github.com/npm/node-semver/releases) - [Changelog](https://github.com/npm/node-semver/blob/main/CHANGELOG.md) - [Commits](npm/node-semver@v7.7.2...v7.7.3) --- updated-dependencies: - dependency-name: semver dependency-version: 7.7.3 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 5 to 6. - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@v5...v6) --- updated-dependencies: - dependency-name: actions/setup-node dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
- Added transpilation of .ts files using TypeScript compiler - Creates temporary .mjs file to load transpiled code - Cleans up temporary files after loading - Fixes 'Unknown file extension .ts' error when using steps_file.ts
- Recursively transpile TypeScript files and their dependencies - Handle .js imports that reference .ts source files - Replace import paths in transpiled code to point to temp .mjs files - Clean up all temporary files on completion or error - Add comprehensive tests for TypeScript support with cross-file imports Fixes issue where steps_file.ts importing other .ts files would fail with 'Cannot find module' error. Now properly transpiles all dependencies and updates import paths to reference the transpiled temporary files.
…js#5580) Gherkin beforeEach/afterEach pass Mocha hook Context into setup/teardown, but asyncWrapper read suite.ctx.currentTest (undefined on Context), so event.test.before received a placeholder test (title "...", empty tags). Also forward Mocha's done callback instead of no-op () => {}, so event.test.before completes before scenario Background Before hooks run. Co-authored-by: Daniil Krapiunitski <daniil.krapiunitski@emplifi.io> Co-authored-by: Cursor <cursoragent@cursor.com>
…ceptjs#5582) Adds three layers of coverage for the bug fixed in 770749e (gherkin beforeEach/afterEach emitting event.test.before with a placeholder test when the hook Context lacked .ctx.currentTest): * Unit (test/unit/bdd_test.js): six tests in a new "Gherkin hook events" block that drive the codeceptjs.before/after hooks with a mock Mocha Context and assert the real scenario title and tags reach event.test.before, event.test.after, the Before/After step-definition hooks, and that done is forwarded. * Runner (test/runner/bdd_test.js + test/data/sandbox): a capture plugin and codecept.bdd.events.js config that serialize event payloads to a temp JSON file so a spawned runner asserts the real @important/@very tags reach listeners end-to-end. * Acceptance (test/acceptance/gherkin/before_hook.feature + steps.js): a new Background+tagged feature whose Then step verifies both the BDD Before() hook and a direct event.test.before listener captured the real scenario. Also flips setup/teardown to prefer suite.ctx.currentTest over suite.currentTest, restoring the regular Scenario path while keeping the BDD fallback. Co-authored-by: DavertMik <davert@testomat.io> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…odeceptjs#5583) * fix(appium): resolve "Unsupported helper type: unknown" in fillField Appium extends WebDriver but WebElement._detectHelperType only checked constructor.name === 'WebDriver', so Appium helpers fell through to 'unknown'. fillRichEditor (added in codeceptjs#5527) wraps elements in WebElement and calls evaluate(), which then threw on every fillField/appendField call from Appium. - WebElement now walks the prototype chain so any subclass of WebDriver/Playwright/Puppeteer is detected correctly. - WebDriver.fillField skips fillRichEditor when isWeb === false, since rich-editor detection needs a DOM that doesn't exist in Appium native. - Appium constructor: appiumV2 defaulted via `|| true` (always truthy); the v1-deprecation banner then fired on every default-config user. Fixed both: explicit `appiumV2: false` is now honored, and the banner only prints when the user opts into v1. - Added regression tests for prototype-chain detection. - Enabled the Android Appium workflow on pull_request to 4.x so the fix can be verified end-to-end via Sauce Labs from the PR. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(appium): raise connectionRetryTimeout to 5min, enable test retries The Sauce Labs Android emulator cold-start regularly exceeds webdriverio's 2-minute connectionRetryTimeout default, producing UND_ERR_HEADERS_TIMEOUT on `POST /wd/hub/session` and failing the suite before any test runs. Bump the default for the Appium helper to 5 minutes (mobile cloud grids are slow by nature). Also re-enable Mocha's `this.retries(1)` in the Appium test suite — already attempted but commented out — so a single transient Sauce Labs hiccup no longer reds the whole job. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(appium): silence "Method is not implemented" log spam from isDisplayed webdriverio v9 implements element.isDisplayed() by injecting a JS visibility check via POST /execute/sync. Appium's native context has no JS engine, so every call lands a "Method is not implemented" error in the wdio logger before the existing _isDisplayedSafe catch swallows it. Tests pass but logs are noisy with these false-positive errors during seeElement, click, grabTextFrom, grabAttributeFrom, etc. Override isDisplayed at the element level: short-circuit to true while in native context so the failing request is never issued. Matches the semantics _isDisplayedSafe already applies (treats "not implemented" as displayed since we found the element). In web/webview context, defer to the original implementation. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: DavertMik <davert@testomat.io> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
…ceptjs#5596) `codecept init` installs tsx and sets `require: ['tsx/cjs']`, but the generated tsconfig.json still carried the legacy ts-node setup: - a `"ts-node": { files: true }` block, though ts-node is never installed and is marked "not recommended" in loaderCheck.js - `"module": "commonjs"`, wrong for an ESM ("type": "module") project Generate a tsconfig that matches the tsx/ESM setup init already configures: drop the ts-node block, use `module: ESNext` with `moduleResolution: bundler` (so extensionless imports resolve as tsx/docs expect), and bump target/lib to ES2022. Co-authored-by: DavertMik <davert@testomat.io> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
codeceptjs#5437) * fix: support Playwright 1.58+ output format in `codeceptjs info` Playwright 1.58 changed the output format of `npx playwright install --dry-run`: - Old format: "browser: chromium version 143.0.7499.4" - New format: "Chrome for Testing 145.0.7632.6 (playwright chromium v1208)" Updated the regex to handle both formats while excluding chromium-headless-shell. Fixes codeceptjs#5422 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: add unit tests for parsePlaywrightBrowsers regex Extract parsePlaywrightBrowsers function and add unit tests to verify both old (Playwright < 1.58) and new (1.58+) output formats are parsed correctly, and that chromium-headless-shell is excluded. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
…ng (4.x) (codeceptjs#5438) * fix: run-workers --by suite parallelization broken by test file sorting (codeceptjs#5412) The sorting of test files in loadTests() (added in codeceptjs#5386) broke the --by suite parallelization. When files were sorted before worker distribution, all workers could receive the same tests instead of different suites being distributed to different workers. Fix: Move testFiles.sort() from loadTests() to run(). This ensures: - Worker distribution uses original (unsorted) file order for consistent distribution across workers - Test execution still uses alphabetical order (sorted in run()) Added unit test to verify files are not sorted after loadTests(). Fixes codeceptjs#5412 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: adapt tests for CI environment - alphabetical_order_test: avoid calling codecept.run() which hangs in CI due to container.started() and event listener setup; test loadTests() directly instead - worker_test: revert custom config assertions to original 4.x relaxed values (exact counts are filesystem-dependent) - worker_test: simplify distribution test to only verify suite distribution without assuming glob order Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: remove fragile worker distribution test The test depends on mocha.loadFiles() with full container globals (Feature, Scenario) which aren't available in unit test context on CI. The core fix (sort moved from loadTests to run) is already verified by alphabetical_order_test.js. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Revert "test: remove fragile worker distribution test" This reverts commit 5c03e7a. * test: fix worker distribution test for CI compatibility Compare loadTests() output against fresh globSync() to verify files are not sorted, instead of assuming a specific non-alphabetical order. This works regardless of filesystem glob order (ext4 vs others). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* support windows * fix path issues on test * fix path issues on test * fix path issues on test * fix path issues on test * fix path issues on test * fix path issues on test * resolve load config file" * DRY * DRY * DRY * DRY * DRY * fix: uts * fix: uts * fix: uts * relax conditions * relax conditions * debug * debug * fix uts * fix uts * fix uts * fix uts * fix uts * fix uts
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Mickael <mickael.desmousseaux@rapid4cloud.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…eptjs#5615) * fix(init): missing file ext of step file when using typescript * add more tests * add proper verification
…odeceptjs#5621) @codeceptjs/detox-helper was listed under optionalDependencies, which npm/pnpm install by default. It hard-depends on detox and react-native, so every codeceptjs install pulled in those heavy packages (and the dtrace-provider build script) even though core never requires the Detox helper — it is a user-configured external helper. Move it to devDependencies (still needed for in-repo helper docs generation). Users who need the Detox helper install it separately. Co-authored-by: DavertMik <davert@testomat.io> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…s#5622) Adds browser-free characterization tests that pin down the error and settle-guarantees of the promise composition core, so it can be fixed or refactored safely. No lib/ code is changed — current behavior (including known-bad behavior) is frozen. - recorder_test.js: errHandler/catch routing, catchWithoutStop terminal vs normal errors, ignoreErr, nested-session id semantics, unbalanced session restore, and task timeout (success + failure). - session_composition_test.js (new): session()/within()/retryTo()/hopeThat() composition with a fake helper registered through the real container, a settles()/drain() harness that turns deadlocks into fast named failures, and hermetic per-test isolation of the recorder singleton. - mocha/asyncWrapper_test.js: test() lifecycle (queued-step failure, sync throw, test.throws pass) and a rejecting injected() before-hook. Characterized divergences (session-id leak on error, within skipping _withinEnd, retryTo retrying past rejection, recorder.retries leaking across runs, stopped-recorder hang) are documented for the follow-up fix plan. Co-authored-by: DavertMik <davert@testomat.io> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ore (codeceptjs#5630) Every pause() call registered two permanent listeners on the global event dispatcher (step.after, test.finished) and never removed them. Repeated pauses — now the normal case because the MCP server drives pause() programmatically via setPauseHandler/pauseNow — accumulated listeners, fired finish() multiple times, and ran an unconditional recorder.session.restore('pause') on every test finish even when no pause session was open, unbalancing the recorder's session stack (the hang class blocking 4.0). - Convert the two anonymous listeners into named handlers (onStepAfter, onTestFinished) and register them through an idempotent helper that removes any prior registration first, so repeated pause()/pauseNow() keep exactly one of each. onTestFinished removes both listeners when the test finishes. - Track an open-pause flag and only restore the 'pause' session when one is actually open (set on session.start in pauseSession, cleared at all three restore sites). - pauseNow now performs the same idempotent registration as pause(). setPauseHandler/pauseNow signatures and resolve semantics are unchanged (bin/mcp-server.js untouched). 4 regression tests cover idempotent registration, listener removal on finish, the no-double-restore guard, and the MCP pauseNow lifecycle; reverting the fix fails the listener tests. Co-authored-by: DavertMik <davert@testomat.io> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…4.x (codeceptjs#5624, codeceptjs#5633) (codeceptjs#5637) * fix(mocha): fail fast when test hook import chain rejects In 4.x the per-test setup()/teardown() hooks load ./test.js via a dynamic import() with no .catch(). If that import rejects, or the .then body throws (e.g. enhanceMochaTest on an undefined test, a listener throwing), done() is never called and the mocha hook hangs forever — the silent-hang failure mode the 4.0 release is most concerned with. This mirrors the existing suiteSetup()/suiteTeardown() shape exactly: append a .catch(err => doneFn(err)) to both import chains. No recorder.errHandler is added (the per-test handler owns the single errFn slot). makeDoneCallableOnce already guards against a double done() call. Adds 3 regression tests: setup() and teardown() with a throwing then-body call done with the error within 1s instead of hanging, plus a happy-path check. Reverting the fix makes the two error-path tests hang (verified). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit ac59a40) * fix(core): restore sessions and propagate errors on session/within/retryTo error paths Fixes four of the latent error-path divergences characterized in codeceptjs#5622, by making the error paths symmetric with the success paths. The characterization assertions are updated to the corrected behavior in the same commit. - within() async error (lib/effects.js): the catch now calls finishHelpers() (so helpers' _withinEnd runs on error, not just success) and returns recorder.promise() (so the error propagates through within() instead of being detached onto a trailing task that a caller awaiting within() never sees). - session() async error (lib/session.js): schedules a recorder task that runs recorder.session.restore so the recorder session is restored on error (it was only restored on success, leaking the session id). The existing restoreVars/listener cleanup is kept as-is — switching to finalize()'s real restoreVars() closes the browser context under BROWSER_RESTART=session. - session() sync error (lib/session.js): the finally recorder.catch now calls recorder.session.restore before re-throwing (the only place that runs on a rejected chain). - retryTo() (lib/effects.js): a thrown callback no longer reject()s the outer promise prematurely — it routes through recorder.throw so the retry logic owns the outcome (retry, or reject once maxTries is exhausted). A callback that throws then succeeds on a later attempt now resolves instead of rejecting. tries now starts at 1 on the first attempt (was 2); the retry count is preserved (tries < maxTries). Verified: unit 748/0, runner 273/0 (incl. all retryFailedStep/rerun tests), acceptance within/session/els green under both BROWSER_RESTART=browser and =session. Not changed here (would be unsafe or out of scope, see PR): recorder.retries clearing (retryFailedStep depends on it), the stopped-recorder no-op contract, and nested cross-level restore ordering. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit dca7626) --------- Co-authored-by: DavertMik <davert@testomat.io> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ently ignored sort() added in codeceptjs#5438 was unconditional — it ran after loadTests() applied shuffle(), overwriting the randomised order every time. Guard the sort with !this.opts.shuffle so alphabetical order is used for normal runs and the shuffled order is preserved when --shuffle is requested. Fixes codeceptjs#5605
Contributor
Author
|
Closing — PR included unintended commits due to fork sync issue. Will reopen with a clean branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5605
What
run()callsthis.testFiles.sort()unconditionally afterloadTests()has already appliedshuffle()when--shuffleis set. The sort overwrites the randomised order every time, making--shufflesilently have no effect.Fix
Guard the sort with
!this.opts.shuffle:Normal runs continue to get alphabetical order.
--shuffleruns now keep their randomised order.Why this happened
The
sort()was introduced in #5438 to fix worker suite distribution — a correct fix for that issue, but it introduced this regression for--shuffleusers.Change
Single guard condition added to
lib/codecept.js— no other behaviour changed.