Skip to content

chore: refactor tests #2225

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tatomyr wants to merge 3 commits into from
Closed

chore: refactor tests #2225

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c57fe53
chore: refactor tests
tatomyr Jul 25, 2025
29175d7
fix path
tatomyr Jul 25, 2025
010399f
fix smoke path
tatomyr Jul 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
8 changes: 4 additions & 4 deletions .github/workflows/performance.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ jobs:
- run: redocly-next --version
- run: redocly --version
- name: Run Benchmark
run: hyperfine -i --warmup 3 'redocly lint packages/core/src/benchmark/benches/rebilly.yaml' 'redocly-next lint packages/core/src/benchmark/benches/rebilly.yaml' --export-markdown benchmark_check.md --export-json benchmark_check.json
run: hyperfine -i --warmup 3 'redocly lint resources/rebilly.yaml' 'redocly-next lint resources/rebilly.yaml' --export-markdown benchmark_check.md --export-json benchmark_check.json

- name: Comment PR
if: ${{ github.event.pull_request.head.repo.full_name == github.repository }}
Expand All @@ -59,12 +59,12 @@ jobs:
run: |
npm run compile
npm run pack:prepare
cd benchmark/
cd tests/performance/
npm i
npm run make-test
- name: Run Benchmark
run: |
cd benchmark/
cd tests/performance/
npm test # This command is generated and injected into package.json in the previous step.
cat benchmark_check.md
npm run chart # Creates benchmark_chart.md with the performance bar chart.
Expand All @@ -73,5 +73,5 @@ jobs:
if: ${{ github.event.pull_request.head.repo.full_name == github.repository }}
uses: thollander/actions-comment-pull-request@v2
with:
filePath: benchmark/benchmark_chart.md
filePath: tests/performance/benchmark_chart.md
comment_tag: historical-versions-comparison
20 changes: 10 additions & 10 deletions .github/workflows/smoke-plugins.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ jobs:
- name: Install dependencies
run: npm ci
- name: Prepare Smoke
run: bash ./__tests__/smoke-plugins/prepare-smoke-plugins.sh
run: bash ./tests/smoke/smoke-plugins/prepare-smoke-plugins.sh
- uses: actions/cache@v3
with:
path: __tests__/smoke-plugins/
path: tests/smoke/smoke-plugins/
key: cache-plugins-${{ github.run_id }}-${{ github.run_attempt }}

run-smoke-plugins--npm--node-22:
Expand All @@ -32,50 +32,50 @@ jobs:
steps:
- uses: actions/cache@v3
with:
path: __tests__/smoke-plugins/
path: tests/smoke/smoke-plugins/
key: cache-plugins-${{ github.run_id }}-${{ github.run_attempt }}
- uses: actions/setup-node@v3
with:
node-version: 22
- run: bash ./__tests__/smoke-plugins/run-smoke.sh
- run: bash ./tests/smoke/smoke-plugins/run-smoke.sh

run-smoke-plugins--npm--node-22--windows:
needs: prepare-smoke-plugins
runs-on: windows-latest
steps:
- uses: actions/cache@v3
with:
path: __tests__/smoke-plugins/
path: tests/smoke/smoke-plugins/
key: cache-plugins-${{ github.run_id }}-${{ github.run_attempt }}
enableCrossOsArchive: true
- uses: actions/setup-node@v3
with:
node-version: 22
- run: bash ./__tests__/smoke-plugins/run-smoke.sh
- run: bash ./tests/smoke/smoke-plugins/run-smoke.sh

run-smoke-plugins--npm--node-20:
needs: prepare-smoke-plugins
runs-on: ubuntu-latest
steps:
- uses: actions/cache@v3
with:
path: __tests__/smoke-plugins/
path: tests/smoke/smoke-plugins/
key: cache-plugins-${{ github.run_id }}-${{ github.run_attempt }}
- uses: actions/setup-node@v3
with:
node-version: 20
- run: bash ./__tests__/smoke-plugins/run-smoke.sh
- run: bash ./tests/smoke/smoke-plugins/run-smoke.sh

run-smoke-plugins--npm--node-20--windows:
needs: prepare-smoke-plugins
runs-on: windows-latest
steps:
- uses: actions/cache@v3
with:
path: __tests__/smoke-plugins/
path: tests/smoke/smoke-plugins/
key: cache-plugins-${{ github.run_id }}-${{ github.run_attempt }}
enableCrossOsArchive: true
- uses: actions/setup-node@v3
with:
node-version: 20
- run: bash ./__tests__/smoke-plugins/run-smoke.sh
- run: bash ./tests/smoke/smoke-plugins/run-smoke.sh
60 changes: 30 additions & 30 deletions .github/workflows/smoke.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ jobs:
- name: Install dependencies
run: npm ci
- name: Prepare Smoke
run: bash ./__tests__/smoke/prepare-smoke.sh
run: bash ./tests/smoke/smoke-general/prepare-smoke.sh
- uses: actions/cache@v3
with:
path: __tests__/smoke/
path: tests/smoke/smoke-general/
key: cache-${{ github.run_id }}-${{ github.run_attempt }}

run-smoke--npm--node-22:
Expand All @@ -32,59 +32,59 @@ jobs:
steps:
- uses: actions/cache@v3
with:
path: __tests__/smoke/
path: tests/smoke/smoke-general/
key: cache-${{ github.run_id }}-${{ github.run_attempt }}
- uses: actions/setup-node@v3
with:
node-version: 22
- run: bash ./__tests__/smoke/run-smoke.sh "npm i redocly-cli.tgz" "npm run"
- run: bash ./tests/smoke/smoke-general/run-smoke.sh "npm i redocly-cli.tgz" "npm run"

run-smoke--npm--node-22--redoc:
needs: prepare-smoke
runs-on: ubuntu-latest
steps:
- uses: actions/cache@v3
with:
path: __tests__/smoke/
path: tests/smoke/smoke-general/
key: cache-${{ github.run_id }}-${{ github.run_attempt }}
- uses: actions/setup-node@v3
with:
node-version: 22
- run: bash ./__tests__/smoke/run-smoke.sh "npm i redoc redocly-cli.tgz" "npm run"
- run: bash ./tests/smoke/smoke-general/run-smoke.sh "npm i redoc redocly-cli.tgz" "npm run"

run-smoke--npm--node-20:
needs: prepare-smoke
runs-on: ubuntu-latest
steps:
- uses: actions/cache@v3
with:
path: __tests__/smoke/
path: tests/smoke/smoke-general/
key: cache-${{ github.run_id }}-${{ github.run_attempt }}
- uses: actions/setup-node@v3
with:
node-version: 20
- run: bash ./__tests__/smoke/run-smoke.sh "npm i redocly-cli.tgz" "npm run"
- run: bash ./tests/smoke/smoke-general/run-smoke.sh "npm i redocly-cli.tgz" "npm run"

run-smoke--npm--node-20--redoc:
needs: prepare-smoke
runs-on: ubuntu-latest
steps:
- uses: actions/cache@v3
with:
path: __tests__/smoke/
path: tests/smoke/smoke-general/
key: cache-${{ github.run_id }}-${{ github.run_attempt }}
- uses: actions/setup-node@v3
with:
node-version: 20
- run: bash ./__tests__/smoke/run-smoke.sh "npm i redoc redocly-cli.tgz" "npm run"
- run: bash ./tests/smoke/smoke-general/run-smoke.sh "npm i redoc redocly-cli.tgz" "npm run"

run-smoke--yarn--node-22:
needs: prepare-smoke
runs-on: ubuntu-latest
steps:
- uses: actions/cache@v3
with:
path: __tests__/smoke/
path: tests/smoke/smoke-general/
key: cache-${{ github.run_id }}-${{ github.run_attempt }}
- uses: actions/setup-node@v3
with:
Expand All @@ -93,7 +93,7 @@ jobs:
run: yarn cache clean
- run: |
for i in {1..2}; do # workaround for yarn cache issue
sleep 5 && bash ./__tests__/smoke/run-smoke.sh "yarn add ./redocly-cli.tgz" "yarn" && break
sleep 5 && bash ./tests/smoke/smoke-general/run-smoke.sh "yarn add ./redocly-cli.tgz" "yarn" && break
done

run-smoke--yarn--node-22--redoc:
Expand All @@ -102,7 +102,7 @@ jobs:
steps:
- uses: actions/cache@v3
with:
path: __tests__/smoke/
path: tests/smoke/smoke-general/
key: cache-${{ github.run_id }}-${{ github.run_attempt }}
- uses: actions/setup-node@v3
with:
Expand All @@ -111,7 +111,7 @@ jobs:
run: yarn cache clean
- run: |
for i in {1..2}; do # workaround for yarn cache issue
sleep 5 && bash ./__tests__/smoke/run-smoke.sh "yarn add redoc ./redocly-cli.tgz" "yarn" && break
sleep 5 && bash ./tests/smoke/smoke-general/run-smoke.sh "yarn add redoc ./redocly-cli.tgz" "yarn" && break
done

run-smoke--yarn--node-20:
Expand All @@ -120,7 +120,7 @@ jobs:
steps:
- uses: actions/cache@v3
with:
path: __tests__/smoke/
path: tests/smoke/smoke-general/
key: cache-${{ github.run_id }}-${{ github.run_attempt }}
- uses: actions/setup-node@v3
with:
Expand All @@ -129,7 +129,7 @@ jobs:
run: yarn cache clean
- run: |
for i in {1..2}; do # workaround for yarn cache issue
sleep 5 && bash ./__tests__/smoke/run-smoke.sh "yarn add ./redocly-cli.tgz" "yarn" && break
sleep 5 && bash ./tests/smoke/smoke-general/run-smoke.sh "yarn add ./redocly-cli.tgz" "yarn" && break
done

run-smoke--yarn--node-20--redoc:
Expand All @@ -138,7 +138,7 @@ jobs:
steps:
- uses: actions/cache@v3
with:
path: __tests__/smoke/
path: tests/smoke/smoke-general/
key: cache-${{ github.run_id }}-${{ github.run_attempt }}
- uses: actions/setup-node@v3
with:
Expand All @@ -147,7 +147,7 @@ jobs:
run: yarn cache clean
- run: |
for i in {1..2}; do # workaround for yarn cache issue
sleep 5 && bash ./__tests__/smoke/run-smoke.sh "yarn add redoc ./redocly-cli.tgz" "yarn" && break
sleep 5 && bash ./tests/smoke/smoke-general/run-smoke.sh "yarn add redoc ./redocly-cli.tgz" "yarn" && break
done

run-smoke--pnpm--node-22:
Expand All @@ -156,7 +156,7 @@ jobs:
steps:
- uses: actions/cache@v3
with:
path: __tests__/smoke/
path: tests/smoke/smoke-general/
key: cache-${{ github.run_id }}-${{ github.run_attempt }}
- uses: actions/setup-node@v3
with:
Expand All @@ -165,29 +165,29 @@ jobs:
uses: pnpm/action-setup@v4
with:
version: 10
- run: bash ./__tests__/smoke/run-smoke.sh "pnpm add ./redocly-cli.tgz" "pnpm run"
- run: bash ./tests/smoke/smoke-general/run-smoke.sh "pnpm add ./redocly-cli.tgz" "pnpm run"

run-smoke--npm--node-22--windows:
needs: prepare-smoke
runs-on: windows-latest
steps:
- uses: actions/cache@v3
with:
path: __tests__/smoke/
path: tests/smoke/smoke-general/
key: cache-${{ github.run_id }}-${{ github.run_attempt }}
enableCrossOsArchive: true
- uses: actions/setup-node@v3
with:
node-version: 22
- run: bash ./__tests__/smoke/run-smoke.sh "npm i redocly-cli.tgz" "npm run"
- run: bash ./tests/smoke/smoke-general/run-smoke.sh "npm i redocly-cli.tgz" "npm run"

run-smoke--yarn--node-22--windows:
needs: prepare-smoke
runs-on: windows-latest
steps:
- uses: actions/cache@v3
with:
path: __tests__/smoke/
path: tests/smoke/smoke-general/
key: cache-${{ github.run_id }}-${{ github.run_attempt }}
enableCrossOsArchive: true
- uses: actions/setup-node@v3
Expand All @@ -198,7 +198,7 @@ jobs:
Start-Sleep -Seconds 20 # workaround for yarn cache issue
for ($i = 1; $i -le 3; $i++) {
try {
bash ./__tests__/smoke/run-smoke.sh "yarn add ./redocly-cli.tgz --no-lockfile --network-timeout 100000" "yarn"
bash ./tests/smoke/smoke-general/run-smoke.sh "yarn add ./redocly-cli.tgz --no-lockfile --network-timeout 100000" "yarn"
if ($LASTEXITCODE -eq 0) { break }
} catch {
if ($i -eq 3) { throw }
Expand All @@ -212,21 +212,21 @@ jobs:
steps:
- uses: actions/cache@v3
with:
path: __tests__/smoke/
path: tests/smoke/smoke-general/
key: cache-${{ github.run_id }}-${{ github.run_attempt }}
enableCrossOsArchive: true
- uses: actions/setup-node@v3
with:
node-version: 20
- run: bash ./__tests__/smoke/run-smoke.sh "npm i redocly-cli.tgz" "npm run"
- run: bash ./tests/smoke/smoke-general/run-smoke.sh "npm i redocly-cli.tgz" "npm run"

run-smoke--yarn--node-20--windows:
needs: prepare-smoke
runs-on: windows-latest
steps:
- uses: actions/cache@v3
with:
path: __tests__/smoke/
path: tests/smoke/smoke-general/
key: cache-${{ github.run_id }}-${{ github.run_attempt }}
enableCrossOsArchive: true
- uses: actions/setup-node@v3
Expand All @@ -237,7 +237,7 @@ jobs:
Start-Sleep -Seconds 20 # workaround for yarn cache issue
for ($i = 1; $i -le 3; $i++) {
try {
bash ./__tests__/smoke/run-smoke.sh "yarn add ./redocly-cli.tgz --no-lockfile --network-timeout 100000" "yarn"
bash ./tests/smoke/smoke-general/run-smoke.sh "yarn add ./redocly-cli.tgz --no-lockfile --network-timeout 100000" "yarn"
if ($LASTEXITCODE -eq 0) { break }
} catch {
if ($i -eq 3) { throw }
Expand All @@ -254,8 +254,8 @@ jobs:
- name: Run docker image
run: |
# Prepare
cp resources/pets.yaml resources/museum.yaml resources/museum-tickets.arazzo.yaml __tests__/smoke/
cd __tests__/smoke/
cp resources/pets.yaml resources/museum.yaml resources/museum-tickets.arazzo.yaml tests/smoke/smoke-general/
cd tests/smoke/smoke-general/

# Run commands
docker run --rm redocly/cli:latest --version
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ jobs:
- name: Redocly version
run: redocly --version
- name: Definition test
run: redocly lint packages/core/src/benchmark/benches/rebilly.yaml --format=stylish
run: redocly lint resources/rebilly.yaml --format=stylish

coverage-report:
runs-on: ubuntu-latest
Expand Down
3 changes: 1 addition & 2 deletions .prettierignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ lib/
output/
packages/core/src/rules/common/__tests__/fixtures/invalid-yaml.yaml
packages/respect-core/src/modules/runtime-expressions/abnf-parser.cjs
benchmark/api-definitions/
tests/performance/api-definitions/
LICENSE.md
snapshot*.txt
.changeset/pre.json
12 changes: 5 additions & 7 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,21 +256,21 @@ npm run pack:prepare
npm i -g redocly-cli.tgz

# Re-build the docs
(cd __tests__/smoke/ && redocly build-docs openapi.yaml -o pre-built/redoc.html)
(cd tests/smoke/smoke-general/ && redocly build-docs openapi.yaml -o pre-built/redoc.html)
```

For other commands you'd have to do something similar.

### Performance benchmark

To run the performance benchmark locally, you should have `hyperfine` (v1.16.1+) installed on your machine.
Prepare the local build, go to the `benchmark` folder, clean it up, do the preparations, and run the actual benchmark:
Prepare the local build, go to the `tests/performance/` folder, clean it up, do the preparations, and run the actual benchmark:

```sh
(npm run compile && npm run pack:prepare && cd benchmark/ && git clean -dX -f . && git clean -dX -ff . && npm i && npm run make-test && npm test)
(npm run compile && npm run pack:prepare && cd tests/performance/ && git clean -dX -f . && git clean -dX -ff . && npm i && npm run make-test && npm test)
```

You might need to adjust the CLI versions that need to be tested in the `benchmark/package.json` file.
You might need to adjust the CLI versions that need to be tested in the `tests/performance/package.json` file.

### Manual tests

Expand All @@ -281,9 +281,7 @@ What should be verified when changes applied to the `respect-core` package:

## Project structure

- **`__tests__`**: contains e2e and smoke tests.

- **`benchmark`**: contains the performance benchmark. <!-- TODO: move it under the __tests__ folder -->
- **`tests`**: contains e2e, performance, and smoke tests.

- **`docs`**: contains the documentation source files. When changes to the documentation are merged, they automatically get published on the [Redocly docs website](https://redocly.com/docs/cli/).

Expand Down
Loading
Loading