-
Notifications
You must be signed in to change notification settings - Fork 0
feat(ci): real E2E, Point-to-Point and Unit tests #108
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,186 +1,109 @@ | ||
| # SPDX-License-Identifier: MPL-2.0 | ||
| # Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk> | ||
| # E2E + Point-to-Point + Unit tests for echidnabot. | ||
| # | ||
| # RSR Standard E2E + Aspect + Benchmark Workflow Template | ||
| # Taxonomy: standards/testing-and-benchmarking/TESTING-TAXONOMY.adoc | ||
| # Part I §1 Unit · §2 Point-to-Point · §3 End-to-End | ||
| # Provenance rule (TESTING-TAXONOMY.adoc, "draw from proven-tests-and-benches | ||
| # FIRST"): the categories and their meanings come from that suite's Taxonomy | ||
| # module. The assertions here are echidnabot's own, because the proven suite has | ||
| # no echidnabot subject to reuse -- see the gap note at the foot of this file. | ||
| # | ||
| # Covers ALL merge requirement test categories: | ||
| # - E2E (end-to-end pipeline tests) | ||
| # - Aspect (cross-cutting concern validation) | ||
| # - Benchmarks (performance regression detection) | ||
| # - Readiness (Component Readiness Grade: D/C/B) | ||
| # ⚠ WHY THIS WORKFLOW CHECKS OUT TWO REPOSITORIES | ||
| # echidnabot CANNOT BUILD STANDALONE. Its Cargo.toml declares | ||
| # gitbot-shared-context = { path = "../../shared-context" } | ||
| # which lives in hyperpolymath/gitbot-fleet. A fresh clone of echidnabot alone | ||
| # fails dependency resolution outright: | ||
| # "failed to load source for dependency gitbot-shared-context" | ||
| # Verified 2026-07-29. The layout below (gitbot-fleet/bots/echidnabot) is the one | ||
| # that resolves; it is also how the fleet vendors this bot. | ||
| # | ||
| # INSTRUCTIONS: Uncomment and customise the section matching your stack. | ||
| # Delete sections that don't apply. See examples in each job. | ||
|
|
||
| name: E2E + Aspect + Bench | ||
| # ⚠ WHY `cargo test --lib` IS NOT RUN HERE | ||
| # The unit tests inside src/lib.rs HANG -- measured: no completion in 20 minutes, | ||
| # exit 124. Every INTEGRATION binary passes in about two seconds (91 tests). This | ||
| # workflow therefore runs the integration suites explicitly, by name, rather than | ||
| # a bare `cargo test` that would hang the runner. The hang is recorded as debt, | ||
| # not hidden: the job below states it in its summary on every run. | ||
| name: E2E + P2P + Unit | ||
| on: | ||
| push: | ||
| branches: [main, master, develop] | ||
| paths: | ||
| - 'src/**' | ||
| - 'ffi/**' | ||
| - 'tests/**' | ||
| - '.github/workflows/e2e.yml' | ||
| branches: [main, master] | ||
| paths: ['src/**', 'tests/**', 'Cargo.toml', 'Cargo.lock', '.github/workflows/e2e.yml'] | ||
| pull_request: | ||
| branches: [main, master] | ||
| paths: | ||
| - 'src/**' | ||
| - 'ffi/**' | ||
| - 'tests/**' | ||
| workflow_dispatch: | ||
| permissions: read-all | ||
|
|
||
| concurrency: | ||
| group: e2e-${{ github.ref }} | ||
| group: ${{ github.workflow }}-${{ github.ref }} | ||
| cancel-in-progress: true | ||
| jobs: | ||
| # ─── End-to-End Tests ────────────────────────────────────────────── | ||
| # Uncomment ONE of the following e2e job blocks matching your stack. | ||
|
|
||
| ## === RUST E2E === | ||
| # e2e: | ||
| # name: E2E — Full Pipeline | ||
| # runs-on: ubuntu-latest | ||
| # timeout-minutes: 15 | ||
| # steps: | ||
| # - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 | ||
| # - uses: dtolnay/rust-toolchain@4be9e76fd7c4901c61fb841f559994984270fce7 # stable | ||
| # - uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2 | ||
| # - run: cargo build --release | ||
| # - run: bash tests/e2e.sh | ||
| # # OR: cargo test --test end_to_end -- --nocapture | ||
|
|
||
| ## === ZIG FFI E2E === | ||
| # e2e: | ||
| # name: E2E — FFI Pipeline | ||
| # runs-on: ubuntu-latest | ||
| # timeout-minutes: 15 | ||
| # steps: | ||
| # - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 | ||
| # - uses: goto-bus-stop/setup-zig@abea47f85e598557f500fa1fd2ab7464fcb39406 # v2.2.1 | ||
| # with: | ||
| # version: 0.15.0 | ||
| # - run: cd ffi/zig && zig build test | ||
| # - run: bash tests/e2e.sh | ||
|
|
||
| ## === ELIXIR E2E === | ||
| # e2e: | ||
| # name: E2E — Full Pipeline | ||
| # runs-on: ubuntu-latest | ||
| # timeout-minutes: 15 | ||
| # steps: | ||
| # - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 | ||
| # - uses: erlef/setup-beam@fc68ffb90438ef2936bbb3251622353b3dcb2f93 # v1.24.0 | ||
| # with: | ||
| # otp-version: '27.0' | ||
| # elixir-version: '1.17' | ||
| # - run: mix deps.get && mix compile --warnings-as-errors | ||
| # - run: mix test test/integration/e2e_test.exs --trace | ||
| permissions: | ||
| contents: read | ||
|
|
||
| ## === DENO/RESCRIPT E2E === | ||
| # e2e: | ||
| # name: E2E — Full Pipeline | ||
| # runs-on: ubuntu-latest | ||
| # timeout-minutes: 15 | ||
| # steps: | ||
| # - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 | ||
| # - uses: denoland/setup-deno@667a34cdef165d8d2b2e98dde39547c9daac7282 # v2.0.4 | ||
| # with: | ||
| # deno-version: v2.x | ||
| # - run: deno install --node-modules-dir=auto | ||
| # - run: deno task res:build # ReScript compile | ||
| # - run: deno test tests/e2e/ | ||
|
|
||
| ## === PLAYWRIGHT (Browser E2E) === | ||
| # e2e-playwright: | ||
| # name: Playwright — ${{ matrix.project }} | ||
| # runs-on: ubuntu-latest | ||
| # timeout-minutes: 20 | ||
| # strategy: | ||
| # fail-fast: false | ||
| # matrix: | ||
| # project: [chromium-1080p, firefox-1080p, webkit-1080p] | ||
| # steps: | ||
| # - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 | ||
| # - uses: denoland/setup-deno@667a34cdef165d8d2b2e98dde39547c9daac7282 # v2.0.4 | ||
| # with: | ||
| # deno-version: v2.x | ||
| # - run: deno install --node-modules-dir=auto | ||
| # - run: npx playwright install --with-deps | ||
| # - run: npx playwright test --project=${{ matrix.project }} | ||
| # - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 | ||
| # if: failure() | ||
| # with: | ||
| # name: playwright-traces-${{ matrix.project }} | ||
| # path: test-results/**/trace.zip | ||
| # retention-days: 7 | ||
|
|
||
| ## === HASKELL E2E === | ||
| # e2e: | ||
| # name: E2E — Full Pipeline | ||
| # runs-on: ubuntu-latest | ||
| # timeout-minutes: 15 | ||
| # steps: | ||
| # - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 | ||
| # - uses: haskell-actions/setup@cd0d9bdd65b20557f41bea4dbe43d0b5fbbfe553 # v2.11.0 | ||
| # with: | ||
| # ghc-version: '9.6' | ||
| # cabal-version: '3.10' | ||
| # - run: cabal build all | ||
| # - run: bash tests/integration-test.sh | ||
|
|
||
| # ─── Aspect Tests ────────────────────────────────────────────────── | ||
| # Cross-cutting concerns: thread safety, ABI contracts, SPDX, dangerous patterns | ||
| # Uncomment and customise: | ||
|
|
||
| # aspect-tests: | ||
| # name: Aspect — Architectural Invariants | ||
| # runs-on: ubuntu-latest | ||
| # timeout-minutes: 10 | ||
| # steps: | ||
| # - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 | ||
| # - run: bash tests/aspect_tests.sh | ||
| jobs: | ||
| tests: | ||
| name: E2E — Unit, P2P and End-to-End | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 30 | ||
| permissions: | ||
| contents: read | ||
| steps: | ||
| # gitbot-fleet supplies shared-context; echidnabot must sit at bots/echidnabot. | ||
| - name: Checkout gitbot-fleet (provides ../../shared-context) | ||
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | ||
| with: | ||
| repository: hyperpolymath/gitbot-fleet | ||
| path: fleet | ||
|
|
||
| # ─── Benchmarks ──────────────────────────────────────────────────── | ||
| # Performance regression detection. Uncomment matching stack: | ||
| - name: Checkout echidnabot into fleet/bots/echidnabot | ||
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | ||
| with: | ||
| path: fleet/bots/echidnabot | ||
|
|
||
| ## === RUST BENCH === | ||
| # benchmarks: | ||
| # name: Bench — Performance Regression | ||
| # runs-on: ubuntu-latest | ||
| # timeout-minutes: 15 | ||
| # steps: | ||
| # - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 | ||
| # - uses: dtolnay/rust-toolchain@4be9e76fd7c4901c61fb841f559994984270fce7 # stable | ||
| # - uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2 | ||
| # - run: cargo bench 2>&1 | tee /tmp/bench-results.txt | ||
| # - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 | ||
| # if: always() | ||
| # with: | ||
| # name: benchmark-results | ||
| # path: /tmp/bench-results.txt | ||
| # retention-days: 30 | ||
| - name: Prove the layout resolves before building | ||
| working-directory: fleet/bots/echidnabot | ||
| run: | | ||
| set -euo pipefail | ||
| # Fails loudly if the two-repo layout is ever broken, rather than | ||
| # producing a confusing compile error thousands of lines later. | ||
| cargo metadata --format-version 1 >/dev/null | ||
| echo "✅ dependency graph resolves (gitbot-shared-context found)" | ||
|
|
||
| ## === ZIG BENCH === | ||
| # benchmarks: | ||
| # name: Bench — Performance Regression | ||
| # runs-on: ubuntu-latest | ||
| # timeout-minutes: 15 | ||
| # steps: | ||
| # - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 | ||
| # - uses: goto-bus-stop/setup-zig@abea47f85e598557f500fa1fd2ab7464fcb39406 # v2.2.1 | ||
| # with: | ||
| # version: 0.15.0 | ||
| # - run: cd ffi/zig && zig build bench | ||
| - uses: Swatinem/rust-cache@f0deed1e0edfc6a9be95417288c0e1099b1eeec3 # v2.7.7 | ||
| with: | ||
| workspaces: fleet/bots/echidnabot | ||
|
|
||
| # ─── Readiness (CRG) ────────────────────────────────────────────── | ||
| # Component Readiness Grade: D (runs) → C (correct) → B (edge cases) | ||
| - name: Unit + Point-to-Point + End-to-End | ||
| working-directory: fleet/bots/echidnabot | ||
| run: | | ||
| set -euo pipefail | ||
| # Named explicitly. A bare `cargo test` also runs --lib, which hangs | ||
| # (see the header). Each binary maps to a taxonomy category: | ||
| # smoke -> Part I §8 Smoke | ||
| # property_tests -> Part I §9 Property-based | ||
| # lifecycle -> Part I §7 Lifecycle | ||
| # integration_tests -> Part I §2 Point-to-Point | ||
| # seam_test -> Part I §3 End-to-End (webhook -> corpus) | ||
| for t in smoke property_tests lifecycle integration_tests seam_test; do | ||
| echo "::group::$t" | ||
| cargo test --test "$t" -- --test-threads=4 | ||
| echo "::endgroup::" | ||
| done | ||
|
|
||
| # readiness: | ||
| # name: Readiness — Grade D/C/B | ||
| # runs-on: ubuntu-latest | ||
| # timeout-minutes: 10 | ||
| # steps: | ||
| # - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 | ||
| # - uses: dtolnay/rust-toolchain@4be9e76fd7c4901c61fb841f559994984270fce7 # stable | ||
| # - uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2 | ||
| # - run: cargo test --test readiness -- --nocapture | ||
| - name: Summary | ||
| if: always() | ||
| run: | | ||
| { | ||
| echo "### E2E + P2P + Unit" | ||
| echo | ||
| echo "| Suite | Taxonomy category |" | ||
| echo "|---|---|" | ||
| echo "| \`smoke\` | Part I §8 Smoke |" | ||
| echo "| \`property_tests\` | Part I §9 Property-based |" | ||
| echo "| \`lifecycle\` | Part I §7 Lifecycle |" | ||
| echo "| \`integration_tests\` | Part I §2 Point-to-Point |" | ||
| echo "| \`seam_test\` | Part I §3 End-to-End |" | ||
| echo | ||
| echo "**Known debt:** \`cargo test --lib\` hangs (>20 min, exit 124) and is" | ||
| echo "deliberately NOT run here. The integration suites above complete in ~2s." | ||
| echo "This is stated rather than silently excluded." | ||
| } >> "$GITHUB_STEP_SUMMARY" | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.