diff --git a/.vortex/docs/content/architecture.mdx b/.vortex/docs/content/architecture.mdx index 6989f029e..5f5f0ebbc 100644 --- a/.vortex/docs/content/architecture.mdx +++ b/.vortex/docs/content/architecture.mdx @@ -185,7 +185,7 @@ the schedule as needed. ## Documentation & onboarding -**Vortex** includes centralized documentation (what you’re reading now), as well as +**Vortex** includes centralized documentation (what you're reading now), as well as a scaffold for adding project-specific docs within your own repository. ## Automation scripts @@ -246,3 +246,18 @@ These variables are either defined in the `.env` file or set within CI/hosting environments as secrets. ➡️ See [Variables](./development/variables) + +## Testing the template + +**Vortex** itself is tested at multiple levels to ensure the template and its +automation scripts work correctly: + +- **Unit tests**: Shell scripts are tested in isolation using + [Bats](https://github.com/bats-core/bats-core) with mocked commands +- **End-to-end tests**: Full build pipelines are tested using + [PHPUnit](https://phpunit.de/) in real Docker containers +- **Real-world validation**: The + [DrevOps website](https://github.com/drevops/website) serves as a production + reference site that receives regular upstream updates + +➡️ See [Contributing > Maintenance > Tests](./contributing/maintenance/tests) diff --git a/.vortex/docs/content/contributing/maintenance/tests.mdx b/.vortex/docs/content/contributing/maintenance/tests.mdx index 9a0e4542d..fc7147353 100644 --- a/.vortex/docs/content/contributing/maintenance/tests.mdx +++ b/.vortex/docs/content/contributing/maintenance/tests.mdx @@ -1,25 +1,81 @@ -# Authoring tests +# Tests -There are 2 types of tests used in **Vortex**: +## Testing strategy -- [PHPUnit](https://phpunit.de/) for functional end-to-end testing of a site build pipeline.
-With help of [`phpunit-helpers`](https://github.com/AlexSkrypnyk/phpunit-helpers) testing library, we run tests -for automations, including container builds, database imports, and Drupal -operations as if they were run by a consumer site developer. +**Vortex** is a project template, not a traditional application. A change to +a script, configuration file, or workflow can affect every project that uses +it. Because of this, **Vortex** uses a multi-layered testing approach to catch +issues at different levels of integration before they reach consumers. -- [Bats](https://github.com/bats-core/bats-core) for unit testing of scripts in `scripts/vortex`.
- With help of [`BATS Helpers`](https://github.com/drevops/bats-helpers) testing - library, we run unit tests for shell scripts with coverage. +### Unit testing - :::info +[Bats](https://github.com/bats-core/bats-core) is used to unit test the +shell scripts in `scripts/vortex/`. Each script is tested in isolation with +external commands (like `drush`, `docker`, `composer`) replaced by mocks. +This allows us to verify that individual scripts handle environment variables, +flags, and edge cases correctly without needing a running Drupal site or +Docker containers. - Heads up! Scripts are changing from Bash to PHP in `2.0` release. - This will make the use of BATS obsolete. Track the progress - in [this issue](https://github.com/drevops/vortex/issues/1192). +The [`bats-helpers`](https://github.com/drevops/bats-helpers) library provides +a step-based testing approach with built-in mocking and assertions, making it +straightforward to define expected inputs and outputs for each script. - ::: +Unit tests execute in seconds, providing fast feedback during development. +There are 25+ test files covering deployment, database operations, +notifications, provisioning, and other automation scripts. -## Functional tests with PHPUnit +:::info + +Scripts are transitioning from Bash to PHP in **Vortex** `2.0`, which will +make the use of BATS obsolete. Track the progress +in [this issue](https://github.com/drevops/vortex/issues/1192). + +::: + +### End-to-end testing + +[PHPUnit](https://phpunit.de/) is used for functional end-to-end testing that +exercises the full site build pipeline in real Docker containers. These tests +simulate what a consumer site developer would experience: installing the +template, building containers, importing databases, running Drupal operations, +linting code, and deploying artifacts. + +The end-to-end tests cover several critical workflows: + +- **Installer** - verifying that a fresh project can be scaffolded from the + template and that custom files are preserved during updates. +- **Docker Compose** - building the container stack, verifying environment + variables, running linters, executing PHPUnit and Behat tests inside + containers, and checking Solr integration. +- **Ahoy workflows** - testing the full developer experience through Ahoy + commands, including build, login, Drush operations, Composer, database + import/export, and code quality checks. +- **Deployment** - testing deployment pipelines for artifact and webhook-based + deployment strategies. + +The [`phpunit-helpers`](https://github.com/AlexSkrypnyk/phpunit-helpers) +library provides test helpers, traits, and assertions purpose-built for running +shell commands and validating their output within PHPUnit tests. + +End-to-end tests take minutes to run because they operate on real containers +and a real Drupal site, but they provide the highest confidence that +everything works together correctly. + +### Example site + +The [DrevOps website](https://github.com/drevops/website) is a real-world +production site built using **Vortex**. It serves as the ultimate validation +ground: if **Vortex** works correctly, the website should continue to build, +test, and deploy without issues after each upstream update. + +The website repository receives regular upstream updates from **Vortex**, +including CI configuration, testing workflows, and infrastructure changes. +This ensures that updates are validated against a real, long-lived codebase +rather than just in test environments. + +## Running tests + +### Functional tests with PHPUnit ```shell cd .vortex/tests @@ -35,9 +91,9 @@ TEST_PACKAGE_TOKEN= TEST_VORTEX_CONTAINER_REGISTRY_USER= TE ``` Functional tests rely on database fixtures - see the section below on -[updating test](#updating-test-assets) assets. +[updating test assets](#updating-test-assets). -## Unit tests with BATS +### Unit tests with BATS ```shell cd .vortex/tests @@ -60,6 +116,9 @@ There are *demo* and *test* database dumps captured as *files* and *container im ### Updating *demo* database dump *file* +
+Show instructions + 1. Run fresh build of **Vortex** locally: ```shell rm .data/db.sql || true @@ -90,8 +149,13 @@ ahoy export-db db.demo.sql ``` 4. Upload `db.demo.sql` to the latest release as an asset and name it `db_d11.demo.sql`. +
+ ### Updating *demo* database *container image* +
+Show instructions + 1. Run fresh build of **Vortex** locally: ```shell rm .data/db.sql || true @@ -130,8 +194,13 @@ chmod +x seed.sh ./seed.sh .data/db.demo_image.sql drevops/vortex-dev-mariadb-drupal-data-demo-11.x:latest ``` +
+ ### Updating *test* database dump *file* +
+Show instructions + 1. Run a fresh install of **Vortex** into a new directory and name the project `Star Wars`: ```shell mkdir /tmp/star-wars @@ -168,8 +237,13 @@ ahoy export-db db.test.sql ``` 5. Upload `db.test.sql` to the latest release as an asset and name it `db_d11.test.sql`. +
+ ### Updating *test* database *container image* +
+Show instructions + 1. Run a fresh install of **Vortex** into a new directory and name the project `Star Wars`: ```shell mkdir /tmp/star-wars @@ -221,3 +295,5 @@ docker push drevops/vortex-dev-mariadb-drupal-data-demo-destination-11.x:vortex- docker tag drevops/vortex-dev-mariadb-drupal-data-demo-11.x:latest drevops/vortex-dev-mariadb-drupal-data-demo-destination-11.x:vortex-dev-didi-database-fi docker push drevops/vortex-dev-mariadb-drupal-data-demo-destination-11.x:vortex-dev-didi-database-fi ``` + +