Design Document on improving the Drasi E2E test framework #13
Open
ruokun-niu wants to merge 15 commits into
Open
Design Document on improving the Drasi E2E test framework #13ruokun-niu wants to merge 15 commits into
ruokun-niu wants to merge 15 commits into
Conversation
Signed-off-by: ruokun-niu <ruokunniu@gmail.com>
ruokun-niu
requested review from
agentofreality,
amansinghoriginal and
danielgerlag
Signed-off-by: ruokun-niu <ruokunniu@gmail.com>
Signed-off-by: ruokun-niu <ruokunniu@gmail.com>
Signed-off-by: ruokun-niu <ruokunniu@gmail.com>
Signed-off-by: ruokun-niu <ruokunniu@gmail.com>
There was a problem hiding this comment.
Pull request overview
This PR adds a new design document proposing improvements to the Drasi E2E test framework, with a focus on better supporting drasi-lib and drasi-server execution modes and reducing config duplication across target variants.
Changes:
- Adds a “Test Framework Plan” document describing current ETF execution patterns (drasi-lib in-process, drasi-server standalone, drasi-platform on Kubernetes) and CI/run workflows.
- Proposes a new configuration model that splits test scenario definition (
test-config.yaml) from target-specific runtime patterns (pattern files) and describes how the framework merges them. - Outlines an expanded test suite scope (indexes, failure recovery), reporting approach, and test data hosting considerations.
Comment on lines
+1
to
+5
| # Test Framework Plan | ||
|
|
||
| * Project Drasi — April 29, 2026 — Ruokun Niu (@ruokun-niu) | ||
|
|
||
| ## Test Framework Requirements |
| ## Test Framework Requirements | ||
|
|
||
|
|
||
| **Runnable manually.** Developers must be able to run the full test suite from their local machine or as a Github Actions workflow (triggered via `workflow_dispatch`). The existing building_comfort examples in the [test-infra](https://github.com/drasi-project/test-infra) repo demonstrate three distinct run patterns: |
| **Runnable manually.** Developers must be able to run the full test suite from their local machine or as a Github Actions workflow (triggered via `workflow_dispatch`). The existing building_comfort examples in the [test-infra](https://github.com/drasi-project/test-infra) repo demonstrate three distinct run patterns: | ||
|
|
||
| - **drasi-lib (in-process):** A single shell script runs the test-service with drasi-lib compiled in — no external processes needed. The script invokes `cargo run --release --manifest-path ./test-service/Cargo.toml -- --config <config.json>` from the `e2e-test-framework` directory. Everything runs in one process. | ||
| - **Drasi Server (standalone):** A shell script first builds and starts the drasi-server binary (from a sibling `../../drasi-server` directory) with a `server-config.yaml`, waits for its health check, and then starts the test-service with a separate `config.json` that dispatches changes via HTTP or gRPC to the running server. Currently it can use either a prebuilt binary or execute `cargo run` from a `drasi-server` repo. It is also worthing noting that we have four variants of the drasi-server pattern: http, http adaptive, grpc and grpc adaptive. |
Comment on lines
+173
to
+176
| #### Proposed Changes | ||
|
|
||
| The goal is to have a single central test config per test scenario that works with any Drasi target. The test config defines the "what" (test ID, data model, queries, stop criteria), and a separate pattern file defines the "how" (dispatchers, handlers, engine/server config, runtime settings). The ETF merges them at load time. | ||
|
|
Comment on lines
+361
to
+363
| - **Hugging Face Hub (recommended for test data files):** Public datasets on Hugging Face are freely downloadable without authentication and without the rate limiting issues of GitHub's API. Files are accessible via a predictable HTTP URL pattern: `https://huggingface.co/datasets/{org}/{repo}/resolve/main/{path}`. We would create a public dataset repo (e.g., `drasi-project/test-data`) and push JSONL files to it. HF Hub is git-based and automatically uses Git LFS for large files, so versioning is built in. The ETF would download files with a simple HTTP GET to the resolve URL — no special client library needed. This is free, public, requires no infrastructure to maintain, and has no rate limits for normal usage. | ||
|
|
||
| The recommended approach is: **test configs and golden files in the repo** (LocalStorage), **JSONL test data files on Hugging Face Hub** (downloaded via HTTP at test time or pre-cached locally). The ETF would need a small addition to support fetching data from an HTTP URL — either a new `HuggingFace` backend or a generic `HttpDownload` backend that accepts a base URL and file paths. Alternatively, a shell script in the test setup phase could pre-download the data into the local data store path before the ETF starts. |
Signed-off-by: ruokun-niu <ruokunniu@gmail.com>
Signed-off-by: ruokun-niu <ruokunniu@gmail.com>
Signed-off-by: ruokun-niu <ruokunniu@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces a design document on areas that we can improve/update the Drasi E2E test framework with (better) support for Lib and Server
Type of change
Fixes: #issue_number