Bfops/test pr#4144
Closed
bfops wants to merge 72 commits into
Closed
Conversation
Use OnceLock to build spacetimedb-cli and spacetimedb-standalone once per test process, then run the pre-built binary directly instead of using `cargo run`. This avoids repeated cargo overhead and ensures consistent binary reuse across parallel tests.
Create `crates/smoketests/` to translate Python smoketests to Rust: - Add `Smoketest` struct with builder pattern for test setup - Implement CLI helpers: `spacetime_cmd()`, `call()`, `sql()`, `logs()`, etc. - Translate `smoketests/tests/sql.py` → `tests/sql.rs` - Translate `smoketests/tests/call.py` → `tests/call.rs` - Reuse `ensure_binaries_built()` from guard crate (now public) Also fix Windows process cleanup in `SpacetimeDbGuard`: - Use `taskkill /F /T /PID` to kill entire process tree - Prevents orphaned `spacetimedb-standalone.exe` processes
- Translate 4 more Python smoketests to Rust: auto_inc, describe, module_nested_op, panic - Simplify the call API by removing the generic call<T: Serialize> method and renaming call_raw to call, since CLI args are strings - Remove unused serde dependency
Translate additional Python smoketests to Rust: - dml.rs: DML subscription tests - filtering.rs: Unique/non-unique index filtering tests - namespaces.rs: C# code generation namespace tests - add_remove_index.rs: Index add/remove with subscription tests - schedule_reducer.rs: Scheduled reducer tests Infrastructure improvements: - Add subscribe_background() and SubscriptionHandle for proper background subscription semantics matching Python tests - Add spacetime_local() for commands that don't need --server flag - Add timing instrumentation for debugging test performance
- Separate build and publish timing in lib.rs to identify bottlenecks - Use --bin-path to skip redundant rebuild during publish - Add DEVELOP.md explaining cargo-nextest for faster test runs Timing breakdown per test: - WASM build: ~12s (75%) - Server publish: ~2s (12%) - Server spawn: ~2s (12%) cargo-nextest runs all test binaries in parallel, reducing total runtime from ~265s to ~160s (40% faster).
Translate from Python smoketests: - detect_wasm_bindgen.rs: Tests build rejects wasm_bindgen and getrandom (2 tests) - default_module_clippy.rs: Tests default module passes clippy - delete_database.rs: Tests deleting database stops scheduled reducers - fail_initial_publish.rs: Tests failed publish doesn't corrupt control DB - modules.rs: Tests module update lifecycle and breaking changes (2 tests) Also adds spacetime_build() method to Smoketest for testing build failures. Total: 16 test files translated, 32 tests
Clear CARGO* environment variables (except CARGO_HOME) when spawning child cargo build processes. When running under `cargo test`, cargo sets env vars like CARGO_ENCODED_RUSTFLAGS that differ from a normal build, causing child cargo processes to think they need to recompile. This reduces single-test runtime from ~45s to ~18s by avoiding redundant rebuilds of spacetimedb-standalone and spacetimedb-cli.
Add test translations for: - connect_disconnect_from_cli.rs - client connection callbacks - domains.rs - database rename functionality - client_connection_errors.rs - client_connected error handling - confirmed_reads.rs - --confirmed flag for subscriptions/SQL - create_project.rs - spacetime init command Also fix subscription race condition by waiting for initial update before returning from subscribe_background_*, matching Python behavior.
Translate tests for: - views.rs: st_view_* system tables, namespace collisions, SQL views - auto_migration.rs: schema changes, add table migration
Add new_identity() method to support multi-identity tests. Translate tests for: - rls.rs: Row-level security filter tests - energy.rs: Energy balance endpoint test - permissions.rs: Private tables, lifecycle reducers, delete protection
Translate tests for: - new_user_flow.rs: Basic publish/call/SQL workflow - servers.rs: Server add/list/edit commands
- Add --config-path to spacetime_local() for test isolation - Fix new_identity() to not pass server arg to logout (matches Python) - Insert --server flag before -- separator in spacetime_cmd() - Update servers.rs to use spacetime_local() for local-only commands - Simplify test files by removing redundant publish_module() calls All 56 smoketests now pass.
Translate smoketests/tests/quickstart.py to Rust. This test validates that the quickstart documentation is correct by extracting code from markdown docs and running it. - Add parse_quickstart() to parse code blocks from markdown with CRLF handling - Add have_pnpm() to check for pnpm availability - Implement QuickstartTest with support for Rust, C#, and TypeScript servers - Rust test passes; C#/TypeScript skip gracefully if dependencies unavailable
…s' of github.com:clockworklabs/SpacetimeDB into tyler/translate-smoketests
Extract static smoketest modules into a nested workspace at crates/smoketests/modules/ that is pre-compiled during warmup. This eliminates per-test WASM compilation overhead. Key changes: - Add 38 precompiled module crates in nested workspace - Add module registry (src/modules.rs) for WASM path lookup - Add precompiled_module() builder and use_precompiled_module() method - Update xtask warmup to build nested workspace - Migrate all static tests to use precompiled modules - Tests using precompiled modules run in ~0.5-3s vs ~4-7s before Tests that need runtime compilation (auto_migration, detect_wasm_bindgen, intentionally-broken modules) continue to use module_code().
- Build CLI/standalone in release mode for faster test execution - Run tests in release mode (faster SpacetimeDB server) - Remove redundant WASM cache warmup (precompiled modules already warm it) - Increase default parallelism to 16 (optimal based on benchmarks) This reduces fresh build time by ~16 seconds and test execution time by ~5x compared to debug mode.
- Remove `dotnet nuget locals all --clear` from csharp_module.rs and quickstart.rs which caused race conditions when tests ran in parallel - Add `<clear />` in NuGet.Config packageSources to avoid inheriting sources from machine/user config (proper isolation without clearing) - Add explicit nuget.org source URL for reliable fallback - Use `-c Release` for dotnet pack to match CI configuration - Add dotnet clean before pack to avoid stale artifacts - Add duplicate source/mapping checks in override_nuget_package - Add durability checks in restart tests before server restart
- Rename existing smoketests job to use cargo smoketest (xtask) which handles pre-building binaries, using nextest, and optimal parallelism - Add cargo-nextest installation step for better parallel execution - Add new "Smoketests (Python Legacy)" job that runs the old Python smoketests alongside the Rust ones for comparison during transition - Add warn-python-smoketests job that posts a PR comment when Python smoketests are modified, encouraging use of Rust smoketests instead - Update smoketests/README.md to note the transition to Rust smoketests
- Add smoketest_args matrix (--docker for Linux, --no-build-cli for Windows) - Add excluded tests: -x clear_database replication teams - Add Build crates step - Add Docker daemon startup and database setup for Linux - Add database startup for Windows - Add container cleanup step - Match Python version to 3.12 on Windows like master
Scans the target directory for smoketest_module_*.wasm files and derives module names from filenames (underscores become hyphens). This eliminates the need to manually maintain the registry.
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com> Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Enable running Rust smoketests against a remote server instead of spawning local servers, similar to Python smoketests --remote-server. - Add SPACETIME_REMOTE_SERVER env var support to skip local server spawn - Add --server CLI option to cargo smoketest - Add skip_if_remote!() macro for tests requiring local server control - Mark restart tests with skip_if_remote!() since they need local server
Smoketests require pre-built binaries and have their own dedicated command (cargo ci smoketests) that builds binaries first. Exclude them from cargo test --all to prevent failures in CI.
Ignore obj/ and bin/ directories which are created when dotnet commands scan for .csproj files during CI.
Update module discovery to respect CARGO_TARGET_DIR env var when looking for precompiled WASM modules. CI sets this for the main workspace, so the modules get built there instead of in the modules workspace's own target directory.
Replace individual autoinc modules (u32, u64, i32, i64) with two consolidated modules that test all 10 integer types (u8, u16, u32, u64, u128, i8, i16, i32, i64, i128), matching the Python tests. - Create autoinc-basic module with macro for all types - Create autoinc-unique module with macro for all types - Update test to iterate over all types in a single test - Remove individual type-specific modules
Use sql_confirmed() instead of sql() for the SQL INSERT in the confirmed subscription test. The confirmed subscription only sends updates after the transaction is durable, so using sql_confirmed() ensures the INSERT is durable before we start collecting updates.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of Changes
API and ABI breaking changes
Expected complexity level and risk
Testing