feat: add verify --basic flag to run a fast test subset#518
Open
ian-flores wants to merge 6 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a fast “confidence check” mode to VIP by introducing a --basic flag for vip verify that excludes scenarios tagged @slow, letting users run a smaller core subset without the longest Workbench checks.
Changes:
- Registers a new
slowpytest marker and tags the four heavy Workbench feature files with@slow. - Extends
vip verifywith--basic, composing it into the existing pytest-mmarker-expression logic (including--categories). - Adds selftests validating marker registration, deselection behavior, and the CLI marker-expression assembly; updates user/docs guidance.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/vip/plugin.py | Registers the new slow marker at runtime via the VIP pytest plugin. |
| src/vip/cli.py | Adds --basic to vip verify and appends and not slow to the marker expression. |
| src/vip_tests/workbench/test_publish_to_connect.feature | Tags the feature as @slow so it’s excluded by --basic. |
| src/vip_tests/workbench/test_jobs.feature | Tags the feature as @slow so it’s excluded by --basic. |
| src/vip_tests/workbench/test_ide_extensions.feature | Tags the feature as @slow so it’s excluded by --basic. |
| src/vip_tests/workbench/test_git_ops.feature | Tags the feature as @slow so it’s excluded by --basic. |
| selftests/test_plugin.py | Adds selftests for marker registration and -m "not slow" deselection. |
| selftests/test_gherkin.py | Adds a guard asserting the intended Workbench features are tagged @slow (and Chronicle isn’t). |
| selftests/test_cli_verify.py | Adds selftests ensuring --basic modifies the assembled pytest -m expression and composes with --categories. |
| README.md | Documents vip verify --basic as a fast confidence check. |
| pyproject.toml | Registers the slow marker in pytest ini options for repo-root runs. |
| docs/test-architecture.md | Documents --basic semantics and how @slow tagging works across categories. |
| AGENTS.md | Updates “Key source files” notes to mention the new marker/flag behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Preview Links
|
Collaborator
Author
|
@samcofer I'd like your opinion here on the suite of tests supported under --basic |
ian-flores
marked this pull request as ready for review
July 21, 2026 22:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #400.
vip verifyruns the full suite on every invocation, including detailed/long-running Workbench checks (IDE extension installation, job execution, Git operations, publishing to Connect) that aren't needed for a quick confidence check. This adds a--basicflag that runs only the core subset by excluding tests tagged@slow.Approach
A denylist that mirrors the existing
--performance-testsopt-in mechanism, inverted. A newslowpytest marker is registered;--basicappendsand not slowto the pytest-mmarker expression. It composes with--categories(e.g.vip verify --categories workbench --basicruns the Workbench category minus its slow scenarios). The mechanism is product-agnostic — any feature can be tagged@slow.Today the
@slowset is the four heavy Workbench features. A--basicrun still exercises auth, IDE launch, sessions, runtime versions, packages, data sources, and Chronicle.Changes
slowmarker in bothsrc/vip/plugin.pyandpyproject.toml.--basictovip verifyand applyand not slowin both marker-expression branches.test_ide_extensions,test_jobs,test_git_ops, andtest_publish_to_connectwith@slow.docs/test-architecture.md,README.md, andAGENTS.md/CLAUDE.md.Testing
just checkclean (ruff lint + format).--basicadds/omitsnot slow, composition with--categories,@slowdeselection under-m "not slow", and a membership guard asserting the four files are tagged and Chronicle is not.selftests/test_load_engine.py(unrelated to this change).