Skip to content

feature: Test tags (TestTag) for UniTest — run testing layers separately#617

Merged
xerial merged 8 commits into
mainfrom
feature/uni-test-layer-tags
Jun 27, 2026
Merged

feature: Test tags (TestTag) for UniTest — run testing layers separately#617
xerial merged 8 commits into
mainfrom
feature/uni-test-layer-tags

Conversation

@xerial

@xerial xerial commented Jun 27, 2026

Copy link
Copy Markdown
Member

Why

Part of making Uni a foundation for multi-layer testing of rich desktop apps (see plans/2026-06-27-multi-layer-testing.md). VSCode keeps separate test commands per layer (unit / integration / UI / smoke); to do the same here, UniTest needs a way to mark which layer a test belongs to and run/skip one layer at a time. This is the mechanism that makes the other layers (UI toolkit, Electron harness in #616) runnable as independent suites in CI.

What

A TestTag varargs API on test(...) that unifies the old flaky boolean and ad-hoc tags:

test("renders the toolbar", UI) { ... }
test("hits the network", Electron, Slow) { ... }
test("retried under load", Flaky) { ... }      // Flaky is itself a tag: failure -> skipped
test("smoke check", TestTag("smoke")) { ... }  // custom tag (or a bare string)
  • trait TestTag { def name: String }; built-ins Flaky, UI, Electron, Integration, Slow exported into UniTest scope; custom via TestTag("name") or a String (given Conversion).
  • test(name: String, tags: TestTag*)(body)isFlaky derives from the presence of Flaky; TestDef stays string-based internally so the runner is unchanged.
  • sbt selection (long options use a -- prefix per repo convention; short options -l/-t: keep one hyphen):
    • --tags:a,b — include filter: run only tests carrying any of these tags
    • --exclude-tags:a,b — exclude filter: skip tests carrying any of these tags
    • exclusion wins over inclusion
  • Documented in .github/instructions/unitest.instructions.md.
./sbt "coreJVM/testOnly * -- --tags:ui,electron"    # ui OR electron
./sbt "coreJVM/testOnly * -- --exclude-tags:slow"   # everything except slow

Breaking change

test(name, flaky = true)test(name, Flaky); test(name, tags = Seq("ui"))test(name, UI) (or TestTag("ui")). Only one in-repo call site used flaky = true; it's migrated. No other open branch uses the old params.

Tests

TagFilterTest (12) covers arg parsing, includesTags semantics, TestTag names, the FlakyisFlaky mapping, layer-tag registration, and the String conversion. Verified end-to-end via the sbt runner (--tags:meta → 1 test, --exclude-tags:meta → rest). Full uni-test JVM suite green (69); JS + Native and every JVM test module compile.

Roadmap: unit → UI (#616) → Electron (#616) → tags (this) → plugin (#618).

🤖 Generated with Claude Code

Lets a `test(...)` carry tags (e.g. "ui", "electron", "slow") so one suite can span
multiple testing layers and still be run layer-by-layer, mirroring VSCode's separate
unit/integration/UI test commands — the mechanism that makes Uni's multi-layer testing
practical in CI.

- `test(name, flaky, tags = Seq("ui"))` records tags on the `TestDef`.
- sbt args: `-tag:a,b` include filter (run tests with any listed tag), `-xtag:a,b`
  exclude filter (skip tests with any listed tag); exclusion wins over inclusion.
- `TestConfig` parses the args and exposes `includesTags`; `UniTestTask` applies the
  filter alongside the existing `-t:` name filter.
- Backward compatible: `tags` defaults to empty, existing `test(...)` / `test(..., flaky=true)`
  call sites are unchanged.

Documented in .github/instructions/unitest.instructions.md. TagFilterTest covers parsing +
filter semantics + end-to-end selection; full uni-test suite green on JVM (65), compiles on
JS and Native.

Part of plans/2026-06-27-multi-layer-testing.md (PR 3 of the unit → UI → Electron → plugin
roadmap).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added the feature New feature label Jun 27, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces test tagging and filtering capabilities to the uni-test framework, allowing users to run or skip specific testing layers (such as unit, UI, or integration tests) using the -tag: and -xtag: command-line options. The changes include updates to the configuration parser, task runner, documentation, and a new suite of unit tests. The feedback suggests trimming and filtering empty strings from the tags during registration to prevent potential issues with leading or trailing whitespace.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +91 to +93
protected def test(name: String, flaky: Boolean = false, tags: Seq[String] = Nil)(
body: => Any
): Unit = _tests += TestDef(name, () => body, _context, isFlaky = flaky, tags = tags.toSet)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To prevent subtle bugs where tests are silently skipped due to accidental leading/trailing whitespace or empty strings in the tags parameter (e.g., Seq(" ui ")), it is safer to trim and filter the tags when registering them, just like we do when parsing command-line arguments.

  protected def test(name: String, flaky: Boolean = false, tags: Seq[String] = Nil)(
      body: => Any
  ): Unit = _tests += TestDef(name, () => body, _context, isFlaky = flaky, tags = tags.map(_.trim).filter(_.nonEmpty).toSet)

xerial and others added 2 commits June 27, 2026 14:44
… / --exclude-tags)

Renames the tag-filter CLI args from -tag:/-xtag: to --tags:/--exclude-tags:, following
the convention of double-hyphen prefixes for long option names. Short options (-l, -t:)
keep their single hyphen. Updates TestConfig parsing, docs, and TagFilterTest.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Unifies the `flaky: Boolean` and `tags: Seq[String]` parameters of `test(...)` into a single
`tags: TestTag*` varargs, with `Flaky` now a built-in `TestTag` rather than a special boolean.
This makes flaky-ness "just another tag" and reads more naturally:

  test("renders", UI)               // was: test("renders", tags = Seq("ui"))
  test("retried", Flaky, Slow)      // was: test("retried", flaky = true, tags = Seq("slow"))

- `TestTag` trait (`name`); `TestTag.Flaky` (failure -> skipped) + common layer tags
  `UI`/`Electron`/`Integration`/`Slow`, all exported into `UniTest` scope.
- Custom tags via `TestTag("name")` or a bare String (given Conversion).
- `TestDef` stays string-based internally, so the runner / `--tags`/`--exclude-tags` filtering
  is unchanged; `isFlaky` derives from the presence of the `Flaky` tag.
- Migrated the one `flaky = true` call site; expanded TagFilterTest; updated docs.

Breaking: `test(name, flaky = true)` / `test(name, tags = Seq(...))` become
`test(name, Flaky)` / `test(name, UI, ...)`. uni-test JVM suite green (69); JS + Native and all
JVM test modules compile.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@xerial xerial changed the title feature: Test tags for UniTest — run testing layers separately feature: Test tags (TestTag) for UniTest — run testing layers separately Jun 27, 2026
xerial and others added 5 commits June 27, 2026 15:11
Scala 3 traits take parameters, so `TestTag(val name: String)` carries the name directly and
the built-ins become one-line case objects (Flaky/UI/Electron/Integration/Slow). This removes
the separate `Named` case class; ad-hoc tags are anonymous instances via `TestTag("name")`.
Behavior is unchanged (filtering is by name); case objects keep clean toStrings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Smoke tests (quick cross-stack sanity checks) are a standard category, so add
`TestTag.Smoke` alongside the other built-ins and export it into UniTest scope. Also adds
doc comments distinguishing Smoke (fast pre-merge subset) from Slow.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The custom-tag example used TestTag("smoke"), which is now a built-in. Show Smoke as a
built-in category tag, switch the custom example to TestTag("legacy"), and add a
--tags:smoke pre-merge subset command.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switches the tag filters from the colon/comma form (--tags:ui,electron) to plain
space-separated values that follow the existing -l two-token style, with no special
characters: --tags <tag>, repeatable to add more (--tags ui --tags electron). Same for
--exclude-tags. Empty/whitespace values are ignored; the comma-splitting helper is removed.

Updates the parser, docstrings, instructions, and TagFilterTest. uni-test JVM suite green
(70); JS + Native compile.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Changes the include filter from OR (any listed tag) to AND (all listed tags): each repeated
`--tags` further narrows the selection, matching how GitHub issue label filters compose.
A test runs iff it carries every `--tags` tag and none of the `--exclude-tags` tags.

OR/union is intentionally not supported (YAGNI) — narrowing is the dominant need, and a future
`--tags <expr>` could add explicit AND/OR expression syntax without breaking plain single-tag
usage (noted in the parser). Single-tag behavior is unchanged.

Updates includesTags (now `includeTags.subsetOf(tags)`), docstrings, instructions, and
TagFilterTest (adds an AND-narrowing case). Verified end-to-end via the sbt runner
(`--tags meta --tags self` -> 1 test; `--tags meta --tags nonexistent` -> 0). uni-test JVM
suite green (71); JS + Native compile.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@xerial
xerial merged commit fd58a79 into main Jun 27, 2026
14 checks passed
@xerial
xerial deleted the feature/uni-test-layer-tags branch June 27, 2026 23:10
xerial added a commit that referenced this pull request Jul 6, 2026
… (wvlet.uni.plugin) (#618)

## Why

Final layer of the multi-layer testing goal: make Uni a foundation for
**apps built from pluggable units, testable in isolation** — VSCode's
extension/plugin testing layer. Uni had no plugin/extension model; this
adds a minimal one plus the harness to test plugins.

Since `wvlet.uni.plugin` is a top-level namespace alongside `design`,
`rx`, `surface` — and uni is not Electron-specific — the core model must
be generic: it hardcodes **no** contribution kinds and has **no**
dependency on `wvlet.uni.http.rpc`. See
[`adr/2026-07-06-plugin-extension-points.md`](https://github.com/wvlet/uni/blob/feature/uni-plugin-model-and-test-harness/adr/2026-07-06-plugin-extension-points.md).

## What

`wvlet.uni.plugin` — a VSCode-style extension model built on **typed
extension points**, shipped in the published `uni` artifact
(cross-platform: JVM/JS/Native):

- **`Plugin`** — `id` + `activate(context)`, mirroring VSCode's
`activate(context)`.
- **`PluginContext`** — deliberately minimal registration surface:
`contribute(point)(value)` + `onDeactivate(hook)`.
- **`ExtensionPoint[A]`** — a typed contribution slot, compared by
identity (define points as shared `val`s).
`ExtensionPoint.keyed(name)(keyOf)` makes contributions unique by key: a
duplicate key — within a plugin or across plugins — is rejected at
activation, so conflicts surface in tests rather than at runtime.
- **`PluginHost`** — activates plugins and owns the per-point
registries: `contributions(point)`, `contribution(point, key)`,
`deactivate()` (FILO hooks + reset). Re-activating the same plugin id is
rejected.

Contribution kinds are defined **next to their types**, so dependency
arrows point *into* the plugin layer:

- **`wvlet.uni.plugin.Command`** — `Command.point` (keyed by command id)
+ extension methods `registerCommand`, `executeCommand`, `commandIds`,
`hasCommand`.
- **`wvlet.uni.http.rpc.RPCPlugin`** — `routerPoint` + extension methods
`registerRpcRouter`, `rpcRouters`, `rpcDispatcher` (an `RPCDispatcher`
over all contributed routers, ready to serve via HTTP or Electron IPC).

New kinds (views, menus, `Design` bindings) are added by defining a
point — not by editing the core.

```scala
import wvlet.uni.plugin.Command.*
import wvlet.uni.http.rpc.RPCPlugin.*

class NotesPlugin extends Plugin:
  def id = "notes"
  def activate(ctx: PluginContext): Unit =
    ctx.registerRpcRouter(RPCRouter.of[NotesApi](NotesApiImpl()))
    ctx.registerCommand("notes.new")(_ => createNote())
    ctx.onDeactivate(() => flushToDisk())
```

## Test harness

- **`wvlet.uni.plugin.testing.PluginTestHost`** — `activate(plugin)` /
`activateAll(plugins*)` activate plugin(s) against a fresh host through
the real activation path and return the host for assertions. Because the
host is cross-platform, **plugin tests run on the JVM with no
Electron/browser runtime**.

```scala
val host = PluginTestHost.activate(NotesPlugin())
host.commandIds shouldContain "notes.new"
host.executeCommand("notes.new") shouldBe expectedNote
host.rpcRouters.flatMap(_.routes.map(_.path)).exists(_.endsWith("/list")) shouldBe true
```

## Tests

`PluginHostTest` (10 tests, JVM green; compiles on JS + Native): command
registration/execution, unknown-command error, RPC-router contribution,
app-defined extension points, unkeyed multi-contribution, lifecycle
teardown, and conflict rejection (duplicate within a plugin, across
plugins, and double-activation).

## Roadmap status

unit → UI (#616) → Electron (#616) → tags (#617) → **plugin (this)**.
Richer contribution points (views, menus, a typed command API, `Design`
bindings), and a JS bridge from `RPCPlugin.rpcDispatcher` into
`ElectronTestbed` for full end-to-end plugin-RPC tests, are natural
follow-ups.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant