feat: per-test configuration for triggers and alerts

[metalwarrior665]

This will need some small changes in source actions but for repos, it is backward compatible

AI SUMMARY BELOW

Summary

This PR introduces a trigger-based test filtering system and alert configuration framework to the actor testing library. Tests and test suites can now be conditionally run based on trigger types (hourly, daily, pull request) and configured to send alerts to specific channels (Slack) when they fail.

Key Changes

• New trigger system (lib/trigger.ts):

  • getCurrentTrigger(): Reads the TEST_TRIGGER environment variable to determine the active trigger type
  • shouldRunForTrigger(): Evaluates whether a test should run based on its runWhen configuration and the current trigger
  • Supports three trigger types: hourly, daily, pullRequest

• Hierarchical trigger inheritance:

  • Added mergeInheritedTriggers() function that merges trigger configurations field-by-field through the describe hierarchy
  • Implements a trigger stack that tracks nested describe blocks, allowing child tests to inherit and override parent trigger settings
  • Both runWhen and alerts configurations are shallow-merged so children only need to override specific keys

• Updated describe() API:

  • Now accepts either a config object (DescribeConfig) or legacy string-based arguments
  • Config object supports name, triggers, and options properties
  • Maintains backward compatibility with existing string-based syntax
  • Pushes/pops triggers onto the stack during suite collection

• Updated testActor() and testStandbyActor() APIs:

  • Now accept either a config object (TestActorConfig) or legacy string-based arguments
  • Config object supports name, triggers, and options properties
  • Merges inherited triggers from enclosing describe blocks with test-level overrides
  • Embeds alerts configuration in task metadata for downstream processing by the report tool

• New type definitions (lib/types.ts):

  • TriggerType: Union of valid trigger types
  • RunWhenConfig: Controls which triggers enable a test
  • AlertsConfig: Defines alerting channels (currently Slack)
  • TriggerConfig: Combines runWhen and alerts
  • DescribeConfig and TestActorConfig: Configuration objects for the new API style
  • DescribeOptions and ActorOptions: Vitest-level test options

• Updated test reporting (bin/test-report.ts):

  • Reads alerts metadata from test results
  • Filters failed assertions by alert configuration (defaults to true for backward compatibility)
  • Only sends Slack notifications for assertions with alerts.slack !== false

• Comprehensive test coverage:

  • test/unit/triggers-merging.test.ts: Tests the trigger merging logic across various inheritance scenarios
  • test/unit/trigger.test.ts: Tests trigger detection and filtering logic

• Exported public API (index.ts):

  • Exports getCurrentTrigger() function
  • Exports new type definitions for trigger and alert configuration

Notable Implementation Details

• The trigger stack is maintained during vitest's synchronous suite collection phase, ensuring consistent state
• Trigger merging uses shallow object spread to allow field-by-field overrides without losing unspecified keys
• The system is backward-compatible: tests without trigger configuration run unconditionally, and the legacy string-based API is fully supported
• Alert configuration is embedded in task metadata so the JSON reporter can serialize it for downstream processing

https://claude.ai/code/session_01BHoLoyAPZsZB41xxysLagd

[ruocco-l]

Other than the pullRequest to be explicitly opt in, LGTM

[Patai5]

I trust the unit tests that were made by the LLM to do their job 😎

[metalwarrior665]

I have reverted the refactoring that moved some stuff to utils.ts, it was making the diff longer and making merging master harder. Let's do it later.

[metalwarrior665]

Corrected after merge conflicts. I will let it sit for a few days, then will do the last review, ping, and merge.

[oklinov]

I'm note sure about the

[oklinov]

is this necesarry? here you changed it from using a constant to inlining but in trigger.ts you're using constant. I'd make the work with environment variables consistent.

[oklinov]

this is a weird name 😄

[oklinov]

I'm not sure about this trigger system. It adds complexity to the library, it's hard to modify/extend (e.g. adding montly trigger) and it relies on an external trigger system (hourly tests won't be run hourly if you misconfigure cron schedule).

I would rather have my tests split in directories based on how often I want to run them:

└── test
    └── platform
        ├── hourly
        ├── daily
        └── montly

// or something different
└── test
    └── platform
        ├── tier1
        ├── tier2
        └── tier3

And it's then your (devs) job to run the correct test folder with correct schedule:

• 0 * * * * for test/platform/hourly or test/platform/tier1
• 0 0 * * * for test/platform/daily or test/platform/tier2

[metalwarrior665]

I will give this another thought. Your proposal would simplify it and we could always run daily tests inside weekly ones etc. But we will still need to define somewhere if it should run on PR and potentially the alerting config.

The advantage of my PR is that it makes everything super (maybe unnecessarily) flexible, you can name your files as you want and just compose via describe which is the intended way for tests. And we can add any other config easily later and reuse the composition system.

it's hard to modify/extend (e.g. adding montly trigger)

We could make this just a plain string and set it workflow but this is not something we will change often, at max we will likely have something like every 10 min, hourly, daily, weekly, monthly so it can be just hardcoded

it relies on an external trigger system (hourly tests won't be run hourly if you misconfigure cron schedule).

I didn't write the github-actions-source part yet but it would basically just replace

with: 
    subtest: core

with something like

with
    trigger: hourly

and you just need to set it to valid value. So it is similar to what we already have, just checked at "action validation time". Currently, you should run core hourly but also can misconfigure it :)

[metalwarrior665]

@oklinov How would you handle the "run on PR: true/false"? Something like this and then make the subtest into a list and pass it also for PR action?

└── test
    └── platform
        ├── hourly-and-pr
        ├── daily-and-pr
        ├── daily-without-pr
        └── montly-without-pr

[JuanGalilea]

what if we leave that up to the team? they could use this to configure different suites and then just let them match the triggers with the suites they want in the github workflow file.

We already have ways of grouping tests and ways of doing specific triggers, this lets us marry the two.

[oklinov]

@metalwarrior665 yes, something like that. We could wrap the logic in npm scripts:

{
    "scripts": {
        "test:platform:hourly": "npx vitest test/platform/{hourly-and-pr}",
        "test:platform:daily": "npx vitest test/platform/{daily-and-pr,daily-without-pr}",
        "test:platform:monthly": "npx vitest test/platform/{hourly-and-pr,daily-and-pr,daily-without-pr}",
        "test:platform:pr": "npx vitest test/platform/{hourly-and-pr,daily-and-pr}",
    }
}

or, with @JuanGalilea suggestion using vitest projects:

{
    "scripts": {
        "test:platform:hourly": "npx vitest --project hourly-and-pr",
        "test:platform:daily": "npx vitest --project daily-and-pr --project daily-without-pr",
        "test:platform:monthly": "npx vitest --project monthly-without-pr",
        "test:platform:pr": "npx vitest --project hourly-and-pr --project daily-and-pr",
    }
}

[metalwarrior665]

Looks good. So let's do it on Google Maps first and if it works well, will make issue for everyone.

[oklinov]

another approach:

test/platform/
├── utils.ts
├── google-places.hourly.test.ts          ← runs hourly, PR, daily
├── core.hourly.test.ts                   ← runs hourly, PR, daily
├── google-maps-extractor.pr.test.ts    ← runs PR, daily
├── google-maps-reviews.pr.test.ts      ← runs PR, daily
├── charge-dedup.daily.test.ts          ← runs daily only
├── google-maps-result-counts.daily.test.ts
├── google-maps-email-extractor.daily.test.ts

"test:platform:hourly": "vitest run test/platform/*.hourly.test.ts",
"test:platform:pr":     "vitest run test/platform/*.{hourly,pr}.test.ts",
"test:platform:daily":  "vitest run test/platform/*.{hourly,pr,daily}.test.ts"

I think I like this one the most as it gives you even more flexibility 🤔 @metalwarrior665 @JuanGalilea wdyt?

---

UPDATE: we could even support nesting so it would be possible do structure it like this:

test/platform/
├── google-places/
│   ├── core.hourly.test.ts
│   └── result-counts.daily.test.ts
├── google-maps-reviews/
│   ├── core.hourly.test.ts
│   └── pagination.pr.test.ts
└── utils.ts

[metalwarrior665]

I'm a bigger fan of the folders because they look more readable, the filenames could work but it should be more polished to a standard format since you have several boolean flags for each file/

[oklinov]

what do you mean by "several boolean flags for each file"? 🤔

[metalwarrior665]

Sorry, my bad, you only need one for scheduling (I confused myself by having separate weekly: true, daily: true but in practice we need only one as hourly implies daily etc.)

So yeah, I think this can work. Best practice should be to always specify all flags but with backward compatibility for now. I would make the PR part more explicit: pagination.pr-off.weekly.test.ts vs pagination.pr-on.weekly.test.ts

[metalwarrior665]

Putting this on hold and going forward with simpler solution for now. I will let this sit here for a while so I think about it again once a while.