Skip to content

feat(core): catalog plugin resource events #1191

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dcramer merged 1 commit into from
Aug 2, 2026
Merged

feat(core): catalog plugin resource events #1191

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions TERMINOLOGY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@ Canonical words used across Junior's code and documentation.
- **Source**: where an inbound event came from, such as Slack, local CLI,
scheduler, or plugin dispatch.
- **Destination**: where Junior sends output or side effects.
- **Resource event**: one normalized change published by a plugin, identified
by namespace, identifier, event type, and an idempotency key.
- **Resource subscription**: a temporary conversation association that delivers
matching resource events back into that conversation.
- **Inbound message**: one normalized source event made available to the agent.
- **Agent input**: the inbound content, context, and runtime metadata selected
for a turn.
Expand Down
52 changes: 29 additions & 23 deletions packages/docs/src/content/docs/extend/github-plugin.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,16 +46,16 @@ You can optionally declare `appPermissions` when registering the plugin. Junior

Set these values in the host environment:

| Variable | Required | Purpose |
| -------------------------- | -------- | ------------------------------------------------------------------- |
| `GITHUB_APP_ID` | Yes | GitHub App identity. |
| `GITHUB_APP_CLIENT_ID` | Yes | GitHub App OAuth client id for user-token auth. |
| `GITHUB_APP_CLIENT_SECRET` | Yes | GitHub App OAuth client secret for user-token auth. |
| `GITHUB_APP_PRIVATE_KEY` | Yes | GitHub App signing key. |
| `GITHUB_INSTALLATION_ID` | Yes | Repository or organization installation target. |
| `GITHUB_APP_BOT_NAME` | Yes | Git author and committer display name. |
| `GITHUB_APP_BOT_EMAIL` | Yes | App bot noreply email used for Git attribution and work ownership. |
| `GITHUB_WEBHOOK_SECRET` | No | Webhook signing secret for deployment and PR watches and reporting. |
| Variable | Required | Purpose |
| -------------------------- | -------- | ---------------------------------------------------------------------- |
| `GITHUB_APP_ID` | Yes | GitHub App identity. |
| `GITHUB_APP_CLIENT_ID` | Yes | GitHub App OAuth client id for user-token auth. |
| `GITHUB_APP_CLIENT_SECRET` | Yes | GitHub App OAuth client secret for user-token auth. |
| `GITHUB_APP_PRIVATE_KEY` | Yes | GitHub App signing key. |
| `GITHUB_INSTALLATION_ID` | Yes | Repository or organization installation target. |
| `GITHUB_APP_BOT_NAME` | Yes | Git author and committer display name. |
| `GITHUB_APP_BOT_EMAIL` | Yes | App bot noreply email used for Git attribution and work ownership. |
| `GITHUB_WEBHOOK_SECRET` | No | Webhook signing secret for deployment, pull request, and issue events. |

`GITHUB_INSTALLATION_ID` selects the GitHub App installation for the deployment.
`GITHUB_APP_BOT_EMAIL` uses the GitHub noreply format
Expand Down Expand Up @@ -129,9 +129,11 @@ Git smart-HTTP push classification is repository-scoped, not branch-scoped. It d
If your team works across multiple repositories, have users include `owner/repo` in their GitHub request whenever the target is not obvious from the conversation.
That only helps when those repositories are covered by the same GitHub App installation ID.

## Watch pull request events
## Watch pull request and issue events

When `GITHUB_WEBHOOK_SECRET` is configured, `github_createPullRequest` and `github_getPullRequest` return a subscribable pull request resource. Junior can then subscribe the current Slack conversation to high-signal PR events for either a newly created or existing PR when watching it serves the user's request.
When `GITHUB_WEBHOOK_SECRET` is configured, GitHub tools can return subscribable
pull request and issue resources. Junior can temporarily watch those resources
and send matching updates back to the current Slack thread.

Subscribed events run headlessly as Junior's `resource-event` system actor.
They can use repository-scoped installation credentials to commit and push a
Expand All @@ -141,15 +143,19 @@ review, still require explicit delegated user authorization.

Supported GitHub webhook deliveries become these Junior resource events:

| GitHub delivery | Junior event types |
| ------------------------------------- | ----------------------------------------------------------------- |
| `check_suite` completed | `checks.failed`, `checks.recovered` |
| `issue_comment` created on a PR | `comment.created` |
| `pull_request_review` submitted | `review.approved`, `review.changes_requested`, `review.commented` |
| `pull_request_review_comment` created | `review_comment.created` |
| `pull_request` closed | `state.merged`, `state.closed_unmerged` |
| GitHub delivery | Junior event types |
| ------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `check_suite` completed | `pull_request.checks.failed`, `pull_request.checks.recovered` |
| `issue_comment` created on a PR | `pull_request.comment.created` |
| `pull_request_review` submitted | `pull_request.review.approved`, `pull_request.review.changes_requested`, `pull_request.review.commented` |
| `pull_request_review_comment` created | `pull_request.review_comment.created` |
| `pull_request` closed | `pull_request.merged`, `pull_request.closed_unmerged` |
| `issues` opened, closed, or reopened | `issue.opened`, `issue.closed`, `issue.reopened` |
| `issue_comment` created on an issue | `issue.comment.created` |

`state.merged` and `state.closed_unmerged` complete the subscription after Junior accepts the event. Other supported events keep the watch active until the subscription expires, is cancelled, or reaches its configured TTL.
`pull_request.merged` and `pull_request.closed_unmerged` complete a
temporary pull request watch after Junior accepts the event. Other events keep
the watch active until it expires or is cancelled.

Webhook events are delivered as normal queued conversation messages. They do not interrupt active work, bypass Slack routing, or act as user-authored commands. Junior uses the subscription intent to decide whether to reply, take a follow-up action, or stay silent.

Expand Down Expand Up @@ -245,7 +251,7 @@ expected follow-up in the original conversation.
- The host-side lease is bounded by the sandbox session and token expiry. It is not exposed as reusable long-lived auth inside the sandbox.
- GitHub webhooks are accepted only when the `X-Hub-Signature-256` header matches `GITHUB_WEBHOOK_SECRET`.
- Resource event subscriptions are conversation-scoped. Core owns subscription records, dedupe, TTL, and mailbox delivery; the GitHub plugin owns signature verification, provider normalization, and its pull request and issue outcome projections.
- Resource-event turns do not inherit a subscriber's user credential. Bot-owned issue, pull request, and smart-HTTP push operations use scoped installation credentials; human-owned operations still enter the normal authorization flow.
- Resource-watch turns do not inherit a subscriber's user credential. Bot-owned issue, pull request, and smart-HTTP push operations use scoped installation credentials; human-owned operations still enter the normal authorization flow.
- The write boundary is the App installation scope, the single-repository token scope, and Junior's endpoint allowlist. `appPermissions` declarations do not narrow write tokens.

## Failure modes
Expand All @@ -256,8 +262,8 @@ expected follow-up in the original conversation.
- `github_getDeployment` returns `403`: grant the GitHub App `Deployments: read`, approve the updated permission on its installation, and retry.
- Deployment metadata is available but Junior never offers to watch it: `GITHUB_WEBHOOK_SECRET` is missing. Set it, redeploy, and run `github_getDeployment` again.
- GitHub webhook delivery returns `401`: the webhook secret in GitHub App settings does not match `GITHUB_WEBHOOK_SECRET`, or GitHub did not send `X-Hub-Signature-256`. Update the app webhook secret and retry the delivery.
- GitHub webhook delivery returns `202 Ignored`: the delivery was signed correctly but does not map to a supported deployment or PR watch event. Use one of the configured event types above.
- GitHub delivery succeeds but no Slack follow-up appears: confirm the original conversation has an active resource event subscription for that PR and event type. A successful webhook alone does not create a subscription.
- GitHub webhook delivery returns `202 Ignored`: the delivery was signed correctly but does not map to a supported deployment, pull request, or issue event. Use one of the configured event types above.
- GitHub delivery succeeds but no Slack follow-up appears: confirm the original conversation has an active resource watch for that identifier and event type. A successful webhook alone does not create a watch.
- Missing repository context: Junior could not determine which repository to use. Include `owner/repo` directly in the GitHub request, or configure a default GitHub repository for that thread, and retry.
- A `403` response that says to use `github_createIssue` or `github_createPullRequest` is a Junior routing denial, not evidence of missing App permissions. Retry with the named tool.
- Private OAuth prompt for a human-identity operation such as a pull request review: the actor has not authorized the GitHub App yet, or the stored user-to-server token expired. Complete the private authorization prompt; do not paste personal access tokens into the chat or sandbox.
Expand Down
82 changes: 58 additions & 24 deletions packages/junior-evals/evals/agent/subscriptions.eval.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ describeEval("Resource Event Subscriptions", slackEvals, (it) => {
criteria: rubric({
pass: [
"The reply confirms the exact deployment target will be monitored through event-based updates.",
"The reply makes clear that the watch is temporary and says when it expires.",
],
fail: [
"Do not claim a polling task or recurring schedule was created.",
Expand All @@ -41,22 +42,24 @@ describeEval("Resource Event Subscriptions", slackEvals, (it) => {
expect.arrayContaining([
expect.objectContaining({
name: "github_getDeployment",
status: "ok",
arguments: {
commitSha,
environment: "Production",
repo: "getsentry/junior-prod",
},
}),
expect.objectContaining({
name: "subscribeToResourceEvents",
name: "watchResourceEvents",
status: "ok",
arguments: expect.objectContaining({
events: expect.arrayContaining([
"deployment.succeeded",
"deployment.failed",
"deployment.error",
]),
provider: "github",
resourceRef: `github:deployment-source:getsentry/junior-prod:production:${commitSha}`,
namespace: "github",
identifier: `deployment-source:getsentry/junior-prod:production:${commitSha}`,
resourceType: "deployment_source",
}),
}),
Expand All @@ -72,16 +75,19 @@ describeEval("Resource Event Subscriptions", slackEvals, (it) => {
}) => {
const result = await run({
overrides: {
github_resource_events: true,
plugin_dirs: ["fixtures/resource-event-plugins"],
plugin_packages: ["@sentry/junior-github"],
},
initialEvents: [
mention(
"/eval-resource-events Use the provider to create a pull request titled 'Prefer event subscriptions', then check it every five minutes and tell this thread if checks fail, review feedback arrives, it merges, or it closes.",
"$eval-resource-events Use the provider to create a pull request titled 'Prefer event subscriptions', then check it every five minutes and tell this thread if checks fail, review feedback arrives, it merges, or it closes.",
),
],
criteria: rubric({
pass: [
"The reply confirms the pull request will be monitored through event-based updates for the requested outcomes.",
"The reply makes clear that the watch is temporary and says when it expires.",
],
fail: [
"Do not ask the user to monitor GitHub manually.",
Expand All @@ -94,6 +100,7 @@ describeEval("Resource Event Subscriptions", slackEvals, (it) => {
expect.arrayContaining([
expect.objectContaining({
name: "callMcpTool",
status: "ok",
arguments: expect.objectContaining({
tool_name:
"mcp__eval-resource-events__create-watchable-pull-request",
Expand All @@ -103,26 +110,27 @@ describeEval("Resource Event Subscriptions", slackEvals, (it) => {
}),
}),
expect.objectContaining({
name: "subscribeToResourceEvents",
name: "watchResourceEvents",
status: "ok",
arguments: expect.objectContaining({
provider: "github",
resourceRef: "github:pull_request:getsentry/junior#208",
namespace: "github",
identifier: "getsentry/junior#208",
resourceType: "pull_request",
events: expect.arrayContaining([
"checks.failed",
"review.changes_requested",
"review.commented",
"review_comment.created",
"state.merged",
"state.closed_unmerged",
"pull_request.checks.failed",
"pull_request.review.changes_requested",
"pull_request.review.commented",
"pull_request.review_comment.created",
"pull_request.merged",
"pull_request.closed_unmerged",
]),
}),
result: expect.objectContaining({
stop_watching: {
execution_tool: "executeTool",
execution_example: {
tool_name: "stopWatchingResources",
arguments: {},
arguments: { id: expect.stringMatching(/^resub_/) },
},
},
}),
Expand All @@ -134,7 +142,7 @@ describeEval("Resource Event Subscriptions", slackEvals, (it) => {
);
});

it("when a follow-up stops monitoring, cancel the conversation's subscriptions before confirming", async ({
it("when a follow-up stops monitoring, cancel the selected watch before confirming", async ({
run,
}) => {
const thread = {
Expand All @@ -144,11 +152,13 @@ describeEval("Resource Event Subscriptions", slackEvals, (it) => {
};
const result = await run({
overrides: {
github_resource_events: true,
plugin_dirs: ["fixtures/resource-event-plugins"],
plugin_packages: ["@sentry/junior-github"],
},
initialEvents: [
mention(
"/eval-resource-events Create a pull request titled 'Stop resource monitoring', watch its checks and review feedback, and keep me posted here.",
"$eval-resource-events Create a pull request titled 'Stop resource monitoring', watch its checks and review feedback, and keep me posted here.",
{ thread },
),
],
Expand All @@ -165,16 +175,40 @@ describeEval("Resource Event Subscriptions", slackEvals, (it) => {
}),
});

expect(toolCalls(result.session)).toEqual(
const calls = toolCalls(result.session);
expect(calls).toEqual(
expect.arrayContaining([
expect.objectContaining({
name: "subscribeToResourceEvents",
name: "watchResourceEvents",
status: "ok",
}),
expect.objectContaining({
name: "stopWatchingResources",
status: "ok",
}),
]),
);
const watch = calls.find((call) => call.name === "watchResourceEvents");
const stop = calls.find((call) => call.name === "stopWatchingResources");
if (!watch || watch.status !== "ok") {
throw new Error("Expected a successful resource watch tool call");
}
if (
!watch.result ||
typeof watch.result !== "object" ||
Array.isArray(watch.result) ||
typeof watch.result.id !== "string"
) {
throw new Error("Resource watch result did not contain an id");
}
expect(stop).toMatchObject({
arguments: { id: watch.result.id },
result: {
stoppedIds: [watch.result.id],
watching_status: "stopped",
},
status: "ok",
});
});

it("when a subscribed PR check fails, summarize the failure and suggest next steps", async ({
Expand All @@ -184,11 +218,11 @@ describeEval("Resource Event Subscriptions", slackEvals, (it) => {
initialEvents: [
resourceEventNotification({
eventKey: "github-delivery-checks-failed",
eventType: "checks.failed",
eventType: "pull_request.checks.failed",
intent:
"Watch the pull request Junior opened for CI failures before review.",
label: "GitHub PR getsentry/junior#691",
resourceRef: "github:pull_request:getsentry/junior#691",
identifier: "getsentry/junior#691",
trustedSummary:
'GitHub PR getsentry/junior#691 checks failed on workflow "test" for commit abcdef123456.',
}),
Expand Down Expand Up @@ -222,11 +256,11 @@ describeEval("Resource Event Subscriptions", slackEvals, (it) => {
githubWebhook({
eventName: "check_suite",
subscription: {
events: ["checks.recovered"],
events: ["pull_request.checks.recovered"],
intent:
"Let the original Slack thread know when Junior's pull request lands.",
label: "GitHub PR getsentry/junior#702",
resourceRef: "github:pull_request:getsentry/junior#702",
identifier: "getsentry/junior#702",
resourceType: "pull_request",
},
body: {
Expand All @@ -251,11 +285,11 @@ describeEval("Resource Event Subscriptions", slackEvals, (it) => {
initialEvents: [
resourceEventNotification({
eventKey: "github-delivery-pr-merged",
eventType: "state.merged",
eventType: "pull_request.merged",
intent:
"Let the original Slack thread know when Junior's pull request lands.",
label: "GitHub PR getsentry/junior#702",
resourceRef: "github:pull_request:getsentry/junior#702",
identifier: "getsentry/junior#702",
trustedSummary: "GitHub PR getsentry/junior#702 was merged.",
}),
],
Expand Down
4 changes: 2 additions & 2 deletions packages/junior-evals/evals/conversation/routing.eval.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ describeEval("Conversation Routing", slackEvals, (it) => {
eventType: "issue.linked",
intent: "Track linked infrastructure work in this Slack thread.",
label: "Linear issue OPS-123",
provider: "linear",
resourceRef: "linear:issue:OPS-123",
namespace: "linear",
identifier: "OPS-123",
thread: steeringThread,
trustedSummary: "Linear issue OPS-123 was linked to this thread.",
}),
Expand Down
Loading
Loading