fix(cmt): add /cmt_dispatch and /cleanup_e2e endpoints

[yasserfaraazkhan]

Problem

CMT Provisioner workflow_dispatch runs on mattermost/desktop and mattermost/mattermost-mobile get the workflow_run.requested webhook delivered to matterwick (verified, 202 OK), but compatibility-matrix-testing.yml is never dispatched as a follow-up. Two recent CMT runs on desktop confirm the symptom: run 25040877733 (2026-04-28) and run 26026866029 (2026-05-18). Mobile has 0 lifetime successful CMT follow-ups.

Root cause

server/workflow_run.go:93 reads payload.WorkflowRun.Inputs[""server_versions""] from the webhook payload:

serverVersionsStr, ok := payload.WorkflowRun.Inputs[""server_versions""]
if !ok || serverVersionsStr == """" {
    logger.Error(""No server_versions found in workflow inputs"")
    return
}

GitHub does not include workflow_dispatch inputs on the workflow_run object in webhook payloads. Verified against three independent sources:

1. Octokit webhook schema — lists 35 fields, none named inputs.
2. The actual REST API representation of run 26026866029 — same 35 fields, no inputs.
3. The official GitHub webhook docs — no inputs in the schema.

So payload.WorkflowRun.Inputs deserializes to an empty map on every CMT delivery; the handler hits the return branch and never reaches dispatchCMTWorkflow. The webhook still 202s (handler returns cleanly), which is why this looked healthy.

The existing unit test workflow_run_test.go:88-105 constructed a fake payload with inputs, masking the issue.

Fix

Move CMT trigger off the webhook path and onto two new authenticated HTTP endpoints. The workflows call matterwick directly, carrying run_id, sha, ref, server_versions, and repo — everything the webhook can't.

Endpoint	Auth header	Called by
POST /cmt_dispatch	X-Trigger-Token	cmt-provisioner.yml in desktop + mobile
POST /cleanup_e2e	X-Cleanup-Token	compatibility-matrix-testing.yml in desktop + mobile

Both endpoints:

• Reject 503 if their secret isn't configured (fail closed)
• Validate token with subtle.ConstantTimeCompare
• Return 202/200 immediately and do the work in a goroutine

The existing handleCMTWithServerVersions is reused without modification.

The broken requested branch in handleWorkflowRunEventWithInputs is removed. The completed branch is kept as a defensive fallback for stuck instances.

What this does NOT touch

• Nightly schedule path
• Master / release-* push path
• PR-label E2E path

Those go through different code paths and (per the merged PR #89) should already be working post-deploy.

Companion PRs

• desktop: https://github.com/mattermost/desktop/pull/new/fix/cmt-direct-dispatch-and-cleanup-endpoint (workflow change to POST /cmt_dispatch)
• mobile: (planned, not yet raised)
• gitops-platform: (planned, not yet raised) — adds CMTTriggerSecret and CleanupSecret to matterwick secret.yaml

Rollout

This PR is safe to land before the workflow PRs:

• Without the secrets in the deployed config, both endpoints return 503, so an accidental early caller fails closed.
• Existing /github_event / /cloud_webhooks / /shrug_wick / / routes are untouched.

Suggested order:

1. Land this PR → no behavior change yet (secrets not deployed).
2. Land + deploy gitops-platform secrets PR with real tokens.
3. Add GitHub repo secrets (MATTERWICK_CMT_TRIGGER_SECRET, rotate MATTERWICK_CLEANUP_SECRET).
4. Roll out matterwick deploy to pick up the new config.
5. Land the desktop workflow PR.
6. Land the mobile workflow PR.
7. End-to-end smoke test by dispatching CMT Provisioner with v11.1.0, v11.2.0.

Testing

• go build ./... — clean
• go vet ./... — clean
• Full existing test suite (go test ./...) — passes, no regressions
• 18 new tests for the two endpoints, all passing:
  • Rejects when secret unconfigured (503)
  • Rejects missing / wrong token (403)
  • Rejects malformed JSON (400)
  • Rejects each missing field (400) including run_id=0, empty server_versions, empty repo
  • Rejects server_versions="",,,"" (parses to empty) (400)
  • Rejects unknown repo type (400)
  • /cleanup_e2e removes matching {repo}-cmt-{run_id}-* keys
  • /cleanup_e2e leaves unrelated keys (e.g. PR instances) untouched
  • /cleanup_e2e handles multiple keys for the same run_id
  • Constant-time comparison rejects same-length wrong tokens

The provisioning-goroutine happy path is not unit-tested here; that path is already covered end-to-end by e2e_dryrun_test.go with mocked cloud + GitHub clients.

Known limitations

1. If matterwick restarts during the ~30 min provisioning goroutine, instances become orphans. The existing periodic cleanupStaleNonPRE2EInstances ticker reaps them by DNS pattern + age. Same behavior as today.
2. CheckLimitRateAndAbortRequest() was deliberately not added to the new endpoints — they don't make GitHub API calls synchronously, so the rate check would block on a network round-trip with no benefit. Inline comment explains this on both handlers.

Test plan

• Reviewer reads new endpoint handlers (server/cmt_dispatch.go, server/cleanup_e2e.go) for auth + parsing correctness
• Reviewer reads the deleted workflow_run.go block and confirms removing the requested branch is safe
• Run go test ./... locally → all pass
• Deploy after secrets land in gitops + repo settings
• Smoke: dispatch CMT Provisioner on desktop → confirm endpoint=/cmt_dispatch log line + matrix workflow dispatched ~30 min later → confirm endpoint=/cleanup_e2e log line at completion → confirm cloud instances destroyed

🤖 Generated with Claude Code

[mm-cloud-bot]

@yasserfaraazkhan: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

I understand the commands that are listed here

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8d79ebee-a14c-4137-ba55-d5f692cb423f

📥 Commits

Reviewing files that changed from the base of the PR and between 1dd004a and d0e3d60.

📒 Files selected for processing (1)

• server/workflow_run.go

---

📝 Walkthrough

Walkthrough

Adds token-protected /cmt_dispatch and /cleanup_e2e HTTP endpoints, two config secrets, tests for handler validation, and refactors workflow_run handling to remove provisioning triggers and rely on SHA-based cleanup.

Changes

CMT Provisioning and Cleanup Flow

Layer / File(s)	Summary
Configuration and Route Registration server/config.go, server/server.go	Configuration adds CMTTriggerSecret and CleanupSecret fields; router registers POST /cmt_dispatch and /cleanup_e2e with comments documenting webhook sources.
CMT Dispatch Endpoint server/cmt_dispatch.go, server/cmt_dispatch_test.go	New /cmt_dispatch handler validates X-Trigger-Token, parses CMTDispatchRequest (owner/repo/sha/ref/run_id/server_versions), derives instance type from repo name, asynchronously provisions via handleCMTWithServerVersions, and returns HTTP 202. Tests cover authentication, JSON validation, field validation, server version parsing, and repo type detection.
E2E Cleanup Endpoint server/cleanup_e2e.go, server/cleanup_e2e_test.go	New /cleanup_e2e handler validates X-Cleanup-Token, parses CleanupE2ERequest (repo/run_id), locates and removes matching CMT tracking keys from e2eInstances map under lock, asynchronously destroys instances, and returns HTTP 200 with status. Tests cover authentication, JSON validation, field validation, key removal semantics, and token comparison.
Workflow Handler Refactoring server/workflow_run.go	Removes CMT provisioning from workflow_run "requested" path; now only uses completed action as a defensive cleanup fallback and always performs SHA-based cleanup for E2E workflows.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• mattermost/matterwick#89: Modifies handleWorkflowRunEventWithInputs cleanup behavior between mw_tracking_key-based cleanup and SHA-based destruction.

Suggested labels

release-note-none

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 8.70% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: adding two new HTTP endpoints (/cmt_dispatch and /cleanup_e2e) to fix the CMT provisioning workflow.
Description check	✅ Passed	The description is comprehensive and directly related to the changeset, thoroughly explaining the problem, root cause, fix, testing, and deployment strategy.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/cmt-direct-dispatch-and-cleanup-endpoint

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (2)

server/cmt_dispatch_test.go (1)

47-57: 💤 Low value

Consider using httptest.NewRequestWithContext to satisfy linter.

Static analysis flags httptest.NewRequest usage. While functionally equivalent in tests, using NewRequestWithContext aligns with best practices and silences the linter.

♻️ Suggested fix

 func doCMTDispatchRequest(t *testing.T, s *Server, body, token string) *httptest.ResponseRecorder {
 	t.Helper()
-	req := httptest.NewRequest(http.MethodPost, "/cmt_dispatch", bytes.NewBufferString(body))
+	req := httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/cmt_dispatch", bytes.NewBufferString(body))
 	req.Header.Set("Content-Type", "application/json")
 	if token != "" {
 		req.Header.Set("X-Trigger-Token", token)
 	}
 	w := httptest.NewRecorder()
 	s.handleCMTDispatch(w, req)
 	return w
 }

You'll need to add "context" to the imports.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@server/cmt_dispatch_test.go` around lines 47 - 57, The test helper
doCMTDispatchRequest uses httptest.NewRequest which the linter flags; update
doCMTDispatchRequest to use httptest.NewRequestWithContext with a background
context (e.g., context.Background()) when creating the request, and add
"context" to the imports; specifically modify the call in doCMTDispatchRequest
(the NewRequest call that constructs req) to NewRequestWithContext and ensure
the X-Trigger-Token header handling and subsequent call to s.handleCMTDispatch
remain unchanged.

server/cleanup_e2e_test.go (1)

35-45: 💤 Low value

Consider using httptest.NewRequestWithContext to satisfy linter.

Same as in cmt_dispatch_test.go, static analysis flags httptest.NewRequest. Using NewRequestWithContext would align with the linter expectations.

♻️ Suggested fix

 func doCleanupE2ERequest(t *testing.T, s *Server, body, token string) *httptest.ResponseRecorder {
 	t.Helper()
-	req := httptest.NewRequest(http.MethodPost, "/cleanup_e2e", bytes.NewBufferString(body))
+	req := httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/cleanup_e2e", bytes.NewBufferString(body))
 	req.Header.Set("Content-Type", "application/json")
 	if token != "" {
 		req.Header.Set("X-Cleanup-Token", token)
 	}
 	w := httptest.NewRecorder()
 	s.handleCleanupE2E(w, req)
 	return w
 }

You'll need to add "context" to the imports.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@server/cleanup_e2e_test.go` around lines 35 - 45, Replace httptest.NewRequest
with httptest.NewRequestWithContext in doCleanupE2ERequest and pass a context
(e.g., context.Background()); also add "context" to the imports. Specifically,
update the call in doCleanupE2ERequest (the helper that builds the POST to
"/cleanup_e2e") to use httptest.NewRequestWithContext(context.Background(),
http.MethodPost, ... ) and ensure the package imports include "context".

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@server/cleanup_e2e_test.go`:
- Around line 35-45: Replace httptest.NewRequest with
httptest.NewRequestWithContext in doCleanupE2ERequest and pass a context (e.g.,
context.Background()); also add "context" to the imports. Specifically, update
the call in doCleanupE2ERequest (the helper that builds the POST to
"/cleanup_e2e") to use httptest.NewRequestWithContext(context.Background(),
http.MethodPost, ... ) and ensure the package imports include "context".

In `@server/cmt_dispatch_test.go`:
- Around line 47-57: The test helper doCMTDispatchRequest uses
httptest.NewRequest which the linter flags; update doCMTDispatchRequest to use
httptest.NewRequestWithContext with a background context (e.g.,
context.Background()) when creating the request, and add "context" to the
imports; specifically modify the call in doCMTDispatchRequest (the NewRequest
call that constructs req) to NewRequestWithContext and ensure the
X-Trigger-Token header handling and subsequent call to s.handleCMTDispatch
remain unchanged.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 7b08b625-de55-47f7-b1f4-061630a8a168

📥 Commits

Reviewing files that changed from the base of the PR and between 68ce3b7 and 1dd004a.

📒 Files selected for processing (7)

• server/cleanup_e2e.go
• server/cleanup_e2e_test.go
• server/cmt_dispatch.go
• server/cmt_dispatch_test.go
• server/config.go
• server/server.go
• server/workflow_run.go