fix(telemetry): name the orchestrator's blocked-drain guard, fingerprint aborts - #1013
Draft
posthog[bot] wants to merge 1 commit into
Draft
fix(telemetry): name the orchestrator's blocked-drain guard, fingerprint aborts#1013posthog[bot] wants to merge 1 commit into
posthog[bot] wants to merge 1 commit into
Conversation
…int aborts The orchestrator's post-drain guard raised `orchestrator drain ended with failed tasks` on a drain where nothing failed but tasks stayed blocked behind a dependency that never completed. The user-facing copy already said "N steps never ran", so only the exception misnamed itself — and it sent triage looking for a failure the queue state does not contain. Split the decision into a small pure `drainFailure()` (mirroring the pi harness's `completionFailure()`) so each cause carries its own message, and fix the "1 steps" pluralisation while we're on the line. Separately, every abort's captured exception was fingerprinted server-side from its stack trace, which carries the install path — pnpm dlx content hashes, `.npm/_npx` dirs, CI checkouts, local clones. One cause therefore shattered into an issue per install, each reading "1 occurrence, 1 user, 1 session". Aborts are raised from a fixed set of guard sites, so `wizardAbort` now supplies a stable message-derived `$exception_fingerprint`, following the pattern already proven on the OAuth flow. A callsite that wants different grouping can still set the property in its `WizardError` context. Verified the derivation against the real message corpus: it collapses the fragmented groups, keeps the two orchestrator branches apart, and leaves distinct API error codes distinguished. Generated-By: PostHog Code Task-Id: 86e6e098-a5ce-4173-8ba6-a04e9c62813a
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The wizard's abort telemetry is misreporting itself in two ways, both on the core onboarding path where a failure matters most.
The orchestrator's post-drain guard raises
WizardError('orchestrator drain ended with failed tasks')whenever tasks failed or tasks stayed blocked. On the blocked-only branch nothing actually failed — the user is correctly told "N steps never ran", but the exception name sends triage looking for a failure the queue state doesn't contain.Separately, aborts are fingerprinted server-side from the stack trace, which carries the install path (pnpm dlx content hashes,
.npm/_npxdirs, CI checkouts, local clones). One cause therefore shatters into an issue per install, each reading "1 occurrence, 1 user, 1 session" and looking like a harmless one-off. This isn't limited to the orchestrator:Authentication failedalone is spread across hundreds of separate issues, and the orchestrator drain abort across more than a dozen.Neither change fixes the underlying agent failure — that's the point. They make the real recurrence rate visible so we can judge whether the agent is genuinely failing to finish work or a guard is simply too aggressive.
Changes
drainFailure()(mirroring the pi harness's existingcompletionFailure()) decides which cause ended the drain:orchestrator drain ended with failed taskswhen retries were exhausted,orchestrator drain ended with tasks that never ranwhen onlyPendingtasks remain. The properties that tell them apart were already being captured, so this is naming and branching, not new instrumentation. Also fixes the "1 steps never ran" pluralisation on the same line.wizardAbortnow supplies a message-derived$exception_fingerprint, following the pattern already proven on the OAuth flow. Truncation is deliberate — it folds together messages that share a cause but differ in their tail. A callsite that wants different grouping still wins by setting the property in itsWizardErrorcontext.Test plan
pnpm build && pnpm test && pnpm lint— 1640 tests pass, 0 lint errors.drain-failure.test.tsasserts the blocked-only branch no longer calls itself a failure;wizard-abort.test.tsasserts the same cause raised from an_npxpath and apnpm dlxpath fingerprints identically, the two guard branches stay apart, and a context-supplied fingerprint overrides the default.Created with PostHog Desktop from this inbox report.