Skip to content

Commit e44cd43

Browse files
committed
fix(orchestration): idempotency-claim parent-epic comment handling — stop webhook-redelivery reply spam (aws-samples#247 UX.20)
CRITICAL live-caught: a no-match @bgagent comment on the parent epic spammed 50+ duplicate disambiguation replies. Root cause: the UX.18 parent-comment handler posts its reply (and 👀) with NO idempotency guard, and Linear REDELIVERS a comment webhook whenever the handler exceeds its ~5s ack window (this path does several Linear API calls and ran 7.7s). Each redelivery re-ran the whole handler → another reply. The iteration path is deduped by ack_replied_at; the disambiguation path I added in UX.18 had nothing. (Mitigated live by throttling the Linear receiver+processor Lambdas to 0 concurrency; 49 spam comments deleted.) Fix: new store op claimCommentAck — a conditional create-once write keyed on (orchestration_id, 'ack#<commentId>') with a TTL, mirroring claimRollup. handleParentEpicCommentTrigger claims BEFORE any side-effect; only the first delivery proceeds (👀 + match/route/reply), redeliveries no-op. The marker self-expires via the table's existing ttl attribute. loadOrchestration now excludes any 'ack#…' (and any '<kind>#' marker) SK from the children list so the dedup rows can't be mistaken for sub-issues (real child SKs are UUIDs or '…__integration', never contain '#'). Tests: claimCommentAck (first-wins / redelivery-loses / error-propagates); loadOrchestration excludes ack# rows; webhook 'redelivery posts EXACTLY ONE reply' + 'matched iteration dedups to one task'. 58 green across store+webhook suites.
1 parent 5f08628 commit e44cd43

4 files changed

Lines changed: 155 additions & 2 deletions

File tree

cdk/src/handlers/linear-webhook-processor.ts

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ import { resolveLinearOauthToken } from './shared/linear-oauth-resolver';
2727
import { logger } from './shared/logger';
2828
import { discoverOrchestration } from './shared/orchestration-discovery';
2929
import { readConcurrencyBudget, releaseReadyChildren } from './shared/orchestration-release';
30-
import { deriveOrchestrationId, loadOrchestration, setStatusCommentId, type OrchestrationReleaseContext } from './shared/orchestration-store';
30+
import { claimCommentAck, deriveOrchestrationId, loadOrchestration, setStatusCommentId, type OrchestrationReleaseContext } from './shared/orchestration-store';
3131
import { buildIterationInstruction, parseCommentTrigger, type CommentTrigger } from './shared/orchestration-comment-trigger';
3232
import { parseParentNodeReference, renderParentDisambiguationReply, suggestClosestNode } from './shared/orchestration-parent-comment';
3333
import { fetchIssueParentId } from './shared/linear-subissue-fetch';
@@ -48,6 +48,12 @@ const DEFAULT_LABEL_FILTER = 'bgagent';
4848
// budget. Unset → release all roots (back-compat; admission still gates).
4949
const USER_CONCURRENCY_TABLE = process.env.USER_CONCURRENCY_TABLE_NAME;
5050
const MAX_CONCURRENT = Number(process.env.MAX_CONCURRENT_TASKS_PER_USER ?? '10');
51+
/**
52+
* TTL (seconds) for the per-comment ack-claim marker (#247 UX.20). Only needs
53+
* to outlive Linear's webhook redelivery window (minutes), but we keep a day of
54+
* slack so a delayed redelivery still dedups; the row self-expires after.
55+
*/
56+
const ACK_CLAIM_TTL_SECONDS = 86_400;
5157

5258
/**
5359
* Post a Linear comment + ❌ reaction without ever propagating an error.
@@ -720,6 +726,27 @@ async function handleParentEpicCommentTrigger(args: {
720726
}): Promise<void> {
721727
const { orchestrationId, snapshot, workspaceId, commentId, replyTargetId, trigger, resolved, registryTableName } = args;
722728
const feedbackCtx = { linearWorkspaceId: workspaceId, registryTableName };
729+
730+
// #247 UX.20: claim-once BEFORE any side-effect. Linear redelivers a comment
731+
// webhook when the handler exceeds its ~5s ack window (this path does several
732+
// Linear API calls and can run >5s), and EACH redelivery would otherwise
733+
// re-react + re-post the disambiguation reply — live-caught spamming 50+
734+
// duplicate replies. The conditional claim (keyed on this comment id) lets
735+
// only the FIRST delivery proceed; redeliveries no-op here. The marker
736+
// self-expires via the table TTL. (The iterate path also has its own
737+
// createTaskCore idempotency key — this is the outer guard that also covers
738+
// the 👀 + the ask-reply, which have no other dedup.)
739+
const ttlEpochSeconds = Math.floor(Date.now() / 1000) + ACK_CLAIM_TTL_SECONDS;
740+
const won = await claimCommentAck(
741+
ddb, ORCHESTRATION_TABLE!, orchestrationId, commentId, new Date().toISOString(), ttlEpochSeconds,
742+
);
743+
if (!won) {
744+
logger.info('A6 comment (parent epic): redelivery — already handled this comment, skipping', {
745+
orchestration_id: orchestrationId, comment_id: commentId,
746+
});
747+
return;
748+
}
749+
723750
// ACK immediately — a parent comment is never silently dropped again.
724751
await reactToComment(feedbackCtx, commentId, EMOJI_STARTED);
725752

cdk/src/handlers/shared/orchestration-store.ts

Lines changed: 44 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -458,6 +458,45 @@ export async function clearRollupClaim(
458458
}));
459459
}
460460

461+
/**
462+
* Claim the one-time "I responded to this comment" marker so a webhook
463+
* REDELIVERY doesn't re-post (#247 UX.20 — live-caught spam). Linear redelivers
464+
* a comment webhook when the handler exceeds its ~5s ack window; without a
465+
* claim, the parent-epic disambiguation reply re-posted on every redelivery
466+
* (50+ duplicates). Keyed on the orchestration + the triggering comment id, so
467+
* the FIRST delivery wins and every redelivery is a no-op. The marker carries a
468+
* TTL (the table's ``ttl`` attribute) so these rows self-expire — they're only
469+
* needed for the redelivery window. Returns true only for the first caller.
470+
*
471+
* @param ttlEpochSeconds absolute epoch-seconds expiry for the marker row.
472+
*/
473+
export async function claimCommentAck(
474+
ddb: DynamoDBDocumentClient,
475+
tableName: string,
476+
orchestrationId: string,
477+
commentId: string,
478+
now: string,
479+
ttlEpochSeconds: number,
480+
): Promise<boolean> {
481+
try {
482+
await ddb.send(new UpdateCommand({
483+
TableName: tableName,
484+
Key: { orchestration_id: orchestrationId, sub_issue_id: `ack#${commentId}` },
485+
// attribute_not_exists on the PK is the standard "create-once" guard —
486+
// a replay finds the row present and the condition fails. ``ttl`` is a
487+
// DynamoDB reserved keyword → must be aliased via ExpressionAttributeNames.
488+
UpdateExpression: 'SET acked_at = :now, #ttl = :ttl',
489+
ConditionExpression: 'attribute_not_exists(orchestration_id)',
490+
ExpressionAttributeNames: { '#ttl': 'ttl' },
491+
ExpressionAttributeValues: { ':now': now, ':ttl': ttlEpochSeconds },
492+
}));
493+
return true;
494+
} catch (err) {
495+
if ((err as { name?: string })?.name === 'ConditionalCheckFailedException') return false;
496+
throw err;
497+
}
498+
}
499+
461500
/** Sort-key of the parent-meta row. Exported so the reconciler can
462501
* separate it from child rows after a Query. */
463502
export const ORCHESTRATION_META_SK = PARENT_META_SK;
@@ -532,7 +571,11 @@ export async function loadOrchestration(
532571
}
533572

534573
const children = items
535-
.filter((i) => i.sub_issue_id !== PARENT_META_SK)
574+
// Exclude the meta row AND non-child marker rows (e.g. ``ack#<commentId>``
575+
// dedup markers, #247 UX.20) — only real sub-issue rows are children.
576+
// A real child SK is a Linear issue UUID or the ``…__integration`` synthetic
577+
// id; markers use a ``<kind>#`` prefix that no real SK has.
578+
.filter((i) => i.sub_issue_id !== PARENT_META_SK && !String(i.sub_issue_id).includes('#'))
536579
.map((i) => i as unknown as OrchestrationChildRow);
537580

538581
const meta: OrchestrationMeta = {

cdk/test/handlers/linear-webhook-processor-orchestration.test.ts

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -504,7 +504,20 @@ describe('linear-webhook-processor — #247 A6 comment trigger', () => {
504504
repo: 'o/r', parent_linear_issue_id: parentIssueId, linear_workspace_id: 'WS',
505505
linear_identifier: 'ABCA-306', title: 'Add a newsletter signup section', child_task_id: 'task-news',
506506
};
507+
// Stateful ack-claim (#247 UX.20): the conditional Update on ack#<comment>
508+
// succeeds the FIRST time and ConditionalCheckFailed on every redelivery.
509+
const claimedAcks = new Set<string>();
507510
ddbSend.mockImplementation(async (cmd: { _type: string; input: Record<string, unknown> }) => {
511+
if (cmd._type === 'Update') {
512+
const sk = (cmd.input.Key as { sub_issue_id?: string })?.sub_issue_id ?? '';
513+
if (sk.startsWith('ack#')) {
514+
if (claimedAcks.has(sk)) {
515+
throw Object.assign(new Error('claim exists'), { name: 'ConditionalCheckFailedException' });
516+
}
517+
claimedAcks.add(sk);
518+
}
519+
return {};
520+
}
508521
if (cmd._type === 'Query' && cmd.input.IndexName === 'LinearIssueIndex') return { Items: [] };
509522
if (cmd._type === 'Query') return { Items: [meta, footer, news] }; // loadOrchestration (parent's own)
510523
if (cmd._type === 'Get') {
@@ -575,5 +588,26 @@ describe('linear-webhook-processor — #247 A6 comment trigger', () => {
575588
expect(createTaskCoreMock).not.toHaveBeenCalled();
576589
expect(replyToCommentMock).toHaveBeenCalledTimes(1);
577590
});
591+
592+
test('#247 UX.20: webhook REDELIVERY of the same parent comment posts EXACTLY ONE reply (no spam)', async () => {
593+
mockParentEpic('PARENT-EPIC');
594+
const evt = eventWith(parentComment('@bgagent looks great, ship it', 'pc-dup'));
595+
// Linear redelivers the same comment webhook 3× (handler exceeded its ack window).
596+
await handler(evt);
597+
await handler(evt);
598+
await handler(evt);
599+
// The conditional ack-claim lets only the FIRST delivery act: one 👀, one reply.
600+
expect(replyToCommentMock).toHaveBeenCalledTimes(1);
601+
expect(reactToCommentMock).toHaveBeenCalledTimes(1);
602+
});
603+
604+
test('#247 UX.20: a matched-iteration parent comment also dedups under redelivery (one task, one ack)', async () => {
605+
mockParentEpic('PARENT-EPIC');
606+
const evt = eventWith(parentComment('@bgagent for the footer change the tagline', 'pc-iter'));
607+
await handler(evt);
608+
await handler(evt);
609+
expect(createTaskCoreMock).toHaveBeenCalledTimes(1); // one iteration, not two
610+
expect(reactToCommentMock).toHaveBeenCalledTimes(1);
611+
});
578612
});
579613
});

cdk/test/handlers/shared/orchestration-store.test.ts

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,8 @@ import {
2424
deriveOrchestrationId,
2525
claimRollup,
2626
clearRollupClaim,
27+
claimCommentAck,
28+
loadOrchestration,
2729
findOrchestrationChildByBranch,
2830
} from '../../../src/handlers/shared/orchestration-store';
2931
import type { SubIssueNode } from '../../../src/handlers/shared/linear-subissue-fetch';
@@ -290,6 +292,53 @@ describe('clearRollupClaim — release the claim so a re-completing epic re-sett
290292
});
291293
});
292294

295+
describe('claimCommentAck — exactly-once per comment (#247 UX.20 redelivery dedup)', () => {
296+
test('first delivery wins → true, conditional create-once on a per-comment SK + TTL', async () => {
297+
const ddb = { send: jest.fn().mockResolvedValueOnce({}) };
298+
const won = await claimCommentAck(ddb as never, TABLE, 'orch_1', 'cmt-9', NOW, 1781800000);
299+
expect(won).toBe(true);
300+
const cmd = ddb.send.mock.calls[0][0] as UpdateCommand;
301+
expect(cmd).toBeInstanceOf(UpdateCommand);
302+
expect(cmd.input.Key).toMatchObject({ orchestration_id: 'orch_1', sub_issue_id: 'ack#cmt-9' });
303+
expect(cmd.input.ConditionExpression).toContain('attribute_not_exists(orchestration_id)');
304+
expect(cmd.input.ExpressionAttributeValues).toMatchObject({ ':ttl': 1781800000 });
305+
// ``ttl`` is a DynamoDB reserved keyword — must be aliased, else the write
306+
// 400s with ValidationException (live-caught: the unaliased form errored
307+
// out the whole handler, silently dropping the comment).
308+
expect(cmd.input.ExpressionAttributeNames).toMatchObject({ '#ttl': 'ttl' });
309+
expect(cmd.input.UpdateExpression).toContain('#ttl');
310+
});
311+
312+
test('redelivery of the same comment loses (ConditionalCheckFailed) → false, no throw', async () => {
313+
const ddb = { send: jest.fn().mockRejectedValueOnce(Object.assign(new Error('c'), { name: 'ConditionalCheckFailedException' })) };
314+
expect(await claimCommentAck(ddb as never, TABLE, 'orch_1', 'cmt-9', NOW, 1781800000)).toBe(false);
315+
});
316+
317+
test('non-conditional error propagates', async () => {
318+
const ddb = { send: jest.fn().mockRejectedValueOnce(new Error('throttle')) };
319+
await expect(claimCommentAck(ddb as never, TABLE, 'orch_1', 'cmt-9', NOW, 1781800000)).rejects.toThrow('throttle');
320+
});
321+
});
322+
323+
describe('loadOrchestration — marker rows are not children (#247 UX.20)', () => {
324+
test('excludes ack#<commentId> marker rows from children (only real sub-issues count)', async () => {
325+
const ddb = {
326+
send: jest.fn().mockResolvedValueOnce({
327+
Items: [
328+
{ orchestration_id: 'orch_1', sub_issue_id: '#meta', parent_linear_issue_id: 'P', linear_workspace_id: 'WS', repo: 'o/r', platform_user_id: 'u1', child_count: 2 },
329+
{ orchestration_id: 'orch_1', sub_issue_id: 'uuid-A', depends_on: [], child_status: 'succeeded' },
330+
{ orchestration_id: 'orch_1', sub_issue_id: 'orch_1__integration', depends_on: ['uuid-A'], child_status: 'succeeded' },
331+
{ orchestration_id: 'orch_1', sub_issue_id: 'ack#cmt-9', acked_at: NOW, ttl: 1781800000 }, // marker — must NOT be a child
332+
],
333+
}),
334+
};
335+
const snap = await loadOrchestration(ddb as never, TABLE, 'orch_1');
336+
expect(snap).not.toBeNull();
337+
const ids = snap!.children.map((c) => c.sub_issue_id).sort();
338+
expect(ids).toEqual(['orch_1__integration', 'uuid-A']); // ack# row excluded; integration kept
339+
});
340+
});
341+
293342
describe('findOrchestrationChildByBranch (#305 A6)', () => {
294343
test('queries the ChildBranchIndex GSI by branch and returns the child row', async () => {
295344
const ddb = makeDdb();

0 commit comments

Comments
 (0)