Skip to content

Commit 9f231d0

Browse files
jariy17Local E2E
andauthored
fix: correct AB test execution role IAM policy and promote stability (#1120)
* fix: add GetEvaluator permission to AB test execution role The AB test API validates evaluator ARNs server-side using the execution role. The auto-created role policy was missing bedrock-agentcore:GetEvaluator, causing a 400 "Access denied when validating evaluator" error on every deploy that included a target-based AB test with a custom evaluator. * fix: correct status assertion in target-based AB test e2e HTTP gateways are not surfaced as top-level resources in agentcore status — they are only used internally to build AB test invocation URLs. The test was asserting on resourceType 'http-gateway' which never appears; fix it to assert on the 'ab-test' resource instead. * fix: add invocationUrl to status resource type cast in e2e test The resource cast was missing invocationUrl, causing a TypeScript error when asserting the AB test's gateway invocation URL was present. * fix: improve promote error message and add local e2e run script - Include stdout in promote failure message so the JSON error is visible - Add scripts/run-e2e-local.sh to replicate the GitHub Actions e2e workflow locally * fix: retry promote stop on 409 UPDATING status When promote is called immediately after resume, the AB test may still be in UPDATING state and reject the STOPPED transition with a 409. Retry up to 6 times with a 10s delay to wait for the transition to complete. * fix: wait for AB test to leave UPDATING state before promoting Poll getABTest until executionStatus is no longer UPDATING before attempting to stop. The service rejects updates with 409 while a state transition is in progress (e.g. after resume). * fix: wait for RUNNING before stopping AB test in promote Poll getABTest until executionStatus === 'RUNNING' before issuing the STOPPED transition. The service 409s if the test is still UPDATING (e.g. transitioning from a prior resume). * fix: resolve evaluator ARNs transitively from AB test online eval configs Previously GetEvaluator was scoped to all project evaluators. Now it's scoped to only the evaluators referenced by the specific online eval configs this AB test uses, handling all three cases: - Custom evaluators: looked up from deployedResources.evaluators - Builtin.* evaluators: ARN constructed from the builtin ID - External ARN references: used as-is * fix: align AB test execution role policy with official docs Replace the hand-rolled per-resource policy with the canonical policy from https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/ab-testing-prereqs.html: - Trust policy: add SourceAccount + SourceArn conditions - Permissions: single AgentCoreResources statement scoped to arn:aws:bedrock-agentcore:*:${accountId}:* with ResourceAccount condition - Add missing actions: GetGatewayTarget, ListGatewayTargets, ListConfigurationBundleVersions - CloudWatch logs scoped to account via PrincipalAccount pattern - Remove per-evaluator/per-resource ARN tracking (no longer needed) * test: update IAM role unit tests to match new policy structure * fix: throw error if AB test never reaches RUNNING before promote * test: add unit tests for promote polling logic and extract to promote-utils Extract waitForRunningThenStop into promote-utils.ts so it can be unit tested without pulling in React/ink. Add 4 tests covering: immediate RUNNING, polling until RUNNING, timeout throws, and error message content. * fix: split DescribeLogGroups into wildcard resource statement * test: verify AB test reaches RUNNING status after deploy in both e2e suites * fix: remove console.log statements that leaked ARNs and fix bundle ARN/version format in config-bundle e2e test * test: remove second deploy and RUNNING check from config-bundle e2e (follow-up with real bundles) --------- Co-authored-by: Local E2E <ci@local>
1 parent 639adf1 commit 9f231d0

8 files changed

Lines changed: 269 additions & 87 deletions

File tree

e2e-tests/ab-test-config-bundle.test.ts

Lines changed: 8 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -105,10 +105,6 @@ describe.sequential('e2e: config-bundle AB test lifecycle', () => {
105105
await retry(
106106
async () => {
107107
const result = await run(['deploy', '--yes', '--json']);
108-
if (result.exitCode !== 0) {
109-
console.log('Initial deploy stdout:', result.stdout);
110-
console.log('Initial deploy stderr:', result.stderr);
111-
}
112108
expect(result.exitCode, `Initial deploy failed`).toBe(0);
113109
const json = parseJsonOutput(result.stdout) as { success: boolean };
114110
expect(json.success).toBe(true);
@@ -123,10 +119,12 @@ describe.sequential('e2e: config-bundle AB test lifecycle', () => {
123119
it.skipIf(!canRun)(
124120
'adds config-bundle AB test with 90/10 split',
125121
async () => {
126-
// Config bundles reference ARNs from deployed resources.
127-
// Use placeholder bundle ARNs — the deploy step will validate or create them.
128-
const controlBundle = `arn:aws:bedrock-agentcore:ap-southeast-2:998846730471:config-bundle/control-v1`;
129-
const treatmentBundle = `arn:aws:bedrock-agentcore:ap-southeast-2:998846730471:config-bundle/treatment-v1`;
122+
// Use placeholder bundle ARNs that satisfy the service format constraints.
123+
// Real config bundles would be created separately; these test the AB test wiring.
124+
const region = process.env.AWS_REGION ?? 'us-east-1';
125+
const account = process.env.AWS_ACCOUNT_ID ?? '000000000000';
126+
const controlBundle = `arn:aws:bedrock-agentcore:${region}:${account}:configuration-bundle/control-bundle-AbCdEfGhIj`;
127+
const treatmentBundle = `arn:aws:bedrock-agentcore:${region}:${account}:configuration-bundle/treatment-bundle-AbCdEfGhIj`;
130128

131129
const result = await run([
132130
'add',
@@ -140,11 +138,11 @@ describe.sequential('e2e: config-bundle AB test lifecycle', () => {
140138
'--control-bundle',
141139
controlBundle,
142140
'--control-version',
143-
'v1',
141+
'00000000-0000-0000-0000-000000000001',
144142
'--treatment-bundle',
145143
treatmentBundle,
146144
'--treatment-version',
147-
'v1',
145+
'00000000-0000-0000-0000-000000000002',
148146
'--control-weight',
149147
'90',
150148
'--treatment-weight',

e2e-tests/ab-test-target-based.test.ts

Lines changed: 25 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -195,10 +195,6 @@ describe.sequential('e2e: target-based AB test lifecycle', () => {
195195
await retry(
196196
async () => {
197197
const result = await run(['deploy', '--yes', '--json']);
198-
if (result.exitCode !== 0) {
199-
console.log('Deploy stdout:', result.stdout);
200-
console.log('Deploy stderr:', result.stderr);
201-
}
202198
expect(result.exitCode, `Deploy failed (stderr: ${result.stderr})`).toBe(0);
203199
const json = parseJsonOutput(result.stdout) as { success: boolean };
204200
expect(json.success).toBe(true);
@@ -210,6 +206,23 @@ describe.sequential('e2e: target-based AB test lifecycle', () => {
210206
600000
211207
);
212208

209+
it.skipIf(!canRun)(
210+
'AB test reaches RUNNING status after deploy',
211+
async () => {
212+
await retry(
213+
async () => {
214+
const result = await run(['ab-test', abTestName, '--json']);
215+
expect(result.exitCode, `ab-test lookup failed: ${result.stdout} ${result.stderr}`).toBe(0);
216+
const json = parseJsonOutput(result.stdout) as { executionStatus: string };
217+
expect(json.executionStatus, 'AB test should be RUNNING after deploy').toBe('RUNNING');
218+
},
219+
12,
220+
15000
221+
);
222+
},
223+
300000
224+
);
225+
213226
it.skipIf(!canRun)(
214227
'status shows all resources deployed',
215228
async () => {
@@ -220,7 +233,7 @@ describe.sequential('e2e: target-based AB test lifecycle', () => {
220233

221234
const json = parseJsonOutput(result.stdout) as {
222235
success: boolean;
223-
resources: { resourceType: string; name: string; deploymentState: string }[];
236+
resources: { resourceType: string; name: string; deploymentState: string; invocationUrl?: string }[];
224237
};
225238
expect(json.success).toBe(true);
226239

@@ -229,9 +242,12 @@ describe.sequential('e2e: target-based AB test lifecycle', () => {
229242
expect(agent, `Agent "${agentName}" should appear in status`).toBeDefined();
230243
expect(agent!.deploymentState).toBe('deployed');
231244

232-
// Gateway should be deployed
233-
const gateway = json.resources.find(r => r.resourceType === 'http-gateway' && r.name === `${abTestName}-gw`);
234-
expect(gateway, 'HTTP gateway should appear in status').toBeDefined();
245+
// AB test should be deployed (HTTP gateways are not surfaced as top-level status resources)
246+
const abTest = json.resources.find(r => r.resourceType === 'ab-test' && r.name === abTestName);
247+
expect(abTest, `AB test "${abTestName}" should appear in status`).toBeDefined();
248+
expect(abTest!.deploymentState).toBe('deployed');
249+
// invocationUrl proves the HTTP gateway was deployed and wired up correctly
250+
expect(abTest!.invocationUrl, 'AB test should have a gateway invocation URL').toBeTruthy();
235251
},
236252
3,
237253
15000
@@ -280,7 +296,7 @@ describe.sequential('e2e: target-based AB test lifecycle', () => {
280296
'promotes AB test (updates agentcore.json)',
281297
async () => {
282298
const result = await run(['promote', 'ab-test', abTestName, '--json']);
283-
expect(result.exitCode, `Promote failed: ${result.stderr}`).toBe(0);
299+
expect(result.exitCode, `Promote failed: ${result.stdout} ${result.stderr}`).toBe(0);
284300
const json = parseJsonOutput(result.stdout) as Record<string, unknown>;
285301
expect(json).toHaveProperty('success', true);
286302
expect(json).toHaveProperty('promoted', true);

scripts/run-e2e-local.sh

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
#!/usr/bin/env bash
2+
# Run E2E tests locally, replicating the GitHub Actions e2e-tests.yml workflow.
3+
#
4+
# Required env vars:
5+
# E2E_ROLE_ARN — IAM role ARN to assume (grants access to the test account)
6+
# E2E_SECRET_ARN — Secrets Manager ARN containing ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY
7+
#
8+
# Optional env vars:
9+
# AWS_REGION — defaults to us-east-1
10+
#
11+
# Usage:
12+
# export E2E_ROLE_ARN=arn:aws:iam::<account>:role/<role>
13+
# export E2E_SECRET_ARN=arn:aws:secretsmanager:<region>:<account>:secret:<name>
14+
# ./scripts/run-e2e-local.sh # runs strands-bedrock.test.ts (CI default)
15+
# ./scripts/run-e2e-local.sh --all # runs the full e2e suite
16+
# ./scripts/run-e2e-local.sh e2e-tests/foo.test.ts # runs a specific test file
17+
#
18+
# Prerequisites: aws CLI, node >=20.19, npm, git, uv, jq
19+
20+
set -euo pipefail
21+
22+
ROLE_ARN="${E2E_ROLE_ARN:-}"
23+
SECRET_ARN="${E2E_SECRET_ARN:-}"
24+
AWS_REGION="${AWS_REGION:-us-east-1}"
25+
26+
if [[ -z "$ROLE_ARN" ]]; then
27+
echo "❌ E2E_ROLE_ARN is not set. Export it before running this script:"
28+
echo " export E2E_ROLE_ARN=arn:aws:iam::<account>:role/<role-name>"
29+
exit 1
30+
fi
31+
32+
if [[ -z "$SECRET_ARN" ]]; then
33+
echo "❌ E2E_SECRET_ARN is not set. Export it before running this script:"
34+
echo " export E2E_SECRET_ARN=arn:aws:secretsmanager:<region>:<account>:secret:<name>"
35+
exit 1
36+
fi
37+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
38+
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
39+
40+
# ── Parse arguments ────────────────────────────────────────────────────────────
41+
RUN_ALL=false
42+
TEST_FILES=()
43+
for arg in "$@"; do
44+
if [[ "$arg" == "--all" ]]; then
45+
RUN_ALL=true
46+
else
47+
TEST_FILES+=("$arg")
48+
fi
49+
done
50+
51+
echo "=== Assuming IAM role ==="
52+
CREDS=$(aws sts assume-role \
53+
--role-arn "$ROLE_ARN" \
54+
--role-session-name "local-e2e-$(date +%s)" \
55+
--duration-seconds 3600 \
56+
--query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
57+
--output text)
58+
59+
export AWS_ACCESS_KEY_ID=$(echo "$CREDS" | awk '{print $1}')
60+
export AWS_SECRET_ACCESS_KEY=$(echo "$CREDS" | awk '{print $2}')
61+
export AWS_SESSION_TOKEN=$(echo "$CREDS" | awk '{print $3}')
62+
export AWS_REGION
63+
64+
echo "✅ Assumed role successfully"
65+
66+
echo "=== Fetching API keys from Secrets Manager ==="
67+
SECRET_JSON=$(aws secretsmanager get-secret-value \
68+
--secret-id "$SECRET_ARN" \
69+
--region "$AWS_REGION" \
70+
--query SecretString \
71+
--output text)
72+
73+
# Mirror the GitHub workflow: parse-json-secrets maps keys to E2E_<KEY> then
74+
# the workflow maps them to the bare names the tests expect.
75+
export ANTHROPIC_API_KEY=$(echo "$SECRET_JSON" | jq -r '.ANTHROPIC_API_KEY // empty')
76+
export OPENAI_API_KEY=$(echo "$SECRET_JSON" | jq -r '.OPENAI_API_KEY // empty')
77+
export GEMINI_API_KEY=$(echo "$SECRET_JSON" | jq -r '.GEMINI_API_KEY // empty')
78+
79+
echo "✅ Secrets loaded (keys present: $(echo "$SECRET_JSON" | jq -r 'keys | join(", ")')"
80+
81+
echo "=== Setting AWS account env var ==="
82+
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
83+
echo "✅ AWS_ACCOUNT_ID=$AWS_ACCOUNT_ID AWS_REGION=$AWS_REGION"
84+
85+
echo "=== Configuring git (required for agentcore create) ==="
86+
git config --global user.email "ci@local" 2>/dev/null || true
87+
git config --global user.name "Local E2E" 2>/dev/null || true
88+
89+
cd "$REPO_ROOT"
90+
91+
echo "=== Installing dependencies ==="
92+
npm ci
93+
94+
echo "=== Building CLI ==="
95+
npm run build
96+
97+
echo "=== Installing CLI globally ==="
98+
TARBALL=$(npm pack | tail -1)
99+
npm install -g "$TARBALL"
100+
echo "✅ Installed: $(agentcore --version)"
101+
102+
echo "=== Running E2E tests ==="
103+
if [[ "$RUN_ALL" == "true" ]]; then
104+
echo "Running full e2e suite"
105+
npx vitest run --project e2e
106+
elif [[ ${#TEST_FILES[@]} -gt 0 ]]; then
107+
echo "Running: ${TEST_FILES[*]}"
108+
npx vitest run --project e2e "${TEST_FILES[@]}"
109+
else
110+
echo "Running default: e2e-tests/strands-bedrock.test.ts"
111+
npx vitest run --project e2e e2e-tests/strands-bedrock.test.ts
112+
fi
Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
import { waitForRunningThenStop } from '../promote-utils.js';
2+
import { beforeEach, describe, expect, it, vi } from 'vitest';
3+
4+
const mockGetABTest = vi.fn();
5+
const mockUpdateABTest = vi.fn();
6+
7+
vi.mock('../../../aws/agentcore-ab-tests', () => ({
8+
getABTest: (...args: unknown[]) => mockGetABTest(...args),
9+
updateABTest: (...args: unknown[]) => mockUpdateABTest(...args),
10+
}));
11+
12+
describe('waitForRunningThenStop', () => {
13+
beforeEach(() => {
14+
vi.clearAllMocks();
15+
mockUpdateABTest.mockResolvedValue({ executionStatus: 'STOPPED' });
16+
});
17+
18+
it('stops immediately when already RUNNING', async () => {
19+
mockGetABTest.mockResolvedValue({ executionStatus: 'RUNNING' });
20+
21+
await waitForRunningThenStop('us-east-1', 'abt-123', 'MyTest', 3, 0);
22+
23+
expect(mockGetABTest).toHaveBeenCalledTimes(1);
24+
expect(mockUpdateABTest).toHaveBeenCalledWith({
25+
region: 'us-east-1',
26+
abTestId: 'abt-123',
27+
executionStatus: 'STOPPED',
28+
});
29+
});
30+
31+
it('polls until RUNNING then stops', async () => {
32+
mockGetABTest
33+
.mockResolvedValueOnce({ executionStatus: 'UPDATING' })
34+
.mockResolvedValueOnce({ executionStatus: 'UPDATING' })
35+
.mockResolvedValueOnce({ executionStatus: 'RUNNING' });
36+
37+
await waitForRunningThenStop('us-east-1', 'abt-123', 'MyTest', 5, 0);
38+
39+
expect(mockGetABTest).toHaveBeenCalledTimes(3);
40+
expect(mockUpdateABTest).toHaveBeenCalledOnce();
41+
});
42+
43+
it('throws if AB test never reaches RUNNING', async () => {
44+
mockGetABTest.mockResolvedValue({ executionStatus: 'UPDATING' });
45+
46+
await expect(waitForRunningThenStop('us-east-1', 'abt-123', 'MyTest', 3, 0)).rejects.toThrow(
47+
'did not reach RUNNING state'
48+
);
49+
50+
expect(mockGetABTest).toHaveBeenCalledTimes(3);
51+
expect(mockUpdateABTest).not.toHaveBeenCalled();
52+
});
53+
54+
it('includes current status in the error message', async () => {
55+
mockGetABTest.mockResolvedValue({ executionStatus: 'STOPPED' });
56+
57+
await expect(waitForRunningThenStop('us-east-1', 'abt-123', 'MyTest', 2, 0)).rejects.toThrow('current: STOPPED');
58+
});
59+
});

src/cli/commands/pause/command.tsx

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import type { OnlineEvalActionOptions } from '../../operations/eval';
77
import { COMMAND_DESCRIPTIONS } from '../../tui/copy';
88
import { requireProject } from '../../tui/guards';
99
import { getRegion } from '../shared/region-utils';
10+
import { waitForRunningThenStop } from './promote-utils';
1011
import type { Command } from '@commander-js/extra-typings';
1112
import { Text, render } from 'ink';
1213
import React from 'react';
@@ -274,12 +275,7 @@ export const registerPromote = (program: Command) => {
274275
process.exit(1);
275276
}
276277

277-
// Stop the AB test
278-
const result = await updateABTest({
279-
region,
280-
abTestId,
281-
executionStatus: 'STOPPED',
282-
});
278+
const result = await waitForRunningThenStop(region, abTestId, name);
283279

284280
// Apply promotion to agentcore.json
285281
const { promoteABTestConfig } = await import('../../operations/ab-test/promote');
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
import { getABTest, updateABTest } from '../../aws/agentcore-ab-tests';
2+
import type { UpdateABTestResult } from '../../aws/agentcore-ab-tests';
3+
4+
/**
5+
* Poll until the AB test reaches RUNNING status, then stop it.
6+
* Throws if the test never reaches RUNNING within the allotted attempts.
7+
*/
8+
export async function waitForRunningThenStop(
9+
region: string,
10+
abTestId: string,
11+
name: string,
12+
maxAttempts = 12,
13+
delayMs = 10_000
14+
): Promise<UpdateABTestResult> {
15+
let currentStatus: string | undefined;
16+
for (let attempt = 0; attempt < maxAttempts; attempt++) {
17+
const current = await getABTest({ region, abTestId });
18+
currentStatus = current.executionStatus;
19+
if (currentStatus === 'RUNNING') break;
20+
await new Promise(resolve => setTimeout(resolve, delayMs));
21+
}
22+
if (currentStatus !== 'RUNNING') {
23+
throw new Error(
24+
`AB test "${name}" did not reach RUNNING state after waiting (current: ${currentStatus}). Cannot promote.`
25+
);
26+
}
27+
return updateABTest({ region, abTestId, executionStatus: 'STOPPED' });
28+
}

src/cli/operations/deploy/__tests__/post-deploy-ab-tests.test.ts

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -516,22 +516,22 @@ describe('setupABTests', () => {
516516
const trustPolicy = JSON.parse(createRoleCall.input.AssumeRolePolicyDocument);
517517
expect(trustPolicy.Statement).toHaveLength(1);
518518
expect(trustPolicy.Statement[0].Principal.Service).toBe('bedrock-agentcore.amazonaws.com');
519+
expect(trustPolicy.Statement[0].Condition.StringEquals['aws:SourceAccount']).toBeDefined();
520+
expect(trustPolicy.Statement[0].Condition.ArnLike['aws:SourceArn']).toContain('ab-test/*');
519521

520522
// Second call: PutRolePolicyCommand with inline policy
521523
const putPolicyCall = mockIAMSend.mock.calls[1]![0];
522524
const policy = JSON.parse(putPolicyCall.input.PolicyDocument);
523525
const sids = policy.Statement.map((s: { Sid: string }) => s.Sid);
524-
expect(sids).toContain('GatewayRuleStatement');
525-
expect(sids).toContain('GatewayReadStatement');
526-
expect(sids).toContain('GatewayListStatement');
527-
expect(sids).toContain('OnlineEvaluationConfigStatement');
528-
expect(sids).toContain('ConfigurationBundleReadStatement');
529-
expect(sids).toContain('CloudWatchLogReadStatement');
530-
expect(sids).toContain('CloudWatchIndexPolicyStatement');
531-
532-
// ListGateways must use wildcard resource (can't be scoped)
533-
const listGatewayStmt = policy.Statement.find((s: { Sid: string }) => s.Sid === 'GatewayListStatement');
534-
expect(listGatewayStmt.Resource).toEqual(['*']);
526+
expect(sids).toContain('AgentCoreResources');
527+
expect(sids).toContain('CloudWatchLogs');
528+
529+
// AgentCoreResources must include all required actions
530+
const agentCoreStmt = policy.Statement.find((s: { Sid: string }) => s.Sid === 'AgentCoreResources');
531+
expect(agentCoreStmt.Action).toContain('bedrock-agentcore:GetEvaluator');
532+
expect(agentCoreStmt.Action).toContain('bedrock-agentcore:GetGateway');
533+
expect(agentCoreStmt.Action).toContain('bedrock-agentcore:GetOnlineEvaluationConfig');
534+
expect(agentCoreStmt.Condition.StringEquals['aws:ResourceAccount']).toBeDefined();
535535
});
536536
});
537537

0 commit comments

Comments
 (0)