feat: AgentCore Evo — config bundles, batch eval, recommendations, AB testing

[notgitika]

Summary

AgentCore Evo private preview features for the CLI. This PR brings the feat/evo-implementation branch (developed on the staging repo) into the public CLI repo.

New capabilities

Config Bundles — Versioned runtime configurations (system prompt, tool descriptions, model params). Create with agentcore add config-bundle or --with-config-bundle on agent creation. Manage versions with agentcore cb versions/diff/create-branch. Config bundle values are injected at runtime via SDK baggage — no code redeploy needed to change behavior.

Batch Evaluation — agentcore run batch-evaluation runs evaluators (Builtin.Correctness, Helpfulness, Faithfulness, etc.) across agent sessions in CloudWatch. Supports multiple evaluators, session filtering, lookback windows, ground truth assertions, and custom names.

Recommendations — agentcore run recommendation optimizes system prompts and tool descriptions using agent traces. Can read from and write back to config bundles for zero-redeploy prompt updates.

AB Testing — Config-bundle mode (same code, different configs) and target-based mode (different runtime endpoints). Traffic splitting, online evaluation, pause/resume/stop/promote lifecycle. TUI wizard with side-by-side variant builder.

HTTP Gateways & Runtime Endpoints — Gateway targets, endpoint aliases, and routing infrastructure for AB tests.

Other changes

• Batch eval API migrated to latest dataplane schema (batchEvaluationName, dataSourceConfig, evaluationMetadata)
• Config bundle baggage passed on agentcore invoke for runtime config injection
• [preview] tags on all evo commands and TUI screens
• Region resolution from aws-targets.json across all evo commands
• CLI command name agentcore (was agentcore-dev), distro mode set to PROD_DISTRO
• Merged with public/main — includes telemetry, web-ui traces, inspector updates

Tested

• E2E on prod and gamma (us-east-1, ap-southeast-2)
• Full config bundle flow: create → deploy → invoke → recommendation → invoke with updated prompt
• Batch eval with single/multiple evaluators, ground truth, stop
• AB test lifecycle (config-bundle and target-based modes)
• Bug bash completed with bug tracker

[agentcore-cli-automation]

Orphaned config bundles never get deleted when all bundles are removed from the spec

In src/cli/commands/deploy/actions.ts around line 571, the post-deploy config bundle block is gated on:

const configBundleSpecs = context.projectSpec.configBundles ?? [];
if (configBundleSpecs.length > 0) {
  // ... calls setupConfigBundles, which also handles orphan deletion
}

If a user removes all configBundles from agentcore.json and redeploys, setupConfigBundles is never called, so orphaned bundles in deployed state are never deleted. They linger in the account (costing money / creating drift) and stay stuck in deployed-state.json forever.

Compare to how HTTP gateways handle this on line 542:

if (httpGatewaySpecs.length > 0 || Object.keys(existingHttpGateways ?? {}).length > 0) {

Two possible fixes:

1. Mirror the HTTP gateway condition: if (configBundleSpecs.length > 0 || Object.keys(existingConfigBundles ?? {}).length > 0).
2. Split orphan deletion into a pre-pass like deleteOrphanedABTests.

Related: even when setupConfigBundles does run, the state-merge block at line 583 is guarded by if (Object.keys(configBundleResult.configBundles).length > 0). configBundles in the result is only populated by create/update/skip paths — deletions never add to it. So if every spec bundle is deleted in one run (e.g. user has bundles A, B; removes A and B and adds nothing), deployed state keeps the stale entries and targetResources.configBundles is never overwritten. The HTTP gateway post-deploy explicitly comments "Always merge HTTP gateway state (even if empty, to clear deleted gateways)" — config bundles should do the same.

[agentcore-cli-automation]

Baggage header is silently dropped for MCP-protocol invocations

In src/cli/commands/invoke/action.ts lines 241-248, MCP invocations construct:

const mcpOpts = {
  region: targetConfig.region,
  runtimeArn: agentState.runtimeArn,
  userId: options.userId,
  headers: options.headers,
  bearerToken: options.bearerToken,
    baggage,  // ← never actually sent
};

But McpInvokeOptions in src/cli/aws/agentcore.ts (around line 543) doesn't declare a baggage field, and neither mcpRpcCall, mcpRpcCallWithBearer, nor buildMcpBearerHeaders reads it or forwards it to the runtime. So config-bundle baggage for MCP agents is lost — the agent never sees the bundle ARN/version, and BedrockAgentCoreContext.get_config_bundle() will return defaults.

Two fixes depending on intent:

1. If MCP agents should support config bundles: add baggage to McpInvokeOptions and forward it in both buildMcpBearerHeaders (as a baggage header) and the SigV4 path via InvokeAgentRuntimeCommand's baggage field.
2. If config bundles are HTTP-only for now: remove the baggage key from the mcpOpts literal so readers don't think it's wired up.

(Side note: the indentation on that baggage, line is off — two extra spaces.)

[agentcore-cli-automation]

Three of the new preview-API clients hardcode amazonaws.com — breaks in GovCloud / China partitions

The five new SigV4 HTTP clients for preview APIs inconsistently handle partition DNS suffixes:

• ✅ src/cli/aws/agentcore-ab-tests.ts (line 205): return `https://bedrock-agentcore.${region}.${dnsSuffix(region)}`;
• ✅ src/cli/aws/agentcore-http-gateways.ts (line 158): uses dnsSuffix(region)
• ❌ src/cli/aws/agentcore-config-bundles.ts (line 181): return `https://bedrock-agentcore-control.${region}.amazonaws.com`;
• ❌ src/cli/aws/agentcore-recommendation.ts (line 228): hardcoded amazonaws.com
• ❌ src/cli/aws/agentcore-batch-evaluation.ts (line 231): hardcoded amazonaws.com

Options:

1. Use dnsSuffix(region) in all five (matching the two that already do). This is the consistent fix.
2. If these preview features are commercial-only and will never be offered in GovCloud / China during preview, at least leave a comment to that effect — otherwise someone reading the two correct files will assume the hardcoded ones are bugs to "fix" later and accidentally break them.

Since all five already share the same signing-helper pattern (essentially duplicated across files per the "interim until SDK support" comments), extracting a shared signedRequest helper would also remove the opportunity for this kind of drift.