fix(cli): Add a CLI pre-deploy check that, for each onlineEvalConfi... (#570)

[aidandaly24]

Refs aws#570

Issues

• Online Eval config -- better error handling on deploy when an associated eval is deleted in the account aws/agentcore-cli#570 — If an online eval config references an evaluator (by Builtin.* ID, ARN, or a previously-deployed custom evaluator) that has since been deleted out-of-band in the console, the next agentcore deploy fails at CloudFormation execution with a raw, unenriched resource error and no actionable guidance. The deploy fails safely (CFN rolls back), but the user is left guessing what went wrong, matching the issue's "some error -> should be better error handling."

Root cause

No pre-deploy evaluator-existence validation exists anywhere. (1) Schema online-eval-config.ts L37-38: evaluators is z.array(z.string().min(1)) accepting custom names, Builtin.* IDs, or ARNs as plain strings, with no existence check. (2) CDK construct AgentCoreOnlineEvaluationConfig.ts evaluatorRefs map (alpha.39 L162-181; identical at the CLI-pinned alpha.19 L134-152 per src/assets/cdk/package.json "@aws/agentcore-cdk": "^0.1.0-alpha.19") passes Builtin/ARN refs straight through as evaluatorId and only validates project-local custom evaluators against its construct-local evaluators map -- so synth always succeeds and the failure only surfaces at CloudFormation execution when the underlying evaluator is gone. (3) preflight.ts validateProject (L66-147) checks runtime names, Dockerfiles, harness specs, and AWS creds but never evaluator existence; deploy/validate.ts validateDeployOptions (L8-11) is a stub that always returns valid; getEvaluator/listAllEvaluators in agentcore-control.ts are used only by import/status/run-eval, never in deploy preflight. (4) The raw CFN error bubbles through handleDeploy's top-level catch (actions.ts L923-926, toError(err)) and renders as deployResult.error.message (command.tsx L60) in the non-interactive path and via formatError in the TUI hook (useCdkPreflight.ts) -- no evaluator-aware enrichment in either path. (5) The graceful post-deploy enable path (post-deploy-online-evals.ts collects errors as warnings) runs only after runDeploy succeeds (actions.ts L782, after the L494 deploy), so it never reaches the deleted-evaluator case. Correction to brief: formatError is the TUI render path only; the headless command path renders error.message directly -- substance (no enrichment) is unchanged. Behavior identical at alpha.19 and alpha.39, so not silently fixed.

The fix

Add a CLI pre-deploy check that, for each onlineEvalConfig, resolves every referenced evaluator that is NOT a project-managed resource (Builtin.* IDs, ARNs, and any custom names not in projectSpec.evaluators) against the account via getEvaluator (agentcore-control.ts ~L496) or listAllEvaluators (~L628), and throws an actionable ValidationError naming the config + missing evaluator (e.g. "Evaluator X referenced by online eval config Y no longer exists in . Recreate it or remove the reference from agentcore.json, then deploy."). Wire it into validateProject in preflight.ts so it gates before synth. Optionally map the matching CloudFormation CREATE/UPDATE_FAILED evaluator error in the actions.ts catch to the same message as a race backstop. Design note: skip on teardown deploys (mirrors the existing credential/harness skip) and only network-validate refs that aren't project-managed, to avoid false positives on evaluators this deploy is itself about to create.

Files touched: CLI: src/cli/operations/deploy/preflight.ts (new evaluator-existence check inside validateProject, after validateHarnessSpecs), reusing src/cli/aws/agentcore-control.ts getEvaluator/listAllEvaluators; optional backstop enrichment in src/cli/commands/deploy/actions.ts (~L923 catch). No CDK change required -- the pass-through in agentcore-l3-cdk-constructs src/cdk/constructs/components/primitives/evaluator/AgentCoreOnlineEvaluationConfig.ts is by design.

Validation evidence

The fix was verified by reproducing the original symptom and re-running after the change:

BEFORE: git show main:src/cli/operations/deploy/preflight.ts has NO evaluator-existence check (grep for validateOnlineEvalEvaluators/getEvaluator/"no longer exists" => "NOT PRESENT ON MAIN"); a deleted evaluator referenced by an online eval config passes schema + CDK synth and only fails opaquely at CloudFormation.

AFTER (real built CLI dist/cli/index.mjs, AWS_PROFILE=deploy, scratch project acfix570 with a harness so it is a real deploy):
(a) onlineEvalConfigs myMonitor referencing "Builtin.NonExistentEvaluatorXYZ" -> deploy --dry-run --yes halts at the "Validate project" preflight step (never reaches synth/CFN) printing exactly: Evaluator "Builtin.NonExistentEvaluatorXYZ" referenced by online eval config "myMonitor" no longer exists in us-east-1. Recreate it or remove the reference from agentcore.json, then deploy. Headless --json path renders the same string in {"success":false,...,"error":"..."}.
(b1) Same config pointing at the genuinely-existing Builtin.Correctness -> {"success":true,...} (no false positive).
(b2) Config pointing at project-managed evaluator "MyProjectEval" (present in projectSpec.evaluators, NOT yet in AWS — this deploy creates it) -> {"success":true,...}, no API call, no false positive.
(c) Teardown deploy (all gate resources emptied incl. harnesses, deployed-state.json targets={default:{}} so isTeardownDeploy=true, bogus Builtin.NonExistentEvaluatorXYZ still referenced) -> {"success":true,...}; deploy log shows "STEP: Validate project".

Test suite: green.

---

Staged on the fork as a draft for human review. Promote to aws/agentcore-cli after vetting.

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	37.18%	13607 / 36592
🔵	Statements	36.45%	14468 / 39685
🔵	Functions	31.82%	2335 / 7338
🔵	Branches	31.12%	9011 / 28948

Generated in workflow #92 for commit 3e39b14 by the Vitest Coverage Report Action