Commit e266576
authored
feat(import): add evaluator and online eval config import subcommands (#780)
* feat(import): add evaluator import subcommand with TUI wizard
Add `agentcore import evaluator` to import existing AWS evaluators into
CLI projects. Refactor import types and utilities for extensibility so
future resource types require minimal new code.
Changes:
- Add import-evaluator.ts handler with toEvaluatorSpec mapping (LLM-as-a-Judge
and code-based evaluators), duplicate detection, and CDK import pipeline
- Enhance getEvaluator API wrapper to extract full evaluatorConfig (model,
instructions, ratingScale) and tags from SDK tagged unions
- Add listAllEvaluators pagination helper filtering out built-in evaluators
- Widen ImportableResourceType union and shared utilities for evaluator support
- Add evaluator to TUI import flow (select, ARN input, progress screens)
- Add 17 unit tests covering spec conversion, template lookup, and error cases
Tested end-to-end against real AWS evaluator (bugbash_eval_1775226567-zrDxm7Gpcw)
with verified field mapping for all config fields, tags, and deployed state.
* fix(import): use correct importType for evaluator in TUI flow
The TUI import wizard hardcoded importType as 'memory' for all non-runtime
resources, causing evaluator imports to fail with "ARN resource type
evaluator does not match expected type memory". Use flow.resourceType
instead so the correct handler is dispatched.
* feat(import): add online eval config import subcommand
Add `agentcore import online-eval` to import existing online evaluation
configs from AWS into CLI-managed projects. Follows the same pattern as
runtime, memory, and evaluator imports.
The command extracts the agent reference from the config's service names
(pattern: {agentName}.DEFAULT), maps evaluator IDs to local names or
ARN fallbacks, and runs the full CDK import pipeline.
Also removes incorrect project-prefix stripping from evaluator and
runtime imports — imported resources come from outside the project
and won't have the project prefix.
Constraint: Agent must exist in project runtimes[] before import (schema enforces cross-reference)
Constraint: Evaluators not in project fall back to ARN format to bypass schema validation
Rejected: Loose agent validation | schema writeProjectSpec() enforces runtimes[] cross-reference
Confidence: high
Scope-risk: moderate
* feat(import): add online eval config to TUI import wizard
Add 'Online Eval Config' option to the interactive import flow so users
can import online evaluation configs via the TUI, not just the CLI.
Follows the same ARN-only pattern as evaluator and memory imports:
select type → enter ARN → import progress → success/error.
* docs: add TUI import wizard screenshots for online eval
Screenshots captured from the TUI import flow showing:
- Import type selection menu with Online Eval Config option
- ARN input screen for online eval config
- ARN input with a real config ARN filled in
* Revert "docs: add TUI import wizard screenshots for online eval"
This reverts commit cb4c675.
* refactor(import): extract generic import orchestrator with descriptor pattern
Reduce ~1,400 lines of duplicated orchestration across four import handlers
(runtime, memory, evaluator, online-eval) to ~600 lines by extracting shared
logic into executeResourceImport(). Each resource type now provides a thin
descriptor declaring its specific behavior.
Constraint: Public handleImport* function signatures unchanged (TUI depends on them)
Constraint: Factory functions needed for runtime/online-eval to share mutable state between hooks
Rejected: Strategy class hierarchy | descriptor objects are simpler and more composable
Confidence: high
Scope-risk: moderate
* refactor(aws): extract paginateAll and fetchTags helpers in agentcore-control
Deduplicates identical pagination loops across 4 listAll* functions and
identical tag-fetching try/catch blocks across 3 getDetail functions.
Also adds optional client param to listEvaluators and
listOnlineEvaluationConfigs for connection reuse during pagination.
Addresses deferred review feedback from PR #763.
Constraint: evaluator listAll still filters out Builtin.* entries
Confidence: high
Scope-risk: narrow
* fix(import): resolve evaluator references via deployed state for imported evaluators
resolveEvaluatorReferences used string-contains matching (evaluatorId.includes(localName))
which only works when the evaluator was deployed by the same project. Imported evaluators
with renamed local names never matched, falling back to raw ARNs in the config.
Now reads deployed-state.json to build an evaluatorId → localName reverse map and checks
it first, before the string-contains heuristic.
Constraint: Deployed state may not exist yet (first import) — .catch() handles gracefully
Rejected: Passing deployed state through descriptor interface | only online-eval needs this
Confidence: high
Scope-risk: narrow
* fix(import): auto-disable online eval configs to unlock evaluators during import
Evaluators referenced by ENABLED online eval configs are locked by the service
(lockedForModification=true), causing CFN import to fail when it tries to apply
stack-level tags. Now the evaluator import detects the lock, temporarily disables
referencing online eval configs, performs the import, then re-enables them.
Constraint: Re-enable runs in finally block so configs are restored on both success and failure
Constraint: Only disables configs that actually reference this specific evaluator
Rejected: Refuse import with manual guidance | user can't pause configs not yet in project
Confidence: high
Scope-risk: moderate
* Revert "fix(import): auto-disable online eval configs to unlock evaluators during import"
This reverts commit 5839391.
* fix(import): block evaluator import when referenced by online eval, use ARN-only references
Evaluators locked by an online eval config cannot be CFN-imported because
CloudFormation triggers a post-import TagResource call that the resource
handler rejects. Instead of stripping tags from the import template, block
the import with a clear error and suggestion to use import online-eval.
Online eval config import now always references evaluators by ARN rather
than resolving to local names, since the evaluators cannot be imported
into the project alongside the config.
Constraint: CFN IMPORT triggers TagResource which fails on locked evaluators
Rejected: Strip Tags from import template | still fails on some resource types
Confidence: high
Scope-risk: narrow
* fix(import): resolve OEC agent reference via deployed state when runtime has custom name
extractAgentName() derives the AWS runtime name from the OEC service
name pattern, but this fails to match when the runtime was imported
with --name since the project spec stores the local name. Now falls
back to listing runtimes to find the runtime ID, then looks up the
local name in deployed-state.json.
* fix(import): strip CDK project prefix from OEC service name when resolving agent
CDK constructs set the OEC service name as "{projectName}_{agentName}.DEFAULT".
extractAgentName() strips ".DEFAULT" but not the project prefix, so the
lookup fails against local runtime names. Now strips the prefix as a fast
path before falling back to the deployed-state API lookup.
* fix(import): show friendly error for non-existent evaluator ID
getEvaluator() now catches ResourceNotFoundException and
ValidationException from the SDK and rethrows a clear message
instead of exposing the raw regex validation error.
* fix(import): validate ARN resource type for online-eval import
import online-eval used a naive regex to extract the config ID from the
ARN, skipping resource type, region, and account validation. Now uses
parseAndValidateArn like all other import commands. Added an ARN resource
type mapping to handle the online-eval vs online-evaluation-config
mismatch between ImportableResourceType and the ARN format.
* refactor(import): address PR review feedback
- Add `red` to ANSI constants, replace inline escape codes
- Type GetEvaluatorResult.level as EvaluationLevel at boundary
- Combine ARN_RESOURCE_TYPE_MAP, collectionKeyMap, idFieldMap into
single RESOURCE_TYPE_CONFIG to prevent drift
- Export IMPORTABLE_RESOURCES as const array, derive type from it,
replace || chains with .includes()
- Fix samplingPercentage === 0 false positive (use == null)
- Document closure state sequencing contract on descriptor hooks
* test(import): remove unreachable empty-level evaluator test
The test exercised a defensive fallback in toEvaluatorSpec for an empty
level string, but now that GetEvaluatorResult.level is typed as
EvaluationLevel, the boundary cast in getEvaluator prevents this case
from ever reaching toEvaluatorSpec.1 parent 380ac6e commit e266576
File tree
16 files changed
+2059
-595
lines changed- src/cli
- aws
- commands/import
- __tests__
- tui/screens/import
16 files changed
+2059
-595
lines changedLarge diffs are not rendered by default.
Lines changed: 437 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 368 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
| 2 | + | |
| 3 | + | |
2 | 4 | | |
| 5 | + | |
3 | 6 | | |
4 | 7 | | |
5 | 8 | | |
6 | 9 | | |
7 | | - | |
8 | | - | |
9 | | - | |
10 | | - | |
11 | | - | |
| 10 | + | |
12 | 11 | | |
13 | 12 | | |
14 | 13 | | |
| |||
151 | 150 | | |
152 | 151 | | |
153 | 152 | | |
| 153 | + | |
| 154 | + | |
154 | 155 | | |
0 commit comments