Skip to content

Commit 2e21eb2

Browse files
authored
feat: Add CloudFormation Language Extensions support (Fn::ForEach) (aws#8637)
* feat: add CloudFormation Language Extensions processing library Implement a local CloudFormation Language Extensions processor supporting: - Fn::ForEach loop expansion in Resources, Conditions, and Outputs - Fn::Length, Fn::ToJsonString intrinsic functions - Fn::FindInMap with DefaultValue support - Conditional DeletionPolicy/UpdateReplacePolicy - Nested ForEach depth validation (max 5 levels) - Partial resolution mode preserving unresolvable references Pipeline architecture: TemplateParsingProcessor -> ForEachProcessor -> IntrinsicResolverProcessor -> DeletionPolicyProcessor -> UpdateReplacePolicyProcessor Includes comprehensive unit tests and CloudFormation compatibility suite. * feat: integrate language extensions into SAM CLI Wire the language extensions library into SAM CLI with two-phase architecture: - Phase 1: expand_language_extensions() -> LanguageExtensionResult - Phase 2: SamTranslatorWrapper.run_plugins() (SAM transform only) Key components: - expand_language_extensions() canonical entry point - SamTranslatorWrapper receives pre-expanded template (Phase 2 only) - SamLocalStackProvider.get_stacks() calls expand_language_extensions() - SamTemplateValidator calls expand_language_extensions() - DynamicArtifactProperty dataclass for Mappings transformation - Fn::ForEach guards in artifact_exporter, normalizer, cdk/utils * feat: add language extensions support to sam build - _get_template_for_output() preserves Fn::ForEach in build output - _update_foreach_artifact_paths() generates Mappings for dynamic artifact properties with per-function build paths - Recursive nested Fn::ForEach support - ForEach-aware path resolution skips Docker image URIs Test templates: static CodeUri, dynamic CodeUri, parameter collections, nested stacks, nested ForEach, dynamic ImageUri, depth validation. * feat: add language extensions support to sam package and deploy Package: - _export() calls expand_language_extensions() for Phase 1 - Preserves Fn::ForEach in packaged template with S3 URIs - Generates Mappings for dynamic artifact properties - _find_artifact_uri_for_resource() handles all export formats: string, {S3Bucket,S3Key}, {Bucket,Key}, {ImageUri} - Recursive nested Fn::ForEach support - Warning for parameter-based collections Deploy: - Uploads original unexpanded template to CloudFormation - Clear error for missing Mapping keys Integration tests for CodeUri, ContentUri, DefinitionUri, ImageUri, BodyS3Location across all packageable resource types. * test: add validate, local invoke, and start-api integration tests - sam validate: valid ForEach, invalid syntax, cloud-dependent collections, dynamic CodeUri, nested depth validation (5 valid, 6 invalid) - sam local invoke: expanded function names from ForEach - sam local start-api: ForEach-generated API endpoints * feat: add telemetry tracking for CloudFormation Language Extensions Track CFNLanguageExtensions as a UsedFeature event when templates with AWS::LanguageExtensions transform are expanded. Emitted once per expansion in expand_language_extensions(). * test: trim language extensions integration tests to essential cases Remove redundant and AWS-dependent integration tests, keeping 9 essential tests across build, package, validate, local invoke, and start-api. Delete 34 orphaned testdata directories. * fix: Fn::Equals uses string comparison to match CloudFormation behavior YAML parsing produces Python booleans for bare true/false values, but parameter overrides from --parameter-overrides are always strings. Fn::Equals was using Python == which returns False for 'true' == True. CloudFormation Fn::Equals performs string comparison, so convert both operands to their string representations before comparing. Booleans are lowercased to produce 'true'/'false' matching CFN serialization. * fix: scope intrinsic resolution to Resources, Conditions, and Outputs only Language extension functions are only supported in these three sections per AWS::LanguageExtensions transform documentation. Previously the intrinsic resolver also processed Parameters, Mappings, Metadata, etc. * refactor: rename iter_resources to iter_regular_resources The name iter_regular_resources better conveys that ForEach blocks are skipped. Removes the backward-compatible alias. * refactor: deduplicate _to_boolean and fix TOCTOU in expansion cache Extract duplicated _to_boolean logic from condition_resolver.py and fn_if.py into IntrinsicFunctionResolver.to_boolean() static method. Replace os.path.isfile() + os.path.getmtime() two-step check with a single try/except around getmtime() to eliminate the race condition. * test: remove integration tests referencing deleted test data Remove 9 integration tests whose test data directories were removed in an earlier commit: validate/language-extensions/, buildcmd/language- extensions-dynamic-imageuri/, language-extensions-foreach/, and language-extensions-nested-foreach-{valid,invalid}/. * refactor: address PR review comments on style and test quality - Move inline imports to top level in sam_stack_provider.py and test_template.py - Add missing assertion in test_handles_empty_mappings - Uncomment Fn::ForEach::Topics block to test non-Lambda resource types - Update mock patch paths to match top-level import locations * feat: add &{identifier} syntax support in Fn::ForEach expansion CloudFormation supports &{identifier} syntax which strips non-alphanumeric characters from the substituted value, useful for generating valid logical IDs from values containing dots, slashes, etc. (e.g., IP addresses). Previously only ${identifier} was handled. This adds &{identifier} substitution in _substitute_identifier alongside the existing ${} handling. Includes 4 unit tests covering keys, values, mixed syntax, and no-op case. * fix: remove unsupported list-of-lists identifier format from Fn::ForEach CloudFormation does not support list identifiers in Fn::ForEach — deploying a template with a list identifier returns 'Fn::ForEach layout is incorrect'. Verified by deploying a test template to CFN. Remove _resolve_identifiers, _resolve_collection_item, and all list-of-lists handling from ForEachProcessor. The identifier is now validated as a single non-empty string, matching the CFN spec. Also removes 30 list-of-lists compatibility test template files. * fix: use proper type annotation for intrinsic_resolver parameter Change Optional[Any] to Optional["IntrinsicResolver"] using TYPE_CHECKING import to avoid circular dependencies while enabling proper type checking. * fix: correct _process_section docstring about which sections it handles * test: add tier1_extra markers and replace NamedTemporaryFile - Add @pytest.mark.tier1_extra to invoke, start-api, and build language extensions tests for windows/finch coverage - Replace NamedTemporaryFile with explicit path in test_deploy_command.py to avoid Windows file locking issues * fix: narrow exception handling and bound expansion cache size - artifact_exporter.py: catch InvalidSamDocumentException instead of bare Exception so programming bugs propagate instead of being silently swallowed - sam_integration.py: add _MAX_CACHE_SIZE=32 with FIFO eviction to prevent unbounded memory growth in long-running processes (sam sync --watch) * test: add tests for cache eviction and narrowed exception handling - test_cache_evicts_oldest_when_full: verifies FIFO eviction at _MAX_CACHE_SIZE - test_export_cloudformation_stack_unexpected_exception_propagates: verifies unexpected exceptions (TypeError etc.) are not silently swallowed - Updated existing expansion_failure test to use InvalidSamDocumentException * fix: improve error visibility and add UTF-8 encoding for file reads - artifact_exporter.py: log at WARNING (not DEBUG) when a child template declares AWS::LanguageExtensions but expansion fails, so users see the error instead of discovering it at deploy time - artifact_exporter.py, package_context.py: add encoding='utf-8' to open() calls to prevent Windows locale encoding issues with non-ASCII templates * fix: update test mocks to expect encoding='utf-8' parameter * fix: MissingMappingKeyError now extends DeployFailedError Previously extended UserException directly, so it bypassed the except DeployFailedError handlers in deploy_context.py. This meant rollback-delete on failure wouldn't execute and sync errors weren't logged before re-raising. * fix: catch unexpected exceptions in child template expansion gracefully An unexpected error from expand_language_extensions on a child template should not abort packaging of the entire parent stack. The fallback (using original template) is always safe since CFN handles expansion server-side. Now catches Exception with WARNING log as a second handler after the specific InvalidSamDocumentException catch. * fix: deep-copy original_template consistently in expand_language_extensions Previously the non-cache path stored a direct reference to the caller's template dict. If any caller mutated it after the call, the stored original_template would be silently corrupted. Now deep-copies in both the had_language_extensions=True and False paths, consistent with the cache path behavior. * test: verify original_template is independent from caller's input * fix(test): Remove hardcoded SNS topic names to prevent AlreadyExists errors Remove TopicName and FifoTopic properties from Fn::ForEach SNS topics in the language-extensions test template. Hardcoded names caused integration test failures when topics from previous runs still existed. CloudFormation will now auto-generate unique names per stack. * fix: thread parent parameters into child template expansion When packaging a nested AWS::CloudFormation::Stack whose child template uses AWS::LanguageExtensions (e.g. Fn::ForEach over a CommaDelimitedList parameter), sam package previously passed only pseudo-parameters to the child expansion. Child parameters supplied via the parent's Properties.Parameters never reached the language-extensions pipeline, so dynamic artifact properties like CodeUri: ./services/${Name} could not be detected and no SAM-generated Mappings were produced. The packaged child template would ship with unresolved local paths and deployment would fail. CloudFormation's own behavior, verified against a live stack in us-west-2, does expand child Fn::ForEach against parent-supplied parameter values. sam package must match that behavior at package time. This change threads the merged parameter map (pseudo + CLI --parameter-overrides + global overrides) through PackageContext._export → Template → exporter instances → CloudFormationStackResource.do_export, and merges in the nested-stack resource's Parameters property (resolving any parent-context intrinsics via the existing create_default_intrinsic_resolver). Values that cannot be resolved locally (e.g. Fn::GetAtt) are dropped so they do not poison expand_language_extensions. Tests: - New TestResolveNestedStackParameters covers literal, ref, unresolvable, and empty inputs to the new helper. - New TestCloudFormationStackResourceChildExpansion asserts parent-passed Parameters reach expand_language_extensions end-to-end. - Existing Template(...) mock-call assertions updated to accept the new parameter_values kwarg. * fix: narrow exception handling in child template LE expansion The previous code caught bare Exception when expanding language extensions on a nested-stack child template, downgrading every failure (including SAM CLI bugs) to a single WARNING line. That masked real defects: any crash in the new library silently fell through to the non-expanded path, leaving the packaged child with unresolved local paths that CloudFormation could not resolve at deploy time. Two changes: 1. InvalidSamDocumentException (the documented failure surface of expand_language_extensions) keeps the warn-and-fallback behavior, but the message now explains the consequence: artifact URIs inside Fn::ForEach blocks will NOT be uploaded. 2. Any other exception is logged at ERROR with traceback (exc_info=True) and a pointer to file an issue. We still fall back so the rest of sam package keeps going, but the failure is observable to users running --debug and to telemetry instead of silently degrading. Also removed the check_using_language_extension branch that was dead: expand_language_extensions only raises InvalidSamDocumentException when it actually tried to expand, which implies the transform was present. Tests: - TestCloudFormationStackResourceExpansionErrorHandling covers both branches (warn vs error) and confirms the non-expanded fallback is still taken. * fix: narrow exception handling in _resolve_nested_stack_parameters The helper caught bare Exception when resolving intrinsics in a nested-stack resource's Parameters property, silently swallowing any resolver bug (TypeError, AttributeError, KeyError from malformed state). A user would see incorrect template expansion with no signal. Now: - UnresolvableReferenceError / InvalidTemplateException continue (expected 'can't resolve at package time' cases — e.g. Ref to a sibling resource). - Any other exception is logged at DEBUG with traceback so --debug surfaces it; the value is still dropped so packaging proceeds. Matches the narrowing pattern applied to do_export's expansion call site in the previous commit. * fix: gate MissingMappingKeyError on SAM-generated mapping names Deployer._create_deploy_error was wrapping *any* CloudFormation "Fn::FindInMap - Key X not found in Mapping Y" failure as MissingMappingKeyError, which emits SAM-specific re-package guidance ("Re-run 'sam package' with the same parameter values" and advice about Fn::ForEach dynamic artifact properties). That guidance is misleading when the Mapping was written by the user — a classic RegionMap → AMI lookup failure, for example, has nothing to do with packaging. Gate the wrapping on a fixed set of SAM-generated Mapping prefixes (SAMCodeUri, SAMImageUri, SAMContentUri, SAMDefinitionUri, SAMSchemaUri, SAMBodyS3Location, SAMDefinitionS3Location, SAMTemplateURL, SAMCode, SAMContent, SAMLayers). Matching also requires an upper-case letter after the prefix so names like SAMPLE/SAMSUNG don't accidentally match. User-authored Fn::FindInMap failures now fall through to the generic DeployFailedError with the raw CloudFormation message — accurate for the user's own mistake, without SAM CLI injecting irrelevant advice. Tests: - TestIsSamGeneratedMapping covers matching and non-matching names incl. SAMPLE / SAMSUNG / bare 'SAM' / lower-case variants. - TestCreateDeployError gains two cases: RegionMap-style user Mapping is NOT wrapped, and SAM-prefix-substring names are NOT wrapped. - Existing SAM-generated cases (SAMCodeUriServices, SAMCodeUriMyLoop) continue to wrap correctly. * fix: share Fn::ForEach and SAM-mapping helpers across modules Two issues flagged by the bot reviewer on PR aws#8637: [BUG] _update_sam_mappings_relative_paths in samcli/commands/_utils/ template.py used mapping_name.startswith("SAM") to decide whether to rewrite Mapping values as relative paths. That loose check corrupts customer-authored mappings whose names happen to start with SAM as a substring — SAMPLE, SAMSUNG, SAMCustomMapping, etc. Values would be silently rewritten with relative-path prefixes. [STYLE] infra_sync_executor.py had two hardcoded resource_logical_id.startswith("Fn::ForEach::") checks instead of using the shared is_foreach_key helper the rest of the codebase adopted in this PR. Maintenance risk if the detection logic ever changes. Consolidation: - Moved the precise SAM-mapping classifier (previously _is_sam_generated_mapping in samcli/commands/deploy/exceptions.py) into samcli/lib/cfn_language_extensions/utils.py as the public is_sam_generated_mapping so both deploy/exceptions.py and commands/_utils/template.py share the same implementation. - _update_sam_mappings_relative_paths now calls is_sam_generated_mapping — SAMPLE/SAMSUNG style names are correctly ignored. - infra_sync_executor.py now calls is_foreach_key at both sites. Tests: - test_skips_sam_prefix_substring_mappings in TestUpdateSamMappingsRelativePaths verifies SAMPLE/SAMSUNG/ SAMCustomMapping are left untouched. - Existing TestIsSamGeneratedMapping and TestCreateDeployError still pass via the re-export in deploy/exceptions.py. * chore: log swallowed exceptions in _partial_resolve at debug level _partial_resolve and _partial_resolve_find_in_map intentionally substitute AWS::NoValue / partial args for false-condition resources when the resolver raises. The swallow matches the Kotlin reference implementation and is not changing. The problem is debuggability: if a resolver bug or unexpected type slips through, the swallow hides it and the user sees a template with AWS::NoValue substitutions that look intentional. Adding LOG.debug with exc_info=True surfaces the traceback under --debug and in telemetry without changing any runtime behavior. Three swallow sites covered: - Ref substitution fallback - Generic Fn::* substitution fallback - Fn::FindInMap secondary fallback inside _partial_resolve_find_in_map The Re-raised InvalidTemplateException branch in _partial_resolve_find_in_map is unchanged — that path must error. * fix: declare parent_parameter_values on Resource base class Previously Template.export() set parent_parameter_values as a duck-typed attribute on exporter instances, and CloudFormationStackResource.do_export read it via getattr(self, 'parent_parameter_values', None). The attribute wasn't declared on any base class, so: - Typos on either the write or read side wouldn't be caught by static analysis or IDE completion. - Any future exporter class for a nested-stack resource type that didn't explicitly handle this attribute would silently get None and skip parameter propagation, producing wrong language-extension expansion for its child template. Declared parent_parameter_values: Optional[Dict] = None on Resource. The read site in artifact_exporter.do_export now uses direct attribute access instead of getattr() with a default. * fix: route all CFN ClientError paths through _create_deploy_error Previously only wait_for_execute, create_and_wait_for_changeset's internal callers (via sync's catch block), and sync itself routed CloudFormation ClientError through _create_deploy_error, which detects the SAM-generated Fn::FindInMap key-not-found signature and surfaces a user-friendly MissingMappingKeyError. create_stack, update_stack, and create_and_wait_for_changeset's outer catch block still raised the raw DeployFailedError. On sam sync the outer catch usually won, but sam deploy / sam package + deploy paths could hit the synchronous validation error from create_stack / update_stack directly and bypass the helpful MissingMappingKeyError entirely — users saw the raw CloudFormation message with no re-package guidance. All five previously raw call sites now go through self._create_deploy_error. The wrapper is a no-op for non-matching errors (including user-authored Fn::FindInMap failures — gated by is_sam_generated_mapping), so no regression for any other error path. Also left execute_changeset and has_stack untouched — neither can surface a Fn::FindInMap failure (has_stack is a pre-flight check; execute_changeset only kicks off the change set, stack-events-phase errors come back through wait_for_execute which was already wired). Tests: TestCreateDeployErrorRouting covers create_stack and update_stack wrapping a SAM-mapping error, and confirms user RegionMap-style errors still fall through to DeployFailedError. * perf: replace defensive deepcopy with deep_freeze immutability expand_language_extensions was deep-copying the template 4-10 times per invocation for defensive isolation (cache put, cache get, no-LE path, LE path). For large templates this is measurable overhead. Replace with a one-time deep_freeze that converts dicts to MappingProxyType and lists to tuples. Frozen templates are truly immutable — any mutation attempt raises TypeError immediately instead of silently corrupting shared state. Callers that need mutable copies use deep_thaw (not copy.deepcopy which cannot pickle MappingProxyType on Python 3.13). Savings: - No-LE fast path: 4 deepcopies -> 1 deep_freeze (same O(n) cost) - Cache hit: 2 deepcopies -> 0 (return cached frozen result directly) - Cold LE path: 3 deepcopies -> 2 deep_freezes (expanded + original) - Cache put: 2 deepcopies -> 0 (frozen values are safe to share) Also: - Fixed isinstance(x, dict) check in build_context to accept Mapping so frozen MappingProxyType templates pass the type check. - Updated callers (build_context, package_context, artifact_exporter, language_extensions_packaging, wrapper) to use deep_thaw instead of copy.deepcopy when they need mutable copies. - Updated tests to verify immutability contract (TypeError on mutation) instead of asserting independent mutable copies. * fix: freeze DynamicArtifactProperty and use tuple for properties list The cached LanguageExtensionResult is returned directly on cache hit. While template fields are deeply frozen via MappingProxyType, the dynamic_artifact_properties field was a mutable list of mutable DynamicArtifactProperty dataclass instances. A caller mutating the list or its elements would corrupt the cached value for all subsequent hits. - DynamicArtifactProperty is now frozen=True - dynamic_artifact_properties field changed from List to Tuple with default () - Construction sites pass tuple() instead of list This completes the immutability contract: all fields of a cached LanguageExtensionResult are now truly immutable. * perf: skip cache and deep_freeze for non-LE templates When a template has no AWS::LanguageExtensions transform, the work being cached is a single dict lookup on template.get('Transform') — O(1). The previous code still deep_freeze'd the entire template O(n) and stored it in the module-level cache, which was pure overhead for the vast majority of users who don't use language extensions. Now the no-LE path returns a LanguageExtensionResult wrapping the caller's original dict directly — zero copies, zero cache entries. Callers that need to mutate (build_context, package_context) already deep_thaw before mutating, so the aliasing is safe. The LE path still freezes and caches as before. * refactor: remove unused expansion cache The module-level _expansion_cache in sam_integration.py was designed to avoid redundant template expansion across multiple calls within a single CLI invocation. In practice, the cache was never hit: - sam build: calls expand_language_extensions once (via get_stacks) - sam package: calls it twice (get_stacks + _export) — the only potential hit, but the ~10ms saved is noise vs seconds of S3 uploads - sam validate/deploy/local: call it once - sam sync --watch: clears the cache before each reload The cache added ~40 lines of complexity (module-level mutable global, hash function, eviction logic, clear_expansion_cache calls in watch_manager and sam_function_provider) plus a thread-safety concern flagged in the review, all for negligible benefit. Removed: - _expansion_cache, _MAX_CACHE_SIZE, _hash_params, _cache_put, clear_expansion_cache from sam_integration.py - clear_expansion_cache() calls from watch_manager.py and sam_function_provider.py - template_path parameter from expand_language_extensions (only used for cache keying) and all call sites - Exports from __init__.py - TestExpansionCache class (replaced with two focused immutability tests) The deep_freeze on the LE path still ensures immutability without needing cache isolation. * refactor: revert deep_freeze/deep_thaw back to copy.deepcopy With the expansion cache removed in the previous commit, deep_freeze and deep_thaw serve no purpose. The freeze/thaw cycle was: 1. expand_language_extensions: deep_freeze the result (O(n)) 2. Every caller: deep_thaw to get a mutable copy (O(n)) Two O(n) walks that cancel each other out. Without shared cached state to protect, plain copy.deepcopy is simpler, well-understood, and doesn't require a custom inverse function. Reverted all call sites back to copy.deepcopy: - sam_integration.py: deep_freeze → copy.deepcopy - build_context.py, package_context.py, artifact_exporter.py, language_extensions_packaging.py, wrapper.py, intrinsic_property_resolver.py, infra_sync_executor.py, sam_stack_provider.py: deep_thaw → copy.deepcopy - Removed deep_freeze, deep_thaw, MappingProxyType from utils.py - Reverted isinstance(x, (dict, Mapping)) back to isinstance(x, dict) - Reverted dynamic_artifact_properties from Tuple back to List - Updated tests to remove frozen-behavior assertions DynamicArtifactProperty remains frozen=True (independently useful as a value object that shouldn't be mutated). * perf: pass template dict directly to Template, skip yaml round-trip Template.__init__ now accepts an optional template_dict parameter. When provided, it skips both file reading and yaml_parse, avoiding the deepcopy → yaml_dump → yaml_parse round-trip that package_context and artifact_exporter were doing. For large templates with many Fn::ForEach expansions, this eliminates two O(n) serialization passes (yaml_dump + yaml_parse) per export. * fix: initialize parent_parameter_values in __init__ and fully init Template Two issues flagged by the bot reviewer: 1. Resource.parent_parameter_values was declared as a class-level attribute but not set in __init__. Any code path that instantiates a CloudFormationStackResource and calls do_export() without going through Template._export_resources() would get AttributeError. Now initialized to None in __init__. 2. Template.__init__ with template_dict left template_dir and code_signer uninitialized, requiring callers to manually patch the object after construction. Now the template_dict branch derives template_dir from template_path and sets code_signer, eliminating the partial-initialization window. Removed the manual assignments from package_context._export and CloudFormationStackResource.do_export. * fix: merge pseudo-parameters in sam_template_validator before expansion expand_language_extensions was called with only self.parameter_overrides, missing pseudo-parameters (AWS::Region, AWS::AccountId, etc.). Every other caller merges IntrinsicsSymbolTable.DEFAULT_PSEUDO_PARAM_VALUES first. Without pseudo-params, templates using Ref: AWS::Region in Fn::ForEach collections or Fn::Sub expressions would fail during sam validate but work correctly with sam build and sam package. * style: use is_foreach_key() consistently in build_context Three sites in build_context.py used hardcoded startswith("Fn::ForEach::") instead of the shared is_foreach_key() utility. Replaced for consistency with the rest of the codebase. * chore: consolidate property lists, mark unused methods, document validator Three cleanup items from the review: aws#5: _ARTIFACT_PATH_PROPERTIES in template.py was a hand-maintained set of property names that duplicated PACKAGEABLE_RESOURCE_ARTIFACT_PROPERTIES in models.py. Replaced with a frozenset derived from the canonical source so the two cannot drift. aws#6: SamTranslatorWrapper.get_original_template() and get_dynamic_artifact_properties() are only called in tests — callers now consume LanguageExtensionResult directly. Added a TODO comment rather than deleting (would require removing ~20 test methods across 3 files — not worth the churn in this PR). aws#7: Documented that get_translated_template_if_valid mutates self.sam_template when language extensions are present. The mutation is idempotent (re-expanding an already-expanded template is a no-op) so double-calling is safe, but the comment makes the contract explicit. * fix: accept digit-leading loop names in is_sam_generated_mapping A ForEach loop named Fn::ForEach::1stBatch produces a mapping name like SAMCodeUri1stBatch. The character after the prefix (SAMCodeUri) is '1', which failed the uppercase-only check. This caused _update_sam_mappings_relative_paths to skip the mapping (broken artifact paths in build output) and _create_deploy_error to miss it (raw CFN error instead of helpful re-package guidance). Now accepts digits as well as uppercase letters after the prefix. All SAM-generated prefixes end in a letter (CodeUri, ImageUri, etc.) so a digit unambiguously indicates the start of the nesting path. * fix: remove dead template_path param, wrap ValueError in InvalidTemplateException Two bot-flagged issues: 1. SamTemplateValidator.template_path was stored but never read — leftover from the removed expansion cache. Removed the parameter from __init__ and both callers (validate.py, resource_mapping_producer.py). 2. sanitize_resource_key_for_mapping raised bare ValueError which wasn't caught by PackageContext._export (only catches InvalidSamDocumentException). A ForEach resource key with no static component (e.g. ${Svc}) would produce an unhandled traceback. Now raises InvalidTemplateException which is caught and converted to a user-friendly error by the existing error handling in expand_language_extensions. * chore: echo note when make test excludes cfn_language_extensions Developers running 'make test' locally now see a visible note that cfn_language_extensions tests are excluded, with a pointer to 'make test-all' for full coverage. * fix: don't error on invalid FindInMap in false-condition resources CloudFormation skips all validation for resources with false conditions, including Fn::FindInMap lookups against non-existent mapping keys. _partial_resolve_find_in_map was re-raising InvalidTemplateException for string-literal keys that don't exist in Mappings, causing SAM CLI to reject valid templates that CloudFormation would accept. Now returns the partially-resolved FindInMap form instead of erroring, matching CloudFormation's behavior. Also updated LanguageExtensionResult docstring to accurately document the no-LE path aliasing contract (expanded_template and original_template share the same reference when had_language_extensions is False). Moved templateFnInMapInWhenFalseConditionInResourceWithInvalidStringPath from ERROR_TEMPLATES_PASSING to SUCCESS_TEMPLATES_PASSING in the Kotlin compatibility tests, with expected output. * test: improve coverage for utils, artifact_exporter, and Template init New test file test_utils.py covers: - derive_partition (standard, cn, gov regions) - derive_url_suffix (standard, cn) - is_foreach_key (valid, regular, non-string) - iter_regular_resources (skips ForEach/non-dict, empty, missing) - is_sam_generated_mapping (bare prefix, lowercase, layers, digits) New tests in test_artifact_exporter.py: - _resolve_nested_stack_parameters: Ref to resource dropped, Fn::Sub partial resolution, unresolvable Ref dropped - Template.__init__ with template_dict sets template_dir and code_signer - Template.__init__ with template_str parses YAML correctly utils.py coverage: 81% -> 94% artifact_exporter.py coverage: 91% -> 92% Overall: 94.01% -> 94.02% (7427 -> 7431 tests) * fix: use direct attribute access for original_template_dict, guard parent_dir=None Two bot-flagged issues: 1. build_context.py used getattr(stack, 'original_template_dict', None) but Stack now always declares this attribute. Replaced with direct stack.original_template_dict access. 2. Template.__init__ with template_dict crashed if parent_dir was None (make_abs_path raises TypeError). Added guard: falls back to os.getcwd() when parent_dir is not provided. * docs: add CloudFormation Language Extensions support documentation Covers: - How the two-phase expansion works (LE then SAM transform) - Fn::ForEach usage with static and parameter-based collections - Dynamic artifact properties and SAM-generated Mappings - Nested stacks with parent-supplied parameters - Nested Fn::ForEach (up to 5 levels) - &{identifier} syntax for logical ID sanitization - Supported intrinsic functions table - Limitations (collection resolvability, package-time fixation) - Telemetry * feat(sync): add CloudFormation Language Extensions support for sam sync Extend sam sync to fully support templates with AWS::LanguageExtensions and Fn::ForEach. Previously, sync only had guard-level support that skipped ForEach keys to avoid crashes. Changes: - Sanitize artifact properties inside Fn::ForEach body resources during template comparison so code-only changes don't trigger unnecessary infra syncs. Handles all code-syncable resource types: functions (zip + image), layers, APIs, HTTP APIs, state machines, and nested stacks. - Detect code changes in ForEach-generated resources by expanding both the current and deployed templates, then comparing expanded resources. Changed resources are queued for code sync instead of triggering a full CloudFormation deployment. - Code sync (sam sync --code) and watch mode (sam sync --watch) already work via SamLocalStackProvider.get_stacks() LE expansion. All sync flows (ADL, Image, API, Layer, StateMachine) work with expanded ForEach resource identifiers via CFN physical ID mapping. * style: fix formatting issues across codebase * Revert "style: fix formatting issues across codebase" This reverts commit ece6393. * style: fix black formatting in test_infra_sync_executor * fix: address review bot comments — exc_info, lowercase loop names, warning level - Add exc_info=True to bare except in _detect_foreach_code_changes so --debug output includes the traceback - Accept lowercase loop names in is_sam_generated_mapping by using nxt.isalnum() instead of restricting to uppercase + digits - Upgrade _resolve_nested_stack_parameters unexpected error logging from debug to warning level so users see it without --debug * fix: Template.__init__ template_str branch missing attrs, declare parent_parameter_values - Set template_dir and code_signer in the elif template_str branch of Template.__init__ to prevent AttributeError when export() is called - Declare parent_parameter_values on CloudFormationStackResource class instead of relying on dynamic monkey-patching in Template.export() * fix: upgrade LE expansion failure to WARNING, include exception in parameter warning - _detect_foreach_code_changes: upgrade from debug to warning level so users see when ForEach code change detection is skipped - _resolve_nested_stack_parameters: include exception message in the warning so users can diagnose without --debug * fix: include exception message in LE expansion warning * fix: pass parameter defaults to expand_language_extensions in code change detection Without parameter values, Fn::ForEach collections using Ref to parameters (e.g. {"Ref": "ServiceNames"}) fail to resolve during expansion, causing _detect_foreach_code_changes to silently skip code change detection for all ForEach-generated resources. * fix: extract parameter defaults independently for each template in code change detection Use each template's own Parameters section to extract defaults, so expansion produces correct results even if parameter definitions changed between the current and deployed template versions. * fix: resolve PLR2004, PLR0911, PLR1714 lint issues in cfn_language_extensions Replace blanket per-file ruff suppression with proper fixes: - PLR2004: Extract magic numbers into named class constants for CloudFormation intrinsic function argument counts - PLR1714: Add is_intrinsic_key() helper to utils.py to replace repeated startswith/equality comparison chains - PLR0911: Refactor _partial_resolve into smaller helper methods to reduce return statement count Reuse existing FOREACH_REQUIRED_ELEMENTS constant from utils.py for ForEach processor validation. * fix: address aws-sam-cli-bot code review comments - Add us-iso- and us-isob- partition/URL suffix handling to derive_partition and derive_url_suffix - Remove phantom caching references from module and function docstrings in sam_integration.py (no caching logic exists) - Remove redundant copy.deepcopy(expanded_template) since process_template_for_sam_cli already returns an independent object - Update deep-copy comment to explain caller isolation intent - Refactor broad except to catch LangExtInvalidTemplateException directly instead of catching Exception and using isinstance - Add docstring note about parameter defaults limitation in _detect_foreach_code_changes * fix: add aws-eusc partition support, remove original_template_dict from Stack.__eq__ - Add eusc- region prefix handling to derive_partition (aws-eusc) and derive_url_suffix (amazonaws.eu) for the AWS European Sovereign Cloud - Revert us-iso-/us-isob- ADC partition branches — not supported elsewhere in the SAM CLI codebase - Remove original_template_dict from Stack.__eq__ — it is operational metadata for deployment, not part of the stack's semantic identity. Including it caused stacks created via different code paths to compare as unequal even when semantically identical
1 parent a90d18f commit 2e21eb2

425 files changed

Lines changed: 53075 additions & 171 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.coveragerc_no_lang_ext

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Coverage config for `make test` which skips cfn_language_extensions tests.
2+
# Extends the base .coveragerc and also excludes the language extensions source
3+
# so coverage % isn't penalized when those tests are skipped.
4+
[run]
5+
branch = True
6+
omit =
7+
samcli/lib/cfn_language_extensions/*
8+
# Inherited from .coveragerc
9+
samcli/lib/iac/plugins_interfaces.py
10+
samcli/lib/init/templates/*
11+
samcli/hook_packages/terraform/copy_terraform_built_artifacts.py
12+
[report]
13+
exclude_lines =
14+
pragma: no cover
15+
raise NotImplementedError.*

Makefile

Lines changed: 24 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -26,11 +26,20 @@ init-latest-release:
2626
bash tests/install-sam-cli-binary.sh
2727

2828
test:
29-
# Run unit tests and fail if coverage falls below 94%
29+
# Run unit tests (excluding cfn_language_extensions) and fail if coverage falls below 94%
30+
@echo "NOTE: Excluding cfn_language_extensions tests. Use 'make test-all' for full coverage."
31+
pytest --cov samcli --cov schema --cov-report term-missing --cov-fail-under 94 tests/unit --ignore=tests/unit/lib/cfn_language_extensions --cov-config=.coveragerc_no_lang_ext
32+
33+
test-lang-ext:
34+
# Run cfn_language_extensions unit tests with coverage
35+
pytest --cov samcli.lib.cfn_language_extensions --cov-report term-missing --cov-fail-under 94 tests/unit/lib/cfn_language_extensions
36+
37+
test-all:
38+
# Run all unit tests including cfn_language_extensions
3039
pytest --cov samcli --cov schema --cov-report term-missing --cov-fail-under 94 tests/unit
3140

3241
test-cov-report:
33-
# Run unit tests with html coverage report
42+
# Run all unit tests with html coverage report
3443
pytest --cov samcli --cov schema --cov-report html --cov-fail-under 94 tests/unit
3544

3645
integ-test:
@@ -58,7 +67,17 @@ lint:
5867
mypy --exclude /testdata/ --exclude /init/templates/ --no-incremental setup.py samcli tests schema
5968

6069
# Command to run everytime you make changes to verify everything works
61-
dev: lint test
70+
# Runs test-all if cfn_language_extensions files changed, otherwise test
71+
dev: lint
72+
@if git diff --name-only origin/develop... 2>/dev/null | grep -qE 'cfn_language_extensions/'; then \
73+
echo "Detected cfn_language_extensions changes — running all tests"; \
74+
$(MAKE) test-all; \
75+
else \
76+
$(MAKE) test; \
77+
fi
78+
79+
# Run full verification including language extensions tests
80+
dev-all: lint test-all
6281

6382
black:
6483
black setup.py samcli tests schema
@@ -72,8 +91,8 @@ format: black
7291
schema:
7392
python -m schema.make_schema
7493

75-
# Verifications to run before sending a pull request
76-
pr: init schema black-check dev
94+
# Verifications to run before sending a pull request — runs ALL tests
95+
pr: init schema black-check lint test-all
7796

7897
# Update all reproducible requirements using uv (can run from any platform)
7998
update-reproducible-reqs:

docs/cfn-language-extensions.md

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
# CloudFormation Language Extensions Support
2+
3+
SAM CLI now supports templates that use the `AWS::LanguageExtensions` transform, including `Fn::ForEach`, `Fn::Length`, `Fn::ToJsonString`, and `Fn::FindInMap` with `DefaultValue`.
4+
5+
## How it works
6+
7+
When SAM CLI detects `AWS::LanguageExtensions` in a template's `Transform` section, it expands language extension constructs locally before running SAM transforms. This enables `sam build`, `sam package`, `sam deploy`, `sam sync`, `sam validate`, `sam local invoke`, and `sam local start-api` to work with templates that use these constructs.
8+
9+
The expansion happens in two phases:
10+
11+
1. **Phase 1 (Language Extensions)**`Fn::ForEach` loops are expanded, intrinsic functions are resolved where possible, and the template is converted to standard CloudFormation.
12+
2. **Phase 2 (SAM Transform)** — The expanded template is processed by the SAM Translator as usual.
13+
14+
The original template (with `Fn::ForEach` intact) is preserved for CloudFormation deployment, since CloudFormation processes the `AWS::LanguageExtensions` transform server-side.
15+
16+
## Fn::ForEach
17+
18+
`Fn::ForEach` generates multiple resources, conditions, or outputs from a single template definition:
19+
20+
```yaml
21+
Transform: AWS::LanguageExtensions
22+
23+
Parameters:
24+
ServiceNames:
25+
Type: CommaDelimitedList
26+
Default: "Users,Orders,Products"
27+
28+
Resources:
29+
Fn::ForEach::Services:
30+
- Name
31+
- !Ref ServiceNames
32+
- ${Name}Function:
33+
Type: AWS::Serverless::Function
34+
Properties:
35+
Handler: index.handler
36+
Runtime: python3.12
37+
CodeUri: ./services/${Name}
38+
```
39+
40+
Running `sam build` expands this into `UsersFunction`, `OrdersFunction`, and `ProductsFunction`, each built from its respective source directory.
41+
42+
### Dynamic artifact properties
43+
44+
When a packageable property (like `CodeUri`, `ContentUri`, `ImageUri`) uses a loop variable (e.g., `./services/${Name}`), SAM CLI generates a CloudFormation `Mappings` section that maps each collection value to its S3 URI. The `Fn::ForEach` body is rewritten to use `Fn::FindInMap` so CloudFormation can resolve the correct artifact at deploy time.
45+
46+
For example, after `sam package`:
47+
48+
```yaml
49+
Mappings:
50+
SAMCodeUriServices:
51+
Users:
52+
CodeUri: s3://my-bucket/abc123
53+
Orders:
54+
CodeUri: s3://my-bucket/def456
55+
Products:
56+
CodeUri: s3://my-bucket/ghi789
57+
58+
Resources:
59+
Fn::ForEach::Services:
60+
- Name
61+
- !Ref ServiceNames
62+
- ${Name}Function:
63+
Type: AWS::Serverless::Function
64+
Properties:
65+
Handler: index.handler
66+
Runtime: python3.12
67+
CodeUri: !FindInMap [SAMCodeUriServices, !Ref Name, CodeUri]
68+
```
69+
70+
### Parameter-based collections
71+
72+
When the `Fn::ForEach` collection is a parameter reference (`!Ref ServiceNames`), the collection values are resolved at package time from:
73+
74+
1. `--parameter-overrides` passed to `sam build` or `sam package`
75+
2. The parameter's `Default` value in the template
76+
77+
**Important:** If you change the parameter value at deploy time (e.g., adding a new service), you must re-package first so the Mappings include entries for the new values.
78+
79+
```bash
80+
# Package with the values you intend to deploy with
81+
sam package --parameter-overrides ServiceNames="Users,Orders,Products"
82+
83+
# Deploy with the same values
84+
sam deploy --parameter-overrides ServiceNames="Users,Orders,Products"
85+
```
86+
87+
### Nested stacks
88+
89+
`Fn::ForEach` in nested stack templates (`AWS::CloudFormation::Stack`) is supported. SAM CLI passes the parent stack's `Parameters` property to the child template expansion, so child `Fn::ForEach` collections that reference parent-supplied parameters resolve correctly.
90+
91+
```yaml
92+
# parent.yaml
93+
Resources:
94+
ChildStack:
95+
Type: AWS::CloudFormation::Stack
96+
Properties:
97+
TemplateURL: ./child.yaml
98+
Parameters:
99+
ServiceNames: "Users,Orders,Products"
100+
```
101+
102+
### Nested Fn::ForEach
103+
104+
Up to 5 levels of nesting are supported, matching CloudFormation's limit:
105+
106+
```yaml
107+
Resources:
108+
Fn::ForEach::Envs:
109+
- Env
110+
- [Dev, Staging, Prod]
111+
- Fn::ForEach::Services:
112+
- Svc
113+
- [Users, Orders]
114+
- ${Env}${Svc}Function:
115+
Type: AWS::Serverless::Function
116+
Properties:
117+
CodeUri: ./services/${Svc}
118+
Environment:
119+
Variables:
120+
STAGE: !Ref Env
121+
```
122+
123+
### &{identifier} syntax
124+
125+
The `&{identifier}` syntax strips non-alphanumeric characters from the substituted value, useful for generating valid logical IDs from values like IP addresses:
126+
127+
```yaml
128+
Fn::ForEach::Hosts:
129+
- IP
130+
- ["10.0.0.1", "10.0.0.2"]
131+
- Host&{IP}:
132+
Type: AWS::EC2::Instance
133+
# Expands to Host10001, Host10002
134+
```
135+
136+
## Supported intrinsic functions
137+
138+
The following intrinsic functions are resolved locally during expansion:
139+
140+
| Function | Description |
141+
|----------|-------------|
142+
| `Fn::ForEach` | Loop expansion |
143+
| `Fn::Length` | Returns count of list elements |
144+
| `Fn::ToJsonString` | Converts value to JSON string |
145+
| `Fn::FindInMap` | Map lookup (with optional `DefaultValue`) |
146+
| `Fn::If` | Conditional value selection |
147+
| `Fn::Sub` | String substitution |
148+
| `Fn::Join` | String concatenation |
149+
| `Fn::Split` | String splitting |
150+
| `Fn::Select` | List element selection |
151+
| `Fn::Base64` | Base64 encoding |
152+
| `Fn::Equals` / `Fn::And` / `Fn::Or` / `Fn::Not` | Condition evaluation |
153+
| `Ref` | Parameter and pseudo-parameter references |
154+
155+
Functions that require deployed resources (`Fn::GetAtt`, `Fn::ImportValue`, `Fn::GetAZs`) are preserved for CloudFormation to resolve at deploy time.
156+
157+
## Limitations
158+
159+
- **Collections must be resolvable at build/package time.** `Fn::ForEach` collections that use `Fn::GetAtt`, `Fn::ImportValue`, or SSM/Secrets Manager dynamic references cannot be expanded locally. Use a parameter with `--parameter-overrides` instead.
160+
- **Parameter values are fixed at package time.** If you change `--parameter-overrides` at deploy time without re-packaging, the Mappings won't include entries for new values and deployment will fail.
161+
- **`DeletionPolicy` and `UpdateReplacePolicy`** are validated and resolved during expansion. They support `Ref` to parameters but not other intrinsic functions.
162+
163+
## Telemetry
164+
165+
SAM CLI tracks usage of `AWS::LanguageExtensions` via the `CFNLanguageExtensions` telemetry feature flag. No template content is transmitted.

pyproject.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,7 @@ max-statements = 80
132132
"integration_uri.py" = ["E501"] # ARNs are long.
133133
"app.py" = ["E501"] # Doc links are long.
134134

135+
135136
[tool.black]
136137
line-length = 120
137138
target_version = ['py310', 'py311', 'py312', 'py313']

0 commit comments

Comments
 (0)