Commit 2e21eb2
authored
feat: Add CloudFormation Language Extensions support (Fn::ForEach) (aws#8637)
* feat: add CloudFormation Language Extensions processing library
Implement a local CloudFormation Language Extensions processor supporting:
- Fn::ForEach loop expansion in Resources, Conditions, and Outputs
- Fn::Length, Fn::ToJsonString intrinsic functions
- Fn::FindInMap with DefaultValue support
- Conditional DeletionPolicy/UpdateReplacePolicy
- Nested ForEach depth validation (max 5 levels)
- Partial resolution mode preserving unresolvable references
Pipeline architecture: TemplateParsingProcessor -> ForEachProcessor ->
IntrinsicResolverProcessor -> DeletionPolicyProcessor ->
UpdateReplacePolicyProcessor
Includes comprehensive unit tests and CloudFormation compatibility suite.
* feat: integrate language extensions into SAM CLI
Wire the language extensions library into SAM CLI with two-phase architecture:
- Phase 1: expand_language_extensions() -> LanguageExtensionResult
- Phase 2: SamTranslatorWrapper.run_plugins() (SAM transform only)
Key components:
- expand_language_extensions() canonical entry point
- SamTranslatorWrapper receives pre-expanded template (Phase 2 only)
- SamLocalStackProvider.get_stacks() calls expand_language_extensions()
- SamTemplateValidator calls expand_language_extensions()
- DynamicArtifactProperty dataclass for Mappings transformation
- Fn::ForEach guards in artifact_exporter, normalizer, cdk/utils
* feat: add language extensions support to sam build
- _get_template_for_output() preserves Fn::ForEach in build output
- _update_foreach_artifact_paths() generates Mappings for dynamic
artifact properties with per-function build paths
- Recursive nested Fn::ForEach support
- ForEach-aware path resolution skips Docker image URIs
Test templates: static CodeUri, dynamic CodeUri, parameter collections,
nested stacks, nested ForEach, dynamic ImageUri, depth validation.
* feat: add language extensions support to sam package and deploy
Package:
- _export() calls expand_language_extensions() for Phase 1
- Preserves Fn::ForEach in packaged template with S3 URIs
- Generates Mappings for dynamic artifact properties
- _find_artifact_uri_for_resource() handles all export formats:
string, {S3Bucket,S3Key}, {Bucket,Key}, {ImageUri}
- Recursive nested Fn::ForEach support
- Warning for parameter-based collections
Deploy:
- Uploads original unexpanded template to CloudFormation
- Clear error for missing Mapping keys
Integration tests for CodeUri, ContentUri, DefinitionUri, ImageUri,
BodyS3Location across all packageable resource types.
* test: add validate, local invoke, and start-api integration tests
- sam validate: valid ForEach, invalid syntax, cloud-dependent collections,
dynamic CodeUri, nested depth validation (5 valid, 6 invalid)
- sam local invoke: expanded function names from ForEach
- sam local start-api: ForEach-generated API endpoints
* feat: add telemetry tracking for CloudFormation Language Extensions
Track CFNLanguageExtensions as a UsedFeature event when templates
with AWS::LanguageExtensions transform are expanded. Emitted once
per expansion in expand_language_extensions().
* test: trim language extensions integration tests to essential cases
Remove redundant and AWS-dependent integration tests, keeping 9 essential
tests across build, package, validate, local invoke, and start-api.
Delete 34 orphaned testdata directories.
* fix: Fn::Equals uses string comparison to match CloudFormation behavior
YAML parsing produces Python booleans for bare true/false values, but
parameter overrides from --parameter-overrides are always strings.
Fn::Equals was using Python == which returns False for 'true' == True.
CloudFormation Fn::Equals performs string comparison, so convert both
operands to their string representations before comparing. Booleans
are lowercased to produce 'true'/'false' matching CFN serialization.
* fix: scope intrinsic resolution to Resources, Conditions, and Outputs only
Language extension functions are only supported in these three sections
per AWS::LanguageExtensions transform documentation. Previously the
intrinsic resolver also processed Parameters, Mappings, Metadata, etc.
* refactor: rename iter_resources to iter_regular_resources
The name iter_regular_resources better conveys that ForEach blocks are
skipped. Removes the backward-compatible alias.
* refactor: deduplicate _to_boolean and fix TOCTOU in expansion cache
Extract duplicated _to_boolean logic from condition_resolver.py and
fn_if.py into IntrinsicFunctionResolver.to_boolean() static method.
Replace os.path.isfile() + os.path.getmtime() two-step check with a
single try/except around getmtime() to eliminate the race condition.
* test: remove integration tests referencing deleted test data
Remove 9 integration tests whose test data directories were removed in
an earlier commit: validate/language-extensions/, buildcmd/language-
extensions-dynamic-imageuri/, language-extensions-foreach/, and
language-extensions-nested-foreach-{valid,invalid}/.
* refactor: address PR review comments on style and test quality
- Move inline imports to top level in sam_stack_provider.py and test_template.py
- Add missing assertion in test_handles_empty_mappings
- Uncomment Fn::ForEach::Topics block to test non-Lambda resource types
- Update mock patch paths to match top-level import locations
* feat: add &{identifier} syntax support in Fn::ForEach expansion
CloudFormation supports &{identifier} syntax which strips non-alphanumeric
characters from the substituted value, useful for generating valid logical
IDs from values containing dots, slashes, etc. (e.g., IP addresses).
Previously only ${identifier} was handled. This adds &{identifier}
substitution in _substitute_identifier alongside the existing ${} handling.
Includes 4 unit tests covering keys, values, mixed syntax, and no-op case.
* fix: remove unsupported list-of-lists identifier format from Fn::ForEach
CloudFormation does not support list identifiers in Fn::ForEach — deploying
a template with a list identifier returns 'Fn::ForEach layout is incorrect'.
Verified by deploying a test template to CFN.
Remove _resolve_identifiers, _resolve_collection_item, and all list-of-lists
handling from ForEachProcessor. The identifier is now validated as a single
non-empty string, matching the CFN spec.
Also removes 30 list-of-lists compatibility test template files.
* fix: use proper type annotation for intrinsic_resolver parameter
Change Optional[Any] to Optional["IntrinsicResolver"] using TYPE_CHECKING
import to avoid circular dependencies while enabling proper type checking.
* fix: correct _process_section docstring about which sections it handles
* test: add tier1_extra markers and replace NamedTemporaryFile
- Add @pytest.mark.tier1_extra to invoke, start-api, and build language
extensions tests for windows/finch coverage
- Replace NamedTemporaryFile with explicit path in test_deploy_command.py
to avoid Windows file locking issues
* fix: narrow exception handling and bound expansion cache size
- artifact_exporter.py: catch InvalidSamDocumentException instead of bare
Exception so programming bugs propagate instead of being silently swallowed
- sam_integration.py: add _MAX_CACHE_SIZE=32 with FIFO eviction to prevent
unbounded memory growth in long-running processes (sam sync --watch)
* test: add tests for cache eviction and narrowed exception handling
- test_cache_evicts_oldest_when_full: verifies FIFO eviction at _MAX_CACHE_SIZE
- test_export_cloudformation_stack_unexpected_exception_propagates: verifies
unexpected exceptions (TypeError etc.) are not silently swallowed
- Updated existing expansion_failure test to use InvalidSamDocumentException
* fix: improve error visibility and add UTF-8 encoding for file reads
- artifact_exporter.py: log at WARNING (not DEBUG) when a child template
declares AWS::LanguageExtensions but expansion fails, so users see the
error instead of discovering it at deploy time
- artifact_exporter.py, package_context.py: add encoding='utf-8' to open()
calls to prevent Windows locale encoding issues with non-ASCII templates
* fix: update test mocks to expect encoding='utf-8' parameter
* fix: MissingMappingKeyError now extends DeployFailedError
Previously extended UserException directly, so it bypassed the
except DeployFailedError handlers in deploy_context.py. This meant
rollback-delete on failure wouldn't execute and sync errors weren't
logged before re-raising.
* fix: catch unexpected exceptions in child template expansion gracefully
An unexpected error from expand_language_extensions on a child template
should not abort packaging of the entire parent stack. The fallback
(using original template) is always safe since CFN handles expansion
server-side. Now catches Exception with WARNING log as a second handler
after the specific InvalidSamDocumentException catch.
* fix: deep-copy original_template consistently in expand_language_extensions
Previously the non-cache path stored a direct reference to the caller's
template dict. If any caller mutated it after the call, the stored
original_template would be silently corrupted. Now deep-copies in both
the had_language_extensions=True and False paths, consistent with the
cache path behavior.
* test: verify original_template is independent from caller's input
* fix(test): Remove hardcoded SNS topic names to prevent AlreadyExists errors
Remove TopicName and FifoTopic properties from Fn::ForEach SNS topics
in the language-extensions test template. Hardcoded names caused
integration test failures when topics from previous runs still existed.
CloudFormation will now auto-generate unique names per stack.
* fix: thread parent parameters into child template expansion
When packaging a nested AWS::CloudFormation::Stack whose child template uses
AWS::LanguageExtensions (e.g. Fn::ForEach over a CommaDelimitedList parameter),
sam package previously passed only pseudo-parameters to the child expansion.
Child parameters supplied via the parent's Properties.Parameters never
reached the language-extensions pipeline, so dynamic artifact properties
like CodeUri: ./services/${Name} could not be detected and no SAM-generated
Mappings were produced. The packaged child template would ship with
unresolved local paths and deployment would fail.
CloudFormation's own behavior, verified against a live stack in us-west-2,
does expand child Fn::ForEach against parent-supplied parameter values.
sam package must match that behavior at package time.
This change threads the merged parameter map (pseudo + CLI
--parameter-overrides + global overrides) through PackageContext._export →
Template → exporter instances → CloudFormationStackResource.do_export, and
merges in the nested-stack resource's Parameters property (resolving any
parent-context intrinsics via the existing create_default_intrinsic_resolver).
Values that cannot be resolved locally (e.g. Fn::GetAtt) are dropped so they
do not poison expand_language_extensions.
Tests:
- New TestResolveNestedStackParameters covers literal, ref, unresolvable,
and empty inputs to the new helper.
- New TestCloudFormationStackResourceChildExpansion asserts parent-passed
Parameters reach expand_language_extensions end-to-end.
- Existing Template(...) mock-call assertions updated to accept the new
parameter_values kwarg.
* fix: narrow exception handling in child template LE expansion
The previous code caught bare Exception when expanding language extensions
on a nested-stack child template, downgrading every failure (including SAM
CLI bugs) to a single WARNING line. That masked real defects: any crash
in the new library silently fell through to the non-expanded path, leaving
the packaged child with unresolved local paths that CloudFormation could
not resolve at deploy time.
Two changes:
1. InvalidSamDocumentException (the documented failure surface of
expand_language_extensions) keeps the warn-and-fallback behavior, but
the message now explains the consequence: artifact URIs inside
Fn::ForEach blocks will NOT be uploaded.
2. Any other exception is logged at ERROR with traceback (exc_info=True)
and a pointer to file an issue. We still fall back so the rest of
sam package keeps going, but the failure is observable to users
running --debug and to telemetry instead of silently degrading.
Also removed the check_using_language_extension branch that was dead:
expand_language_extensions only raises InvalidSamDocumentException when
it actually tried to expand, which implies the transform was present.
Tests:
- TestCloudFormationStackResourceExpansionErrorHandling covers both
branches (warn vs error) and confirms the non-expanded fallback is
still taken.
* fix: narrow exception handling in _resolve_nested_stack_parameters
The helper caught bare Exception when resolving intrinsics in a
nested-stack resource's Parameters property, silently swallowing any
resolver bug (TypeError, AttributeError, KeyError from malformed state).
A user would see incorrect template expansion with no signal.
Now:
- UnresolvableReferenceError / InvalidTemplateException continue
(expected 'can't resolve at package time' cases — e.g. Ref to a
sibling resource).
- Any other exception is logged at DEBUG with traceback so --debug
surfaces it; the value is still dropped so packaging proceeds.
Matches the narrowing pattern applied to do_export's expansion call
site in the previous commit.
* fix: gate MissingMappingKeyError on SAM-generated mapping names
Deployer._create_deploy_error was wrapping *any* CloudFormation
"Fn::FindInMap - Key X not found in Mapping Y" failure as
MissingMappingKeyError, which emits SAM-specific re-package guidance
("Re-run 'sam package' with the same parameter values" and advice
about Fn::ForEach dynamic artifact properties). That guidance is
misleading when the Mapping was written by the user — a classic
RegionMap → AMI lookup failure, for example, has nothing to do with
packaging.
Gate the wrapping on a fixed set of SAM-generated Mapping prefixes
(SAMCodeUri, SAMImageUri, SAMContentUri, SAMDefinitionUri, SAMSchemaUri,
SAMBodyS3Location, SAMDefinitionS3Location, SAMTemplateURL, SAMCode,
SAMContent, SAMLayers). Matching also requires an upper-case letter
after the prefix so names like SAMPLE/SAMSUNG don't accidentally match.
User-authored Fn::FindInMap failures now fall through to the generic
DeployFailedError with the raw CloudFormation message — accurate for
the user's own mistake, without SAM CLI injecting irrelevant advice.
Tests:
- TestIsSamGeneratedMapping covers matching and non-matching names
incl. SAMPLE / SAMSUNG / bare 'SAM' / lower-case variants.
- TestCreateDeployError gains two cases: RegionMap-style user Mapping
is NOT wrapped, and SAM-prefix-substring names are NOT wrapped.
- Existing SAM-generated cases (SAMCodeUriServices, SAMCodeUriMyLoop)
continue to wrap correctly.
* fix: share Fn::ForEach and SAM-mapping helpers across modules
Two issues flagged by the bot reviewer on PR aws#8637:
[BUG] _update_sam_mappings_relative_paths in samcli/commands/_utils/
template.py used mapping_name.startswith("SAM") to decide whether to
rewrite Mapping values as relative paths. That loose check corrupts
customer-authored mappings whose names happen to start with SAM as a
substring — SAMPLE, SAMSUNG, SAMCustomMapping, etc. Values would be
silently rewritten with relative-path prefixes.
[STYLE] infra_sync_executor.py had two hardcoded
resource_logical_id.startswith("Fn::ForEach::") checks instead of
using the shared is_foreach_key helper the rest of the codebase
adopted in this PR. Maintenance risk if the detection logic ever
changes.
Consolidation:
- Moved the precise SAM-mapping classifier (previously
_is_sam_generated_mapping in samcli/commands/deploy/exceptions.py)
into samcli/lib/cfn_language_extensions/utils.py as the public
is_sam_generated_mapping so both deploy/exceptions.py and
commands/_utils/template.py share the same implementation.
- _update_sam_mappings_relative_paths now calls
is_sam_generated_mapping — SAMPLE/SAMSUNG style names are
correctly ignored.
- infra_sync_executor.py now calls is_foreach_key at both sites.
Tests:
- test_skips_sam_prefix_substring_mappings in
TestUpdateSamMappingsRelativePaths verifies SAMPLE/SAMSUNG/
SAMCustomMapping are left untouched.
- Existing TestIsSamGeneratedMapping and TestCreateDeployError
still pass via the re-export in deploy/exceptions.py.
* chore: log swallowed exceptions in _partial_resolve at debug level
_partial_resolve and _partial_resolve_find_in_map intentionally substitute
AWS::NoValue / partial args for false-condition resources when the
resolver raises. The swallow matches the Kotlin reference implementation
and is not changing.
The problem is debuggability: if a resolver bug or unexpected type slips
through, the swallow hides it and the user sees a template with
AWS::NoValue substitutions that look intentional. Adding LOG.debug with
exc_info=True surfaces the traceback under --debug and in telemetry
without changing any runtime behavior.
Three swallow sites covered:
- Ref substitution fallback
- Generic Fn::* substitution fallback
- Fn::FindInMap secondary fallback inside _partial_resolve_find_in_map
The Re-raised InvalidTemplateException branch in
_partial_resolve_find_in_map is unchanged — that path must error.
* fix: declare parent_parameter_values on Resource base class
Previously Template.export() set parent_parameter_values as a duck-typed
attribute on exporter instances, and CloudFormationStackResource.do_export
read it via getattr(self, 'parent_parameter_values', None). The attribute
wasn't declared on any base class, so:
- Typos on either the write or read side wouldn't be caught by static
analysis or IDE completion.
- Any future exporter class for a nested-stack resource type that didn't
explicitly handle this attribute would silently get None and skip
parameter propagation, producing wrong language-extension expansion
for its child template.
Declared parent_parameter_values: Optional[Dict] = None on Resource.
The read site in artifact_exporter.do_export now uses direct attribute
access instead of getattr() with a default.
* fix: route all CFN ClientError paths through _create_deploy_error
Previously only wait_for_execute, create_and_wait_for_changeset's
internal callers (via sync's catch block), and sync itself routed
CloudFormation ClientError through _create_deploy_error, which
detects the SAM-generated Fn::FindInMap key-not-found signature and
surfaces a user-friendly MissingMappingKeyError.
create_stack, update_stack, and create_and_wait_for_changeset's
outer catch block still raised the raw DeployFailedError. On sam
sync the outer catch usually won, but sam deploy / sam package +
deploy paths could hit the synchronous validation error from
create_stack / update_stack directly and bypass the helpful
MissingMappingKeyError entirely — users saw the raw CloudFormation
message with no re-package guidance.
All five previously raw call sites now go through
self._create_deploy_error. The wrapper is a no-op for non-matching
errors (including user-authored Fn::FindInMap failures — gated by
is_sam_generated_mapping), so no regression for any other error
path.
Also left execute_changeset and has_stack untouched — neither can
surface a Fn::FindInMap failure (has_stack is a pre-flight check;
execute_changeset only kicks off the change set, stack-events-phase
errors come back through wait_for_execute which was already wired).
Tests: TestCreateDeployErrorRouting covers create_stack and
update_stack wrapping a SAM-mapping error, and confirms user
RegionMap-style errors still fall through to DeployFailedError.
* perf: replace defensive deepcopy with deep_freeze immutability
expand_language_extensions was deep-copying the template 4-10 times per
invocation for defensive isolation (cache put, cache get, no-LE path,
LE path). For large templates this is measurable overhead.
Replace with a one-time deep_freeze that converts dicts to
MappingProxyType and lists to tuples. Frozen templates are truly
immutable — any mutation attempt raises TypeError immediately instead
of silently corrupting shared state. Callers that need mutable copies
use deep_thaw (not copy.deepcopy which cannot pickle MappingProxyType
on Python 3.13).
Savings:
- No-LE fast path: 4 deepcopies -> 1 deep_freeze (same O(n) cost)
- Cache hit: 2 deepcopies -> 0 (return cached frozen result directly)
- Cold LE path: 3 deepcopies -> 2 deep_freezes (expanded + original)
- Cache put: 2 deepcopies -> 0 (frozen values are safe to share)
Also:
- Fixed isinstance(x, dict) check in build_context to accept Mapping
so frozen MappingProxyType templates pass the type check.
- Updated callers (build_context, package_context, artifact_exporter,
language_extensions_packaging, wrapper) to use deep_thaw instead of
copy.deepcopy when they need mutable copies.
- Updated tests to verify immutability contract (TypeError on mutation)
instead of asserting independent mutable copies.
* fix: freeze DynamicArtifactProperty and use tuple for properties list
The cached LanguageExtensionResult is returned directly on cache hit.
While template fields are deeply frozen via MappingProxyType, the
dynamic_artifact_properties field was a mutable list of mutable
DynamicArtifactProperty dataclass instances. A caller mutating the
list or its elements would corrupt the cached value for all subsequent
hits.
- DynamicArtifactProperty is now frozen=True
- dynamic_artifact_properties field changed from List to Tuple with
default ()
- Construction sites pass tuple() instead of list
This completes the immutability contract: all fields of a cached
LanguageExtensionResult are now truly immutable.
* perf: skip cache and deep_freeze for non-LE templates
When a template has no AWS::LanguageExtensions transform, the work
being cached is a single dict lookup on template.get('Transform') —
O(1). The previous code still deep_freeze'd the entire template O(n)
and stored it in the module-level cache, which was pure overhead for
the vast majority of users who don't use language extensions.
Now the no-LE path returns a LanguageExtensionResult wrapping the
caller's original dict directly — zero copies, zero cache entries.
Callers that need to mutate (build_context, package_context) already
deep_thaw before mutating, so the aliasing is safe.
The LE path still freezes and caches as before.
* refactor: remove unused expansion cache
The module-level _expansion_cache in sam_integration.py was designed to
avoid redundant template expansion across multiple calls within a single
CLI invocation. In practice, the cache was never hit:
- sam build: calls expand_language_extensions once (via get_stacks)
- sam package: calls it twice (get_stacks + _export) — the only
potential hit, but the ~10ms saved is noise vs seconds of S3 uploads
- sam validate/deploy/local: call it once
- sam sync --watch: clears the cache before each reload
The cache added ~40 lines of complexity (module-level mutable global,
hash function, eviction logic, clear_expansion_cache calls in
watch_manager and sam_function_provider) plus a thread-safety concern
flagged in the review, all for negligible benefit.
Removed:
- _expansion_cache, _MAX_CACHE_SIZE, _hash_params, _cache_put,
clear_expansion_cache from sam_integration.py
- clear_expansion_cache() calls from watch_manager.py and
sam_function_provider.py
- template_path parameter from expand_language_extensions (only used
for cache keying) and all call sites
- Exports from __init__.py
- TestExpansionCache class (replaced with two focused immutability
tests)
The deep_freeze on the LE path still ensures immutability without
needing cache isolation.
* refactor: revert deep_freeze/deep_thaw back to copy.deepcopy
With the expansion cache removed in the previous commit, deep_freeze
and deep_thaw serve no purpose. The freeze/thaw cycle was:
1. expand_language_extensions: deep_freeze the result (O(n))
2. Every caller: deep_thaw to get a mutable copy (O(n))
Two O(n) walks that cancel each other out. Without shared cached state
to protect, plain copy.deepcopy is simpler, well-understood, and
doesn't require a custom inverse function.
Reverted all call sites back to copy.deepcopy:
- sam_integration.py: deep_freeze → copy.deepcopy
- build_context.py, package_context.py, artifact_exporter.py,
language_extensions_packaging.py, wrapper.py,
intrinsic_property_resolver.py, infra_sync_executor.py,
sam_stack_provider.py: deep_thaw → copy.deepcopy
- Removed deep_freeze, deep_thaw, MappingProxyType from utils.py
- Reverted isinstance(x, (dict, Mapping)) back to isinstance(x, dict)
- Reverted dynamic_artifact_properties from Tuple back to List
- Updated tests to remove frozen-behavior assertions
DynamicArtifactProperty remains frozen=True (independently useful as
a value object that shouldn't be mutated).
* perf: pass template dict directly to Template, skip yaml round-trip
Template.__init__ now accepts an optional template_dict parameter.
When provided, it skips both file reading and yaml_parse, avoiding
the deepcopy → yaml_dump → yaml_parse round-trip that package_context
and artifact_exporter were doing.
For large templates with many Fn::ForEach expansions, this eliminates
two O(n) serialization passes (yaml_dump + yaml_parse) per export.
* fix: initialize parent_parameter_values in __init__ and fully init Template
Two issues flagged by the bot reviewer:
1. Resource.parent_parameter_values was declared as a class-level
attribute but not set in __init__. Any code path that instantiates
a CloudFormationStackResource and calls do_export() without going
through Template._export_resources() would get AttributeError.
Now initialized to None in __init__.
2. Template.__init__ with template_dict left template_dir and
code_signer uninitialized, requiring callers to manually patch
the object after construction. Now the template_dict branch
derives template_dir from template_path and sets code_signer,
eliminating the partial-initialization window. Removed the manual
assignments from package_context._export and
CloudFormationStackResource.do_export.
* fix: merge pseudo-parameters in sam_template_validator before expansion
expand_language_extensions was called with only self.parameter_overrides,
missing pseudo-parameters (AWS::Region, AWS::AccountId, etc.). Every
other caller merges IntrinsicsSymbolTable.DEFAULT_PSEUDO_PARAM_VALUES
first. Without pseudo-params, templates using Ref: AWS::Region in
Fn::ForEach collections or Fn::Sub expressions would fail during
sam validate but work correctly with sam build and sam package.
* style: use is_foreach_key() consistently in build_context
Three sites in build_context.py used hardcoded
startswith("Fn::ForEach::") instead of the shared is_foreach_key()
utility. Replaced for consistency with the rest of the codebase.
* chore: consolidate property lists, mark unused methods, document validator
Three cleanup items from the review:
aws#5: _ARTIFACT_PATH_PROPERTIES in template.py was a hand-maintained set
of property names that duplicated PACKAGEABLE_RESOURCE_ARTIFACT_PROPERTIES
in models.py. Replaced with a frozenset derived from the canonical source
so the two cannot drift.
aws#6: SamTranslatorWrapper.get_original_template() and
get_dynamic_artifact_properties() are only called in tests — callers
now consume LanguageExtensionResult directly. Added a TODO comment
rather than deleting (would require removing ~20 test methods across
3 files — not worth the churn in this PR).
aws#7: Documented that get_translated_template_if_valid mutates
self.sam_template when language extensions are present. The mutation
is idempotent (re-expanding an already-expanded template is a no-op)
so double-calling is safe, but the comment makes the contract explicit.
* fix: accept digit-leading loop names in is_sam_generated_mapping
A ForEach loop named Fn::ForEach::1stBatch produces a mapping name
like SAMCodeUri1stBatch. The character after the prefix (SAMCodeUri)
is '1', which failed the uppercase-only check. This caused
_update_sam_mappings_relative_paths to skip the mapping (broken
artifact paths in build output) and _create_deploy_error to miss it
(raw CFN error instead of helpful re-package guidance).
Now accepts digits as well as uppercase letters after the prefix.
All SAM-generated prefixes end in a letter (CodeUri, ImageUri, etc.)
so a digit unambiguously indicates the start of the nesting path.
* fix: remove dead template_path param, wrap ValueError in InvalidTemplateException
Two bot-flagged issues:
1. SamTemplateValidator.template_path was stored but never read — leftover
from the removed expansion cache. Removed the parameter from __init__
and both callers (validate.py, resource_mapping_producer.py).
2. sanitize_resource_key_for_mapping raised bare ValueError which wasn't
caught by PackageContext._export (only catches InvalidSamDocumentException).
A ForEach resource key with no static component (e.g. ${Svc}) would
produce an unhandled traceback. Now raises InvalidTemplateException
which is caught and converted to a user-friendly error by the existing
error handling in expand_language_extensions.
* chore: echo note when make test excludes cfn_language_extensions
Developers running 'make test' locally now see a visible note that
cfn_language_extensions tests are excluded, with a pointer to
'make test-all' for full coverage.
* fix: don't error on invalid FindInMap in false-condition resources
CloudFormation skips all validation for resources with false conditions,
including Fn::FindInMap lookups against non-existent mapping keys.
_partial_resolve_find_in_map was re-raising InvalidTemplateException
for string-literal keys that don't exist in Mappings, causing SAM CLI
to reject valid templates that CloudFormation would accept.
Now returns the partially-resolved FindInMap form instead of erroring,
matching CloudFormation's behavior.
Also updated LanguageExtensionResult docstring to accurately document
the no-LE path aliasing contract (expanded_template and
original_template share the same reference when had_language_extensions
is False).
Moved templateFnInMapInWhenFalseConditionInResourceWithInvalidStringPath
from ERROR_TEMPLATES_PASSING to SUCCESS_TEMPLATES_PASSING in the Kotlin
compatibility tests, with expected output.
* test: improve coverage for utils, artifact_exporter, and Template init
New test file test_utils.py covers:
- derive_partition (standard, cn, gov regions)
- derive_url_suffix (standard, cn)
- is_foreach_key (valid, regular, non-string)
- iter_regular_resources (skips ForEach/non-dict, empty, missing)
- is_sam_generated_mapping (bare prefix, lowercase, layers, digits)
New tests in test_artifact_exporter.py:
- _resolve_nested_stack_parameters: Ref to resource dropped, Fn::Sub
partial resolution, unresolvable Ref dropped
- Template.__init__ with template_dict sets template_dir and code_signer
- Template.__init__ with template_str parses YAML correctly
utils.py coverage: 81% -> 94%
artifact_exporter.py coverage: 91% -> 92%
Overall: 94.01% -> 94.02% (7427 -> 7431 tests)
* fix: use direct attribute access for original_template_dict, guard parent_dir=None
Two bot-flagged issues:
1. build_context.py used getattr(stack, 'original_template_dict', None)
but Stack now always declares this attribute. Replaced with direct
stack.original_template_dict access.
2. Template.__init__ with template_dict crashed if parent_dir was None
(make_abs_path raises TypeError). Added guard: falls back to
os.getcwd() when parent_dir is not provided.
* docs: add CloudFormation Language Extensions support documentation
Covers:
- How the two-phase expansion works (LE then SAM transform)
- Fn::ForEach usage with static and parameter-based collections
- Dynamic artifact properties and SAM-generated Mappings
- Nested stacks with parent-supplied parameters
- Nested Fn::ForEach (up to 5 levels)
- &{identifier} syntax for logical ID sanitization
- Supported intrinsic functions table
- Limitations (collection resolvability, package-time fixation)
- Telemetry
* feat(sync): add CloudFormation Language Extensions support for sam sync
Extend sam sync to fully support templates with AWS::LanguageExtensions
and Fn::ForEach. Previously, sync only had guard-level support that
skipped ForEach keys to avoid crashes.
Changes:
- Sanitize artifact properties inside Fn::ForEach body resources during
template comparison so code-only changes don't trigger unnecessary
infra syncs. Handles all code-syncable resource types: functions (zip
+ image), layers, APIs, HTTP APIs, state machines, and nested stacks.
- Detect code changes in ForEach-generated resources by expanding both
the current and deployed templates, then comparing expanded resources.
Changed resources are queued for code sync instead of triggering a
full CloudFormation deployment.
- Code sync (sam sync --code) and watch mode (sam sync --watch) already
work via SamLocalStackProvider.get_stacks() LE expansion. All sync
flows (ADL, Image, API, Layer, StateMachine) work with expanded
ForEach resource identifiers via CFN physical ID mapping.
* style: fix formatting issues across codebase
* Revert "style: fix formatting issues across codebase"
This reverts commit ece6393.
* style: fix black formatting in test_infra_sync_executor
* fix: address review bot comments — exc_info, lowercase loop names, warning level
- Add exc_info=True to bare except in _detect_foreach_code_changes so
--debug output includes the traceback
- Accept lowercase loop names in is_sam_generated_mapping by using
nxt.isalnum() instead of restricting to uppercase + digits
- Upgrade _resolve_nested_stack_parameters unexpected error logging
from debug to warning level so users see it without --debug
* fix: Template.__init__ template_str branch missing attrs, declare parent_parameter_values
- Set template_dir and code_signer in the elif template_str branch of
Template.__init__ to prevent AttributeError when export() is called
- Declare parent_parameter_values on CloudFormationStackResource class
instead of relying on dynamic monkey-patching in Template.export()
* fix: upgrade LE expansion failure to WARNING, include exception in parameter warning
- _detect_foreach_code_changes: upgrade from debug to warning level so
users see when ForEach code change detection is skipped
- _resolve_nested_stack_parameters: include exception message in the
warning so users can diagnose without --debug
* fix: include exception message in LE expansion warning
* fix: pass parameter defaults to expand_language_extensions in code change detection
Without parameter values, Fn::ForEach collections using Ref to
parameters (e.g. {"Ref": "ServiceNames"}) fail to resolve during
expansion, causing _detect_foreach_code_changes to silently skip
code change detection for all ForEach-generated resources.
* fix: extract parameter defaults independently for each template in code change detection
Use each template's own Parameters section to extract defaults,
so expansion produces correct results even if parameter definitions
changed between the current and deployed template versions.
* fix: resolve PLR2004, PLR0911, PLR1714 lint issues in cfn_language_extensions
Replace blanket per-file ruff suppression with proper fixes:
- PLR2004: Extract magic numbers into named class constants for
CloudFormation intrinsic function argument counts
- PLR1714: Add is_intrinsic_key() helper to utils.py to replace
repeated startswith/equality comparison chains
- PLR0911: Refactor _partial_resolve into smaller helper methods
to reduce return statement count
Reuse existing FOREACH_REQUIRED_ELEMENTS constant from utils.py
for ForEach processor validation.
* fix: address aws-sam-cli-bot code review comments
- Add us-iso- and us-isob- partition/URL suffix handling to
derive_partition and derive_url_suffix
- Remove phantom caching references from module and function docstrings
in sam_integration.py (no caching logic exists)
- Remove redundant copy.deepcopy(expanded_template) since
process_template_for_sam_cli already returns an independent object
- Update deep-copy comment to explain caller isolation intent
- Refactor broad except to catch LangExtInvalidTemplateException
directly instead of catching Exception and using isinstance
- Add docstring note about parameter defaults limitation in
_detect_foreach_code_changes
* fix: add aws-eusc partition support, remove original_template_dict from Stack.__eq__
- Add eusc- region prefix handling to derive_partition (aws-eusc) and
derive_url_suffix (amazonaws.eu) for the AWS European Sovereign Cloud
- Revert us-iso-/us-isob- ADC partition branches — not supported
elsewhere in the SAM CLI codebase
- Remove original_template_dict from Stack.__eq__ — it is operational
metadata for deployment, not part of the stack's semantic identity.
Including it caused stacks created via different code paths to compare
as unequal even when semantically identical1 parent a90d18f commit 2e21eb2
425 files changed
Lines changed: 53075 additions & 171 deletions
File tree
- docs
- samcli
- commands
- _utils
- build
- deploy
- package
- validate
- lib
- build
- cfn_language_extensions
- processors
- resolvers
- deploy
- iac/cdk
- package
- providers
- samlib
- sync
- telemetry
- translate
- warnings
- tests
- integration
- buildcmd
- deploy
- local
- invoke
- start_api
- package
- testdata
- buildcmd
- language-extensions-dynamic-codeuri
- Alpha
- Beta
- language-extensions-nested-foreach-dynamic-codeuri
- services
- Orders
- Users
- package
- language-extensions-dynamic-codeuri
- Alpha
- Beta
- language-extensions-foreach
- src
- start_api
- language-extensions-api
- endpoints
- orders
- products
- users
- validate
- language-extensions-depth-limit
- language-extensions-invalid-syntax
- validate
- unit
- commands
- _utils
- buildcmd
- deploy
- local/lib
- package
- validate/lib
- lib
- cfn_language_extensions
- compatibility
- templates
- forEach
- conditions
- outputs
- resources
- deploy
- intrinsic_resolver
- package
- providers
- samlib
- sync
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
30 | 39 | | |
31 | 40 | | |
32 | 41 | | |
33 | | - | |
| 42 | + | |
34 | 43 | | |
35 | 44 | | |
36 | 45 | | |
| |||
58 | 67 | | |
59 | 68 | | |
60 | 69 | | |
61 | | - | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
62 | 81 | | |
63 | 82 | | |
64 | 83 | | |
| |||
72 | 91 | | |
73 | 92 | | |
74 | 93 | | |
75 | | - | |
76 | | - | |
| 94 | + | |
| 95 | + | |
77 | 96 | | |
78 | 97 | | |
79 | 98 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
132 | 132 | | |
133 | 133 | | |
134 | 134 | | |
| 135 | + | |
135 | 136 | | |
136 | 137 | | |
137 | 138 | | |
| |||
0 commit comments