Skip to content

Remove SAM translator dependency, validate SAM templates via schemas#4491

Open
kddejong wants to merge 16 commits into
aws-cloudformation:mainfrom
kddejong:remove-sam-translator
Open

Remove SAM translator dependency, validate SAM templates via schemas#4491
kddejong wants to merge 16 commits into
aws-cloudformation:mainfrom
kddejong:remove-sam-translator

Conversation

@kddejong
Copy link
Copy Markdown
Contributor

@kddejong kddejong commented Apr 30, 2026

Summary

Replace the aws-sam-translator runtime dependency with direct JSON Schema validation of SAM resource types. SAM templates are now validated against schemas derived from the SAM project's own schema definitions, using the same provider schema infrastructure as CloudFormation resources.

Changes

  • Schema generation (scripts/update_sam_schemas.py): Downloads the SAM JSON Schema, decomposes it into per-resource-type schemas, generates PassThrough property patches from CFN schemas, and updates provider modules with SAM type entries.
  • Globals support (_sam_globals.py): Merges Globals section properties into SAM resources at validation time, respecting IgnoreGlobals.
  • E3065: Validates that AWS::Serverless::Connector and IgnoreGlobals usage requires the SAM transform declaration.
  • E3722: Validates Globals section structure against the SAM Globals schema.
  • Lambda rule keywords: Extended Lambda rules (E2531, E2533, etc.) to also match AWS::Serverless::Function paths.
  • Ref/GetAtt: SAM resources use the same sub-resource wildcard mechanism as Modules (via module_names). GetAtt on the SAM resource itself is validated against readOnlyProperties from the mapped CFN type.
  • CFN mappings: All SAM types with CFN equivalents have primaryIdentifier and readOnlyProperties from their mapped type (e.g., Function → Lambda::Function, GraphQLApi → AppSync::GraphQLApi).
  • Removed: aws-sam-translator dependency, _sam.py transform module, ManagedPolicies.json.

Behavioral Notes

  • Input validation: SAM resource properties, Globals, events, and policies are validated against SAM schemas. This is equivalent to what the old translator checked.
  • Output validation: We no longer validate the transformed CFN output (generated IAM roles, API Gateway stages, etc.). SAM owns that correctness.
  • Sub-resources: !Ref MyFunctionRole and !GetAtt MyFunctionRole.Arn pass via the wildcard mechanism. Attribute validation is skipped for sub-resources (same as Modules).

Testing

  • Unit tests for schema generation, globals merging, and new rules
  • Integration tests for good/bad SAM templates, IgnoreGlobals interaction, and Module sub-resources
  • All existing integration tests pass
  • Verified no false positives on real-world SAM templates

Note

Commit message for 9d6ef6b incorrectly says E3064 — the actual rule ID is E3065 (E3064 is taken by VpcEndpointPrivateDnsDuplicate).

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 30, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.55%. Comparing base (9bd40ec) to head (db70b67).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4491      +/-   ##
==========================================
- Coverage   93.57%   93.55%   -0.03%     
==========================================
  Files         466      468       +2     
  Lines       15230    15250      +20     
  Branches     2945     2953       +8     
==========================================
+ Hits        14252    14267      +15     
- Misses        599      601       +2     
- Partials      379      382       +3     
Flag Coverage Δ
unittests 93.55% <100.00%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@kddejong kddejong force-pushed the remove-sam-translator branch 4 times, most recently from 2e7370a to 56e6be1 Compare May 13, 2026 15:42
kddejong added 13 commits May 13, 2026 13:21
Replace the samtranslator-based transform with direct SAM schema validation:

- Add SAM resource schemas (11 types) decomposed from the official SAM JSON Schema
- Add Globals section schema and minimal Globals merger
- Register SAM types in all 46 provider modules
- Inject synthetic implicit resources (ServerlessRestApi, ServerlessHttpApi, {Id}Role)
- Skip circular dependency checks for SAM resources (SAM splits them during transform)
- Fix Ref class to handle schemas without primaryIdentifier
- Add Globals as valid template-level property

Remove samtranslator dependency:
- Delete _sam.py transform wrapper
- Delete ManagedPolicies.json and update script
- Remove aws-sam-translator from requirements/base.txt
- Remove samtranslator mypy override

Known gaps documented in docs/sam-schema-gaps.md:
- Generated resources with suffixed logical IDs (partially mitigated)
- Runtime deprecation rules don't fire on SAM types yet
- SAM event source validation not available without transform
- 3 SAM types without confirmed CFN mappings
Keyword expansion:
- Runtime deprecation rules (E2531, E2533, W2531) now fire on
  AWS::Serverless::Function and Globals/Function/Runtime
- Lambda extension schema rules (E3673-E3677) now fire on
  AWS::Serverless::Function properties

New rule E3715 (GlobalsTransform):
- Validates Globals section requires the SAM transform
- Validates Globals structure against schema (section names,
  property names per section)
- Stripped $ref from globals schema for standalone validation
Schema generation:
- update_sam_schemas.py now generates PassThroughProp patches from
  CFN schema property mappings (174 ops across 6 SAM types) and
  applies them before writing schemas so they are baked into the
  hash files at rest
- Inline $ref pointers from CFN definitions (depth 3) so patches
  are self-contained (FilterCriteria, DestinationConfig, etc.)
- Strip description fields to match CFN schema conventions
- Auto-clean stale SAM schema files on regeneration

Manual patches (not auto-generated):
- Add State property to EventBridgeRuleEventProperties for
  Function and StateMachine (missing from SAM schema source)

Resource configuration:
- Allow Connectors as a resource-level attribute (SAM embedded
  connector syntax)

Context:
- Inject synthetic {Id}.Version and {Id}.Alias resources for
  Functions with AutoPublishAlias/DeploymentPreference

Maintenance:
- Add update_sam_schemas.py to maintenance workflow
- Remove samtranslator version pinning and
  update_serverless_aws_policies.py
- Remove backup.json

Fix E3720 rule ID conflict with BlockDeviceMappingVirtualName
…d Serverless dir

- Add IgnoreGlobals to resource configuration schema (accepts
  '*' string or list of property names)
- Add timeout=30 to urlopen in update_sam_schemas.py
- Remove orphaned src/cfnlint/data/Serverless/ directory
New rule E3064 (ServerlessTransformAttributes) validates that
Connectors and IgnoreGlobals resource attributes are only used
when the AWS::Serverless-2016-10-31 transform is declared.
- Inline $ref pointers when copying readOnlyProperties from CFN
  schemas (fixes dangling SnapStartResponse $ref)
- Skip SAM schemas from CFN provider schema validation (SAM
  schemas don't have primaryIdentifier or CFN definition naming)
- Build keywords from SAM schemas for rule keyword validation
- Add Globals, Connectors, IgnoreGlobals to found keywords list
Upstream main added E3720 for DbInstanceKmsKeyStorageEncrypted in aws-cloudformation#4480.
Renaming GlobalsTransform to the next available ID.
- test_sam_globals.py: 16 tests for merge_globals (100% coverage)
- test_globals_transform.py: 4 tests for E3722 rule (100% coverage)
- test_serverless_transform_attributes.py: 4 tests for E3065 rule (100% coverage)
- test_inject_sam_implicit.py: 15 tests for implicit resource injection (95% coverage)
…and readOnlyProperties

Remove blanket permissive GetAtt for all AWS::Serverless:: types.
SAM types with CFN mappings now use the mapped type's readOnlyProperties
for GetAtt validation and primaryIdentifier for Ref validation.
Connector correctly errors on GetAtt since it has no primary resource.
… permissions, Api Domain/UsagePlan, implicit API stages
@kddejong kddejong force-pushed the remove-sam-translator branch from fdb75ab to aefb4dc Compare May 13, 2026 20:23
@kddejong kddejong changed the title WIP: Remove SAM translator dependency, validate SAM templates via schemas Remove SAM translator dependency, validate SAM templates via schemas May 13, 2026
@kddejong kddejong force-pushed the remove-sam-translator branch from aefb4dc to 054499d Compare May 13, 2026 22:43
@kddejong kddejong marked this pull request as ready for review May 14, 2026 15:48
…cript

- Fix SAM_TO_CFN_TYPE mapping: AWS::Lambda::CapacityProvider (nonexistent)
  → AWS::ECS::CapacityProvider
- Remove bogus schema file and duplicate provider module entries
- Replace exec() with ast.literal_eval for parsing provider modules
- Fix _inline_refs to recurse into lists (e.g. anyOf/oneOf arrays)
- Add warning logs to _resolve_cfn_property on failure
- Escalate readOnlyProperties lookup failure to error level

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant