feat: add opt-in minimizeRdsVpcEndpoints to the SQL data source strategy#3449
Conversation
…2e assertion [#3449] Released transformer (graphql-model-transformer@2.14.2) still emits all 5 endpoints in the Gen1 amplify-push e2e. The reduction to only 'ssm' is covered by the unit test sql-model-definition.test.ts. Re-enable once >=3.x ships.
…evert [#3449] Runs only the 4 affected RDS groups + prerequisites. Remove before merge.
|
Nice cleanup — thanks for tracking this down! Removing those four unused VPC endpoints is a real cost win for customers, and the root-cause writeup in #3409 makes it easy to see why only I’ve got two things I’d like us to sort out before merging — one about communicating this to existing users, and one about the e2e tests. Details inline. 🙏 |
| // Although the Lambda function will only invoke SSM directly, internally the SDK makes calls to other services as well | ||
| const services = ['ssm', 'ssmmessages', 'ec2', 'ec2messages', 'kms']; | ||
| // Only the SSM VPC endpoint is needed for Lambda to access Parameter Store | ||
| const services = ['ssm']; |
There was a problem hiding this comment.
One thing worth calling out: for anyone who already has an RDS-in-VPC API deployed, the next amplify push / cdk deploy after they pick up this change will have CloudFormation delete these four endpoints from their live stack. Functionally that’s fine since nothing was using them — but it does mean an upgrade silently modifies their infrastructure.
Could we add a CHANGELOG / release-note entry flagging this? Folks running cost-anomaly alerts or compliance tooling on their VPCs will see four resources disappear and may wonder why. A one-liner like “RDS-in-VPC deployments will have four unused VPC endpoints (ssmmessages, ec2, ec2messages, kms) removed on next deploy — this is expected and safe” would save someone a confused investigation later.
There was a problem hiding this comment.
Implemented as feature flag instead to avoid breaking existing user experience.
| expect(rdsLambdaFunction.Properties.VpcConfig.SecurityGroupIds.length).toBeGreaterThan(0); | ||
|
|
||
| expect(getResource(resources, `${resourceNames.sqlVpcEndpointPrefix}ssm`, CDK_VPC_ENDPOINT_TYPE)).toBeDefined(); | ||
| // TODO(#3449): re-enable the reduced VPC-endpoint assertion (only 'ssm' expected) once graphql-model-transformer >=3.x ships in the released @aws-amplify/cli-internal that this Gen1 amplify-push e2e installs. The reduction is validated by the unit test sql-model-definition.test.ts (endpoints.length === 1). |
There was a problem hiding this comment.
I think these e2e assertions are a bit of a time bomb. They still expect all 5 endpoints (toBeDefined()), and they only pass today because the installed CLI pulls @aws-amplify/graphql-model-transformer from npm at 2.14.2, which still emits 5. Once this change ships and the published package updates, these will start failing on their own without anyone touching them.
The TODO points at graphql-model-transformer >=3.x, but I couldn’t find a timeline for a 3.x release — so this could sit as latent CI breakage for a while. Could we either:
- flip the four assertions to
toBeUndefined()now (with a short note on the Gen1 CLI version lag), or - gate them on the installed transformer version if that’s detectable?
Either way, a concrete tracking issue would be better than a version that may not land soon.
There was a problem hiding this comment.
The feature flag approach no longer require the e2e trick.
…2e assertion [#3449] Released transformer (graphql-model-transformer@2.14.2) still emits all 5 endpoints in the Gen1 amplify-push e2e. The reduction to only 'ssm' is covered by the unit test sql-model-definition.test.ts. Re-enable once >=3.x ships.
…evert [#3449] Runs only the 4 affected RDS groups + prerequisites. Remove before merge.
71227ca to
4d59cd7
Compare
Only the SSM VPC endpoint is needed for Lambda to access Parameter Store. The other 4 endpoints (ssmmessages, ec2, ec2messages, kms) are never used by the code but cost customers ~$350/year per AZ. Fixes #3409
…2e assertion [#3449] Released transformer (graphql-model-transformer@2.14.2) still emits all 5 endpoints in the Gen1 amplify-push e2e. The reduction to only 'ssm' is covered by the unit test sql-model-definition.test.ts. Re-enable once >=3.x ships.
…evert [#3449] Runs only the 4 affected RDS groups + prerequisites. Remove before merge.
…ts feature flag (default off)
4d59cd7 to
de5d4fd
Compare
Description of changes
Adds an opt-in
minimizeRdsVpcEndpointsfield to the SQL data source strategy (SQLLambdaModelDataSourceStrategy).When a SQL (RDS) data source is installed into a VPC, the transformer provisions a set of interface VPC endpoints so the SQL Lambda can reach AWS services. Historically it always provisioned five endpoints —
ssm,ssmmessages,ec2,ec2messages,kms— but at runtime the SQL Lambda only consumes thessmendpoint (to read the database connection secret). The other four are unused and incur ongoing cost.minimizeRdsVpcEndpointslets a customer opt into provisioning only thessmendpoint:minimizeRdsVpcEndpointsfalse(default)ssm,ssmmessages,ec2,ec2messages,kms(5)truessm(1)The field lives next to
vpcConfigurationon the SQL strategy and only takes effect whenvpcConfigurationis set; it is a no-op for SQL data sources that are not installed into a VPC.Why scope it to the SQL data source strategy (not a global flag)?
VPC-endpoint provisioning is a property of an individual SQL (RDS) data source — it is only meaningful for a SQL strategy that declares a
vpcConfiguration. A backend can define multiple data sources, so a globalTransformParameter/translationBehaviorflag would apply indiscriminately and could not express "minimize endpoints for this RDS data source but not that one." Placing the field onSQLLambdaModelDataSourceStrategykeeps the option next to thevpcConfigurationit modifies and scopes it precisely to the resource it affects. An earlier revision of this PR plumbed the flag throughTransformParameters/translationBehavior; this revision removes that global plumbing in favor of the data-source-scoped field.Why does the default stay
false? (breaking-change rationale)The default remains
false(all five endpoints) deliberately. Removing endpoints from an already-deployed API is a destructive CloudFormation change: on the next deploy CFN deletes the four now-unreferencedAWS::EC2::VPCEndpointresources. Defaulting totrue, or minimizing unconditionally, would silently delete those endpoints from every existing RDS-in-VPC stack on redeploy. Gating the behavior behind an explicit opt-in means existing stacks are untouched until the customer knowingly enables the optimization.Coordinated changes
This is 1 of 3 coordinated repository changes that surface
minimizeRdsVpcEndpointsend-to-end:aws-amplify/amplify-category-api(this PR) — adds the field toSQLLambdaModelDataSourceStrategyand rewires the RDS transformer consumer.aws-amplify/amplify-data— surfaces the field indata-schema-types.aws-amplify/amplify-backend— surfaces the field throughbackend-data.Summary of code changes
minimizeRdsVpcEndpoints?: boolean(@default false) toSQLLambdaModelDataSourceStrategy— both the public construct type (amplify-graphql-api-construct) and the internal transformer interface (amplify-graphql-transformer-interfaces).RdsModelResourceGeneratorto readstrategy.minimizeRdsVpcEndpoints(instead ofcontext.transformParameters.minimizeRdsVpcEndpoints) and thread it intogetSsmEndpoint/createRdsLambda, which provision['ssm']when enabled and the full five-endpoint set otherwise.minimizeRdsVpcEndpointsis deleted fromTransformParameters,defaultTransformParameters,TranslationBehavior,PartialTranslationBehavior, anddefaultTranslationBehavior.amplify-graphql-api-construct.jsii/API.mdfor the moved public API surface (the diff is limited to theminimizeRdsVpcEndpointsmove).CDK / CloudFormation Parameters Changed
No CDK/CFN template parameters are added. When
minimizeRdsVpcEndpoints: true, the number of provisionedAWS::EC2::VPCEndpointresources for a VPC-bound SQL data source drops from five to one (ssmonly). With the default (false), generated resources are unchanged.Issue #, if available
Related to #3409
Description of how you validated changes
amplify-graphql-model-transformerunit tests —amplify-sql-resource-generator.test.ts: 12/12 pass, including:amplify-graphql-api-constructfunctional tests —sql-model-definition.test.ts: 7/7 pass, including "ssm as credential store with minimizeRdsVpcEndpoints enabled" (asserts the strategy-scoped field, nottranslationBehavior).tscbuilds pass for the affected packages (amplify-graphql-transformer-interfaces,amplify-graphql-transformer-core,amplify-graphql-transformer-test-utils,amplify-graphql-model-transformer);jsiiregenerates cleanly foramplify-graphql-api-construct.Checklist
yarn testpassesBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.