feat(DRIVERS-2903): use custom aws configuration (#1743)

Co-authored-by: Jeff Yemin <jeff.yemin@mongodb.com>
Co-authored-by: Kevin Albertson <kevin.albertson@mongodb.com>

Author: 3 people

source/auth/auth.md

@@ -987,12 +987,29 @@ those credentials will be used by default if AWS auth environment variables are
 application. Alternatively, you can create an AWS profile specifically for your MongoDB credentials and set the
 `AWS_PROFILE` environment variable to that profile name."
 
+##### Custom Credential Providers
+
+Drivers that choose to use the AWS SDK to fetch credentials MAY also allow users to provide a custom credential provider
+as an option to the `MongoClient`. The interface for the option provided depends on the individual language SDK and
+drivers MUST consult AWS SDK documentation to determine that format when implementing. The name of the option MUST be
+`AWS_CREDENTIAL_PROVIDER` and be part of the authentication mechanism properties options that can be provided to the
+client.
+
+Drivers MAY expose API for default providers for the following scenarios when applicable in their language's SDK:
+
+1. The default SDK credential provider.
+2. A custom credential provider chain.
+3. A single credential provider of any available SDK options provided by the SDK.
+
+##### Credential Fetching Order
+
 The order in which Drivers MUST search for credentials is:
 
 1. The URI
 2. Environment variables
-3. Using `AssumeRoleWithWebIdentity` if `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN` are set.
-4. The ECS endpoint if `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` is set. Otherwise, the EC2 endpoint.
+3. A custom AWS credential provider if the driver supports it.
+4. Using `AssumeRoleWithWebIdentity` if `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN` are set.
+5. The ECS endpoint if `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` is set. Otherwise, the EC2 endpoint.
 
 > [!NOTE]
 > See *Should drivers support accessing Amazon EC2 instance metadata in Amazon ECS* in [Q & A](#q-and-a)
@@ -1306,6 +1323,12 @@ in the MONGODB-OIDC specification, including sections or blocks that specificall
         check MUST be performed after SRV record resolution, if applicable. This property is only required for drivers
         that support the [Human Authentication Flow](#human-authentication-flow).
 
+    - AWS_CREDENTIAL_PROVIDER
+
+        A function or object from the AWS SDK that can be used to return AWS credentials. Drivers MAY allow the user to
+        specify the callback using a `MongoClient` configuration instead of a mechanism property, depending on what is
+        idiomatic for the driver.
+
 <span id="built-in-provider-integrations"/>
 
 #### Built-in OIDC Environment Integrations
@@ -2134,6 +2157,8 @@ practice to avoid this. (See
 
 ## Changelog
 
+- 2025-01-29: Add support for custom AWS credential providers.
+
 - 2024-10-02: Add Kubernetes built-in OIDC provider integration.
 
 - 2024-08-19: Clarify Reauthentication and Speculative Authentication combination behavior.

source/auth/tests/mongodb-aws.md

@@ -21,6 +21,16 @@ SecretAccessKey=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
 Token=AQoDYXdzEJr...<remainder of security token>
 ```
 
+If the driver supports user provided custom AWS credential providers, then the driver MUST also test the above scenarios
+2-6 with a user provided `AWS_CREDENTIAL_PROVIDER` auth mechanism property. This value MUST be the default credential
+provider from the AWS SDK. If the default provider does not cover all scenarios above, those not covered MAY be skipped.
+In these tests the driver MUST also assert that the user provided credential provider was called at least once in each
+test.
+
+If the driver supports a custom AWS credential provider, it MUST verify the custom provider was used when testing. This
+may be via a custom function or object that wraps the calls to the custom provider and asserts that it was called at
+least once.
+
 ## Regular credentials
 
 Drivers MUST be able to authenticate by providing a valid access key id and secret access key pair as the username and

source/client-side-encryption/client-side-encryption.md

@@ -380,6 +380,7 @@ class AutoEncryptionOpts {
    // without the MongoDB Enterprise Advanced licensed crypt_shared library.
    bypassQueryAnalysis: Optional<Boolean>; // Default false.
    keyExpirationMS: Optional<Uint64>; // Default 60000. 0 means "never expire".
+   credentialProviders: Optional<CredentialProviders>;
 }
 ```
 
@@ -475,6 +476,44 @@ See
 <span id="GCPKMSOptions"></span> <span id="AWSKMSOptions"></span> <span id="KMSProvider"></span>
 <span id="KMSProviders"></span> <span id="AzureAccessToken"></span> <span id="kmsproviders"></span>
 
+#### credentialProviders
+
+The `credentialProviders` property may be specified on [ClientEncryptionOpts](#ClientEncryptionOpts) or
+[AutoEncryptionOpts](#AutoEncryptionOpts). Current support is for AWS only, but is designed to be able to accommodate
+additional providers in the future. If a custom credential provider is present, it MUST be used instead of the default
+flow for fetching automatic credentials and if the `kmsProviders` are not configured for automatic credential fetching
+an error MUST be thrown.
+
+```typescript
+interface CredentialProviders {
+   aws?: AWSCredentialProvider
+}
+
+// The type of the AWS credential provider is dictated by the AWS SDK's credential provider for the specific
+// language.
+type AWSCredentialProvider = Function | Object;
+```
+
+The following shows an example object of `CredentialProviders` for Node.js:
+
+```typescript
+import { fromNodeProviderChain } from '@aws-sdk/credential-providers';
+
+const client = new MongoClient(process.env.MONGODB_URI, {
+  autoEncryption: {
+    keyVaultNamespace: 'keyvault.datakeys',
+    kmsProviders: {
+      // Set to empty map to use `credentialProviders`.
+      aws: {}
+   },
+    credentialProviders: {
+      // Acquire credentials for AWS:
+      aws: fromNodeProviderChain()
+    }
+  }
+}
+```
+
 #### kmsProviders
 
 The `kmsProviders` property may be specified on [ClientEncryptionOpts](#ClientEncryptionOpts) or
@@ -593,11 +632,14 @@ Once requested, drivers MUST create a new [KMSProviders](#kmsproviders) $P$ acco
     [ClientEncryptionOpts](#ClientEncryptionOpts) or [AutoEncryptionOpts](#AutoEncryptionOpts).
 2. Initialize $P$ to an empty [KMSProviders](#kmsproviders) object.
 3. If $K$ contains an `aws` property, and that property is an empty map:
-    1. Attempt to obtain credentials $C$ from the environment using similar logic as is detailed in
-        [the obtaining-AWS-credentials section from the Driver Authentication specification](../auth/auth.md#obtaining-credentials),
-        but ignoring the case of loading the credentials from a URI
-    2. If credentials $C$ were successfully loaded, create a new [AWSKMSOptions](#AWSKMSOptions) map from $C$ and insert
-        that map onto $P$ as the `aws` property.
+    1. If a custom credential provider is supplied via the `credentialProviders.aws` applicable encryption option, use
+        that to fetch the credentials from AWS.
+    2. Otherwise:
+        1. Attempt to obtain credentials $C$ from the environment using similar logic as is detailed in
+            [the obtaining-AWS-credentials section from the Driver Authentication specification](../auth/auth.md#obtaining-credentials),
+            but ignoring the case of loading the credentials from a URI
+        2. If credentials $C$ were successfully loaded, create a new [AWSKMSOptions](#AWSKMSOptions) map from $C$ and
+            insert that map onto $P$ as the `aws` property.
 4. If $K$ contains an `gcp` property, and that property is an empty map:
     1. Attempt to obtain credentials $C$ from the environment logic as is detailed in
         [Obtaining GCP Credentials](#obtaining-gcp-credentials).
@@ -1051,6 +1093,7 @@ interface ClientEncryptionOpts {
    keyVaultClient: MongoClient;
    keyVaultNamespace: String;
    kmsProviders: KMSProviders;
+   credentialProviders: CredentialProviders;
    tlsOptions?: KMSProvidersTLSOptions; // Maps KMS provider to TLS options.
    keyExpirationMS: Optional<Uint64>; // Default 60000. 0 means "never expire".
 };
@@ -2420,6 +2463,8 @@ explicit session parameter as described in the [Drivers Sessions Specification](
 
 ## Changelog
 
+- 2024-02-19: Add custom options AWS credential provider.
+
 - 2024-10-09: Add retry prose test.
 
 - 2024-07-29: Document range as stable.

source/client-side-encryption/tests/README.md

@@ -3683,3 +3683,91 @@ Run an aggregate operation on `db.csfle` with the following pipeline:
 ```
 
 Expect an exception to be thrown with a message containing the substring `Upgrade`.
+
+### 26. Custom AWS Credentials
+
+These tests require valid AWS credentials for the remote KMS provider via the secrets manager (FLE_AWS_KEY and
+FLE_AWS_SECRET). These tests MUST NOT run inside an AWS environment that has the same credentials set in order to
+properly ensure the tests would fail using on-demand credentials.
+
+#### Case 1: ClientEncryption with `credentialProviders` and incorrect `kmsProviders`
+
+Create a MongoClient named `setupClient`.
+
+Create a [ClientEncryption](../client-side-encryption.md#clientencryption) object with the following options:
+
+```typescript
+class ClientEncryptionOpts {
+  keyVaultClient: <setupClient>,
+  keyVaultNamespace: "keyvault.datakeys",
+  kmsProviders: { "aws": { "accessKeyId": <set from secrets manager>, "secretAccessKey": <set from secrets manager> } },
+  credentialProviders: { "aws": <default provider from AWS SDK> }
+}
+```
+
+Assert that an error is thrown.
+
+#### Case 2: ClientEncryption with `credentialProviders` works
+
+Create a MongoClient named `setupClient`.
+
+Create a [ClientEncryption](../client-side-encryption.md#clientencryption) object with the following options:
+
+```typescript
+class ClientEncryptionOpts {
+  keyVaultClient: <setupClient>,
+  keyVaultNamespace: "keyvault.datakeys",
+  kmsProviders: { "aws": {} },
+  credentialProviders: { "aws": <object/function that returns valid credentials from the secrets manager> }
+}
+```
+
+Use the client encryption to create a datakey using the "aws" KMS provider. This should successfully load and use the
+AWS credentials that were provided by the secrets manager for the remote provider. Assert the datakey was created and
+that the custom credential provider was called at least once.
+
+An example of this in Node.js:
+
+```typescript
+import { ClientEncryption, MongoClient } from 'mongodb';
+
+let calledCount = 0;
+const masterKey = {
+  region: '<aws region>',
+  key: '<key for arn>'
+};
+const keyVaultClient = new MongoClient(process.env.MONGODB_URI);
+const options = {
+  keyVaultNamespace: 'keyvault.datakeys',
+  kmsProviders: { aws: {} },
+  credentialProviders: {
+    aws: async () => {
+      calledCount++;
+      return {
+        accessKeyId: process.env.FLE_AWS_KEY,
+        secretAccessKey: process.env.FLE_AWS_SECRET
+      };
+    }
+  }
+};
+const clientEncryption = new ClientEncryption(keyVaultClient, options);
+const dk = await clientEncryption.createDataKey('aws', { masterKey });
+expect(dk).to.be.a(Binary);
+expect(calledCount).to.be.greaterThan(0);
+```
+
+#### Case 3: `AutoEncryptionOpts` with `credentialProviders` and incorrect `kmsProviders`
+
+Create a `MongoClient` object with the following options:
+
+```typescript
+class AutoEncryptionOpts {
+  autoEncryption: {
+    keyVaultNamespace: "keyvault.datakeys",
+    kmsProviders: { "aws": { "accessKeyId": <set from secrets manager>, "secretAccessKey": <set from secrets manager> } },
+    credentialProviders: { "aws": <default provider from AWS SDK> }
+  }
+}
+```
+
+Assert that an error is thrown.