Error when decrypting data from kinesis stream "providerID doesnt match to with MasterKeyProvider ID"

[Codewolf]

Setup/structure:
Data being sent to the kinesis stream from a pgsql database whenever a field/row is updated, data is encrypted using kms.

Code:
where

• kmsClient is the AWS sdk v2 kms client
• key is the base64 encoded key value in the kinesis record
• data is the base64 encoded databaseActivityEvents value in the kinesis record
• clusterResourceId is what it sounds like

        decodedKey, err := base64.StdEncoding.DecodeString(key)
	if err != nil {
		return err
	}
	decodedBody, err := base64.StdEncoding.DecodeString(data)
	if err != nil {
		return err
	}
	decryptedKey, err := kmsClient.Decrypt(ctx, &kms.DecryptInput{CiphertextBlob: decodedKey, EncryptionContext: map[string]string{"aws:rds:dbc-id": clusterResourceId}})
	if err != nil {
		return err
	}

this results in a decrypted struct, in which is the ARN for the key used, this is then used below

• encryptionClient is basically client.NewClientWithConfig(encryptionConfig)

provider, err := kmsprovider.New(*decryptedKey.KeyId)
	if err != nil {
		return err
	}
	cmm, err := materials.NewDefault(provider)
	if err != nil {
		return err
	}
	d, header, err := encryptionClient.Decrypt(ctx, decodedBody, cmm)
	if err != nil {
		return err
	}

Issue:
multiple different configurations tried, including manually providing the ARN for the key, and all of them result in the error shown below with the pertinent part being "BC" providerID doesnt match to with MasterKeyProvider ID "aws-kms"

SDK error: decryption error
decrypt materials: no data key, last error: CMM error
unable to decrypt any data key, member error: MKP error
DecryptDataKeyFromList validate expected error: MKP decrypt error
"BC" providerID doesnt match to with MasterKeyProvider ID "aws-kms"

What i would like to know is if this is potentially a bug or is there a misconfiguration on my end, if its the latter then i would appreciate a pointer in the right direction

[Codewolf]

@wobondar if it helps, what i am trying to replicate is https://github.com/aws-samples/aurora-das-processing/blob/main/lambda_function.py

specifically this code block

def decrypt_payload(payload, data_key):
    my_key_provider = MyRawMasterKeyProvider(data_key)
    my_key_provider.add_master_key("DataKey")
    #Decrypt the records using the master key.
    decrypted_plaintext, header = enc_client.decrypt(
        source=payload,
        materials_manager=aws_encryption_sdk.materials_managers.default.DefaultCryptoMaterialsManager(master_key_provider=my_key_provider))
    return decrypted_plaintext

[Codewolf]

It is also worth noting that when i set it up manually using a new raw provider (using the code below) i get a similar error, just without the "BC" providerID doesnt match to with MasterKeyProvider ID "aws-kms"
all variable values are as per the original post unless specified otherwise

Code:

provider, _ := rawprovider.NewWithOpts("BC", rawprovider.WithStaticKey("DataKey", decryptedKey))
cmm, err := materials.NewDefault(provider)
if err != nil {
return err
}

d, header, err := encryptionClient.Decrypt(ctx, decodedBody, cmm)

this results in

SDK error: decryption error
decrypt materials: no data key, last error: CMM error
unable to decrypt any data key, member error: MKP error
unable to decrypt data key: MKP decrypt error

[wobondar]

@Codewolf
Thanks for finding this out, it is definitely a bug.

Yesterday, during my initial research, I came across a similar Python example:
https://github.com/aws-samples/decrypt-das-aws-rds/blob/main/rds-das-decrypt-kinesis-firehose.py

Guess what? I couldn't find a single clear mention of using AWS Encryption SDK in documentation, except for a tiny bit of code. That's pretty surprising!
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/DBActivityStreams.CodeExample.html

Anyway, I've replicated that behaviour already and will fix it shortly.

[Codewolf]

@wobondar Thats brilliant, thank you so much for getting back to me so quickly on this, i thought i was going crazy! 😆

[hezachary]

So far, is there any work around?
Amazon just pull off all of their encryption-sdk for go: aws/aws-encryption-sdk#759
It seems so far there is no way to decrypt databaseActivityEvents by go T_T

BTW, I tried changing the BC to aws-kms, like: rawprovider.NewWithOpts("aws-kms", rawprovider.WithStaticKey("DataKey", decryptedKey)); it result a panic:

        {
            "path": "runtime/signal_unix.go",
            "line": 925,
            "label": "sigpanic"
        },
        {
            "path": "github.com/chainifynet/aws-encryption-sdk-go@v0.5.0/pkg/providers/rawprovider/rawprovider.go",
            "line": 205,
            "label": "(*RawKeyProvider[...]).DecryptDataKeyFromList"
        },

[wobondar]

Amazon just pull off all of their encryption-sdk for go: aws/aws-encryption-sdk#759 It seems so far there is no way to decrypt databaseActivityEvents by go T_T

yup, I have seen that, hopefully they'll make it better.

BTW, I tried changing the BC to aws-kms, like: rawprovider.NewWithOpts("aws-kms", rawprovider.WithStaticKey("DataKey", decryptedKey)); it result a panic:

        {
            "path": "runtime/signal_unix.go",
            "line": 925,
            "label": "sigpanic"
        },
        {
            "path": "github.com/chainifynet/aws-encryption-sdk-go@v0.5.0/pkg/providers/rawprovider/rawprovider.go",
            "line": 205,
            "label": "(*RawKeyProvider[...]).DecryptDataKeyFromList"
        },

Thanks, I will have to look into that as well.