Unable to access bucket (HTTP 403) when using IAM Roles attached to Service Accounts on EKS

[NicholasFiorentini]

Please specify whether your issue is about:

• a possible bug
• a question about package functionality
• a suggested code or documentation change, improvement to the code, or feature request

I'm running my code from rshiny server. The session token is provided using https://docs.aws.amazon.com/eks/latest/userguide/pod-configuration.html.

## Install package
install.packages(c(
    "aws.ec2metadata",
    "aws.signature", 
    "aws.s3",
  ),
  repos=c(cloudyr = "http://cloudyr.github.io/drat", getOption("repos"))
)

# tried this:
# install.packages("aws.s3", repos="https://rforge.net")
# but the packages is not loaded

## load package
library(aws.ec2metadata)
library(aws.signature)
library(aws.s3)

## Minimal example
credentials <- aws.signature::locate_credentials()
    
bucket_exist <- aws.s3::bucket_exists("name_of_my_bucket")
cat(file=stderr(), paste0("AWS_DEFAULT_REGION: ", Sys.getenv("AWS_DEFAULT_REGION"), "\n"))
cat(file=stderr(), paste0("Bucket exists: ", bucket_exist, "\n"))

The bucket_exists call fails with the error:

Client error: (403) Forbidden
 Warning: Error in : Bucket name_of_my_bucket does not exist.
] Error : Bucket name_of_my_bucket does not exist.

Further details

The output of credentials shows the correct key id, secret, toke, and region.

The attached token is valid: running awscli from the same pod I can access the bucket without error.

Output of sessionInfo()

platform = "x86_64-pc-linux-gnu"
arch = "x86_64"
os = "linux-gnu"
system = "x86_64, linux-gnu"
status = ""
major = "4"
minor = "2.3"
`svn rev` = "83980"
language = "R"
version.string = "R version 4.2.3 (2023-03-15)"
nickname = "Shortstop Beagle"
LC_CTYPE=en_US.UTF-8;
LC_NUMERIC=C;
LC_TIME=en_US.UTF-8;
LC_COLLATE=en_US.UTF-8;
LC_MONETARY=en_US.UTF-8;
LC_MESSAGES=en_US.UTF-8;
LC_PAPER=en_US.UTF-8;
LC_NAME=C;
LC_ADDRESS=C;
LC_TELEPHONE=C;
LC_MEASUREMENT=en_US.UTF-8;
LC_IDENTIFICATION=C
Package = "aws.s3", Type = "Package", Title = "'AWS S3' Client Package", Version = "0.3.22"
Package = "aws.signature", Type = "Package", Title = "Amazon Web Services Request Signatures", Version = "0.6.0",
Package = "aws.ec2metadata", Type = "Package", Title = "Get EC2 Instance Metadata", Version = "0.2.0"

[NicholasFiorentini]

Just to double-check, I tried this:

cat(file=stderr(), paste0("Key: ", aws_credentials$key, "\n"))
cat(file=stderr(), paste0("Secret: ", aws_credentials$secret, "\n"))
cat(file=stderr(), paste0("Token: ", aws_credentials$session_token, "\n"))
cat(file=stderr(), paste0("Region: ", aws_credentials$region, "\n"))

bucket_access <- aws.s3::bucket_exists(
        "my_bucket",
        key=aws_credentials$key,
        secret=aws_credentials$secret,
        session_token=aws_credentials$session_token,
        region=aws_credentials$region)

Output:

Key: ***
Secret: ***
Token: ***
Region: us-west-2

Client error: (403) Forbidden

Does aws.s3 support STS tokens?

[andrewhharmon]

I'm facing the same issue. Any workarounds?

[NicholasFiorentini]

Ultimately, I moved to a different library to overcome this limitation.

The workaround would be using an AWS API key, but I would avoid it.

[mfoos]

hi folks, not sure if this is the same problem but I hope people getting a 403 despite all evidence credentials are right will find it. The issue for me was that, according to my IT team, "the aws.ec2metadata R package doesn't support IMDSv2". Currently more info here: cloudyr/aws.ec2metadata#10

[s-u]

Just reiterating above - the solution seems to be to use cloudyr/aws.ec2metadata#11

Unfortunately I could not reach the maintainer and I don't have rights in the repo to accept the PR (or fix it myself).

[asaha13]

Hi All,
Do we have any solution with the package? We are seeing the same issue while trying to access bucket from PODS using aws.s3

[s-u]

@asaha13 see above - install aws.ec2metadata with IMDSv2 support referenced above.

[asaha13]

@s-u : Thank you for the note. I tried to setup aws.ec2metadata and afterwards tried aw.s3 but still seeing the same error on pods.

library(aws.ec2metadata)
library(aws.s3)
get_bucket(bucket = 'xxxx')
List of 4
$ Code : chr "AccessDenied"
$ Message : chr "Access Denied"
$ RequestId: chr "KHYBMWR72AGMPDBR"

• attr(*, "headers")=List of 7
• attr(*, "class")= chr "aws_error"
  NULL
  Error in parse_aws_s3_response(r, Sig, verbose = verbose) :
  Forbidden (HTTP 403).

Strangely the I am able to list out buckets/folders/files if I use aws cli on the same pod.

[throrin19]

@NicholasFiorentini Which different library are you using? We facing same problem after try to switch from using access/secret to service account with attached IAM role