Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting an intermittent access denied error when running get_object #3091

Closed
thebkbuffalo opened this issue Aug 20, 2024 · 4 comments
Closed
Assignees
Labels
guidance Question that needs advice or information. p2 This is a standard priority issue response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.

Comments

@thebkbuffalo
Copy link

Describe the bug

When trying to retrieve an object from an s3 bucket using get_object I'm intermittently getting an access denied error from AWS. The only info I'm getting back from AWS is #<Aws::S3::Errors::AccessDenied: Access Denied> which isn't really all that helpful. I'm assuming it's not a credentials problem since I'm also writing to s3 in this process with no issues, and also because get_object works sometimes, but definitely not every time.
This process is happening on a Sidekiq worker on an ECS instance.

csv_str = client.get_object(bucket: bucketname, key: filename).body.read
parsed_csv = CSV.parse(csv_str, headers: true, encoding: "ISO8859-1:utf-8")
return parsed_csv

Expected Behavior

I expected get_object to work every time.

Current Behavior

Currently get_object only works sometimes.

Reproduction Steps

instantiate a new s3 client, call get_object on the client.

Possible Solution

No response

Additional Information/Context

No response

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk-s3

Environment details (Version of Ruby, OS environment)

Ruby version 3.2.1, AWS ECS Sidekiq worker env.

@thebkbuffalo thebkbuffalo added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Aug 20, 2024
@mullermp
Copy link
Contributor

Aws::S3::Errors::AccessDenied is most certainly a service side error, so I'm not sure it's specifically a Ruby SDK issue. How are you configuring credentials - is it manual or automatically? I see that you're using ECS. Is it possible you are getting different sets of credentials intermittently? Or those credentials become stale? If you are using ECSCredentials, try configuring them with ECSCredentials.new(http_debug_output: <some logger>) and observe if, after failures, credentials are refreshed?

@mullermp mullermp added investigating Issue is being investigated and removed bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Aug 20, 2024
@RanVaknin
Copy link

Hi @thebkbuffalo

To add on top of what @mullermp said, if you are seeing intermittent access denied errors, it is possible that your requests are using an incorrect set of credentials. It can happen for many reasons, and all likely stem from a failure of the SDK's credential provider chain from being able to retrieve credentials for the desired ECS environment. It will then attempt to resolve credentials by invoking other providers in the credential chain, and if one of those is configured to return a set of credentials it might result in the wrong role / principal to be assumed and obtained and therefore might lead to access denied errors.

In an ECS environment, you usually have a token associated with the ECS container. Under the hood, the SDK makes an implicit invisible call to exchange that token for a set of temporary credentials. Configuring a logger would result in those implicit API calls to become visible in your logging infrastructure and you can see what is failing and why.

Let us know if you have any questions.

Thanks,
Ran~

@RanVaknin RanVaknin added guidance Question that needs advice or information. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. p2 This is a standard priority issue and removed investigating Issue is being investigated labels Aug 28, 2024
@thebkbuffalo
Copy link
Author

@mullermp you were totally right. it was a server side error that happening. thank you and @RanVaknin for your help!

Copy link

github-actions bot commented Sep 4, 2024

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
guidance Question that needs advice or information. p2 This is a standard priority issue response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days.
Projects
None yet
Development

No branches or pull requests

3 participants