Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] aws-core-sdk 3.207.0 #258

Closed
nastiazaran opened this issue Sep 24, 2024 · 7 comments · Fixed by #260
Closed

[BUG] aws-core-sdk 3.207.0 #258

nastiazaran opened this issue Sep 24, 2024 · 7 comments · Fixed by #260
Assignees
@dlvenable
Labels
bug Something isn't working
Milestone
v2.0.3

Comments

@nastiazaran
Copy link

Describe the bug
There is a new minor version of aws-core-sdk gem -> https://rubygems.org/gems/aws-sdk-core/versions/3.207.0
This version requires the 'account_id' parameter -> https://github.com/aws/aws-sdk-ruby/blob/version-3/gems/aws-sdk-core/CHANGELOG.md?plain=1#L13
My current config, which was working fine with the gem version 3.205.0, uses the aws_iam auth_type to authenticate with the iam profile of the instance:

        auth_type => {
            type => 'aws_iam'
            region => 'us-east-1'
        }

Since the new version release I'm not able to use this output plugin with my current config, I've tried to add the 'account_id' parameter but I get the same error:

Pipeline error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined method account_id' for #LogStash::Outputs::OpenSearch::HttpClient::AWSIAMCredential:0x4bc2e154>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-3.207.0/lib/aws-sdk-core/credential_provider_chain.rb:49:in static_credentials'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-3.207.0/lib/aws-sdk-core/credential_provider_chain.rb:13:in block in resolve'", "org/jruby/RubyArray.java:1821:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-3.207.0/lib/aws-sdk-core/credential_provider_chain.rb:12:in resolve'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:85:in aws_iam_auth_initialization'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:55:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client.rb:333:in build_adapter'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client.rb:337:in build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client.rb:58:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client_builder.rb:119:in create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client_builder.rb:112:in build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/plugin_mixins/opensearch/common.rb:38:in build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch.rb:227:in register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233:in block in register_plugins'", "org/jruby/RubyArray.java:1821:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:598:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:245:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:190:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:142:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/logs.conf"], :thread=>"#<Thread:0x6fe7fcc9 run>"}

To Reproduce
Steps to reproduce the behavior:

  1. Install this plugin --version '2.0.2' logstash-output-opensearch
  2. Add an output block with this auth type:
        auth_type => {
            type => 'aws_iam'
            region => 'us-east-1'
        }
  1. See error

Expected behavior
Logstash process should start and index data to the aws opensearch cluster using the IAM profile of the instance where logstash is running.
Plugins

  • logstash-integration-jdbc (5.1.8)
  • logstash-integration-aws (0.1.0.pre)
  • logstash-output-opensearch (2.0.2)

Host/Environment (please complete the following information):

  • Amazon Linux 2
  • logstash 7.17.18
@nastiazaran nastiazaran added bug Something isn't working untriaged labels Sep 24, 2024
@dblock dblock removed the untriaged label Sep 24, 2024
@dblock
Copy link
Member

dblock commented Sep 24, 2024

I'd dig a little deeper to find out the change in the SDK that caused this, what does that account ID need to be set to?

Did you try adding the account id here:

logstash-output-opensearch/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb

Line 29 in ccd571f

AWSIAMCredential = Struct.new(

@nastiazaran
Copy link
Author

Hi,

I don't know what is the purpose for the account id parameter in this case, I just want to use the instance IAM profile. I'm sorry but I've not so much knowledge about ruby/packages, I'm just a 'final user'

I've tried what you suggested (adding account id to manticore_adapter.rb but then I have the following error:

[2024-09-24T13:01:20,688][ERROR][logstash.javapipeline ] Pipeline error {:pipeline_id=>"main", :exception=>#<NameError: uninitialized constant Aws::Sigv4>, :backtrace=>["org/jruby/RubyModule.java:3766:in const_missing'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:205:in sign_aws_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:167:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:272:in perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:226:in health_check_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:233:in block in healthcheck!'", "org/jruby/RubyHash.java:1415:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:231:in healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:320:in update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:92:in update_initial_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:86:in start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client.rb:354:in build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client.rb:58:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client_builder.rb:119:in create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client_builder.rb:112:in build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/plugin_mixins/opensearch/common.rb:38:in build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch.rb:227:in register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233:in block in register_plugins'", "org/jruby/RubyArray.java:1821:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:598:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:245:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:190:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:142:in block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/logs.conf"], :thread=>"#<Thread:0x713c4a76 run>"}

Thanks for your help

@dblock
Copy link
Member

dblock commented Sep 24, 2024

Looks like it's more complicated. Just to set expectations I won't have cycles to debug this, hopefully someone reading this can help.

@weizhu-us
Copy link

I am seeing the same error on aws-sdk-core-3.209.1
is there a work around? I am stuck

@kabiradar
Copy link

kabiradar commented Oct 8, 2024 •
edited
Loading

I am running into same error while running unit tests
command I am running

/bin/bash scripts/unit-test/docker-setup.sh
/bin/bash scripts/unit-test/docker-run.sh

Error trace

unit-test-logstash-1  | Failures:
unit-test-logstash-1  |
unit-test-logstash-1  |   1) LogStash::Outputs::OpenSearch::HttpClient::ManticoreAdapter aws_iam with a signer should validate AWS IAM service_name config
unit-test-logstash-1  |      Failure/Error: @credentials = Aws::CredentialProviderChain.new(credential_config).resolve
unit-test-logstash-1  |
unit-test-logstash-1  |      NoMethodError:
unit-test-logstash-1  |        undefined method `account_id' for #<LogStash::Outputs::OpenSearch::HttpClient::AWSIAMCredential:0x67373143>
unit-test-logstash-1  |      # ./lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:85:in `aws_iam_auth_initialization'
unit-test-logstash-1  |      # ./lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:55:in `initialize'
unit-test-logstash-1  |      # ./spec/unit/outputs/opensearch/http_client/manticore_adapter_spec.rb:71:in `block in subject'
unit-test-logstash-1  |      # ./spec/unit/outputs/opensearch/http_client/manticore_adapter_spec.rb:96:in `block in <main>'
unit-test-logstash-1  |
unit-test-logstash-1  |   2) LogStash::Outputs::OpenSearch::HttpClient::ManticoreAdapter aws_iam with a signer should validate signing aws request
unit-test-logstash-1  |      Failure/Error: @credentials = Aws::CredentialProviderChain.new(credential_config).resolve
unit-test-logstash-1  |
unit-test-logstash-1  |      NoMethodError:
unit-test-logstash-1  |        undefined method `account_id' for #<LogStash::Outputs::OpenSearch::HttpClient::AWSIAMCredential:0x2a763c46>
unit-test-logstash-1  |      # ./lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:85:in `aws_iam_auth_initialization'
unit-test-logstash-1  |      # ./lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:55:in `initialize'
unit-test-logstash-1  |      # ./spec/unit/outputs/opensearch/http_client/manticore_adapter_spec.rb:71:in `block in /usr/share/plugins/plugin/spec/unit/outputs/opensearch/http_client/manticore_adapter_spec.rb'
unit-test-logstash-1  |      # ./spec/unit/outputs/opensearch/http_client/manticore_adapter_spec.rb:101:in `block in <main>'
unit-test-logstash-1  |
unit-test-logstash-1  |   3) LogStash::Outputs::OpenSearch::HttpClient::ManticoreAdapter aws_iam with a signer should validate AWS IAM credentials initialization
unit-test-logstash-1  |      Failure/Error: @credentials = Aws::CredentialProviderChain.new(credential_config).resolve
unit-test-logstash-1  |
unit-test-logstash-1  |      NoMethodError:
unit-test-logstash-1  |        undefined method `account_id' for #<LogStash::Outputs::OpenSearch::HttpClient::AWSIAMCredential:0x25f21a0d>
unit-test-logstash-1  |      # ./lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:85:in `aws_iam_auth_initialization'
unit-test-logstash-1  |      # ./lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:55:in `initialize'
unit-test-logstash-1  |      # ./spec/unit/outputs/opensearch/http_client/manticore_adapter_spec.rb:71:in `block in /usr/share/plugins/plugin/spec/unit/outputs/opensearch/http_client/manticore_adapter_spec.rb'
unit-test-logstash-1  |      # ./spec/unit/outputs/opensearch/http_client/manticore_adapter_spec.rb:91:in `block in <main>'
unit-test-logstash-1  |
unit-test-logstash-1  |   4) LogStash::Outputs::OpenSearch::HttpClient::ManticoreAdapter aws_iam sign_aws_request handles UTF-8
unit-test-logstash-1  |      Failure/Error: @credentials = Aws::CredentialProviderChain.new(credential_config).resolve
unit-test-logstash-1  |
unit-test-logstash-1  |      NoMethodError:
unit-test-logstash-1  |        undefined method `account_id' for #<LogStash::Outputs::OpenSearch::HttpClient::AWSIAMCredential:0x7e3fa3eb>
unit-test-logstash-1  |      # ./lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:85:in `aws_iam_auth_initialization'
unit-test-logstash-1  |      # ./lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:55:in `initialize'
unit-test-logstash-1  |      # ./spec/unit/outputs/opensearch/http_client/manticore_adapter_spec.rb:71:in `block in /usr/share/plugins/plugin/spec/unit/outputs/opensearch/http_client/manticore_adapter_spec.rb'
unit-test-logstash-1  |      # ./spec/unit/outputs/opensearch/http_client/manticore_adapter_spec.rb:122:in `block in <main>'
unit-test-logstash-1  |
unit-test-logstash-1  |   5) LogStash::Outputs::OpenSearch::HttpClient::ManticoreAdapter aws_iam sign_aws_request encodes body before signing to match manticore adapter encoding
unit-test-logstash-1  |      Failure/Error: @credentials = Aws::CredentialProviderChain.new(credential_config).resolve
unit-test-logstash-1  |
unit-test-logstash-1  |      NoMethodError:
unit-test-logstash-1  |        undefined method `account_id' for #<LogStash::Outputs::OpenSearch::HttpClient::AWSIAMCredential:0xd0e6c29>
unit-test-logstash-1  |      # ./lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:85:in `aws_iam_auth_initialization'
unit-test-logstash-1  |      # ./lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:55:in `initialize'
unit-test-logstash-1  |      # ./spec/unit/outputs/opensearch/http_client/manticore_adapter_spec.rb:71:in `block in /usr/share/plugins/plugin/spec/unit/outputs/opensearch/http_client/manticore_adapter_spec.rb'
unit-test-logstash-1  |      # ./spec/unit/outputs/opensearch/http_client/manticore_adapter_spec.rb:139:in `block in <main>'
unit-test-logstash-1  |
unit-test-logstash-1  | Finished in 1 minute 17.05 seconds (files took 12.41 seconds to load)
unit-test-logstash-1  | 178 examples, 5 failures

I made few changes in manitcore_adapter.rb
And unit tests are passing now.
Added account_id in AWSIAMCredential struct

       AWSIAMCredential = Struct.new(
         :access_key_id,
         :secret_access_key,
         :session_token,
         :account_id,
         :profile,
         :instance_profile_credentials_retries,
         :instance_profile_credentials_timeout,
         :region)

Added getter and setter for account_id

        def get_account_id()
          @account_id
        end
        
        def set_account_id(account_id)
          @account_id = account_id
        end

But I am not sure thats right way to fix this issue
@dblock can you check and advise here?

@dblock
Copy link
Member

dblock commented Oct 8, 2024

But I am not sure thats right way to fix this issue
@dblock can you check and advise here?

It looks right to me, but I would rely on tests and trying out the change. Appreciate a PR.

@dlvenable
Copy link
Member

I've looked into this issue some. The Ruby SDK added account_id in commit a3f158e. So it appears to be related to DynamoDB.

There are a few closed issues in the Ruby SDK related to this:

From what I can tell, the account_id is intended to support account-based endpoints. However, this is not relevant for Amazon OpenSearch Service which does not support these.

I think a short-term solution may be to add the :account_id and keep it nil. I haven't tried this to see if it works though.

Longer-term, we may need another way to create credentials. This API is marked as private and thus they may make breaking changes like this more often.

https://github.com/aws/aws-sdk-ruby/blob/ed6eec59a7f3a8a6aea7a2ab3f58218090a3e0b0/gems/aws-sdk-core/lib/aws-sdk-core/credential_provider_chain.rb#L4

dlvenable added a commit to dlvenable/logstash-output-opensearch that referenced this issue Oct 17, 2024
@dlvenable
Support the latest versions of Logstash and the AWS SDK for Ruby.
519126e 
Adds the account_id configuration to the AWSIAMCredential struct to work with the latest versions of the AWS SDK for Ruby.

Test against the latest versions of Logstash during the integration testing to also test against these versions of the AWS SDK.

Resolves opensearch-project#258

Signed-off-by: David Venable <[email protected]>
@dlvenable dlvenable mentioned this issue Oct 17, 2024
Merged
5 tasks
@dlvenable dlvenable added this to the v2.0.3 milestone Oct 17, 2024
@dlvenable dlvenable self-assigned this Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dlvenable dlvenable

Labels
bug Something isn't working
Projects
None yet
Milestone
v2.0.3
5 participants
@dlvenable @dblock @weizhu-us @nastiazaran @kabiradar