Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kafka-connect-s3 - Trusted Account Credentials Documentation #366

Open
Tamir-Schwarz opened this issue Oct 1, 2020 · 9 comments · May be fixed by #697
Open

kafka-connect-s3 - Trusted Account Credentials Documentation #366

Tamir-Schwarz opened this issue Oct 1, 2020 · 9 comments · May be fixed by #697

Comments

@Tamir-Schwarz
Copy link

Hi,

We are using kafka-connect-s3 connector version 5.5.1.
We tried to use the Trusted Account Credentials feature.

According to the documentation,
we need to add the following properties:

s3.credentials.provider.class=AwsAssumeRoleCredentialsProvider sts.role.arn=arn:aws:iam::012345678901:role/my-restricted-role sts.role.session.name=session-name sts.role.external.id=external-id

It didn't work for us.

After digging the code,
we found that in order to make it work we need to add a prefix 's3.credentials.provider' to 'sts.role.arn', 'sts.role.session.name' and 'sts.role.external.id'

Following is the configuration that worked for us:

s3.credentials.provider.class=AwsAssumeRoleCredentialsProvider s3.credentials.provider.sts.role.arn=arn:aws:iam::012345678901:role/my-restricted-role s3.credentials.provider.sts.role.session.name=session-name s3.credentials.provider.sts.role.external.id=external-id

In addition,
It worth mentioning that 'external.id' is mandatory, while it is not mandatory on AWS

@hafizmujadidKhalid
Copy link

@Tamir-Schwarz How it worked for you? I am getting following error:

connector.class=io.confluent.connect.s3.S3SinkConnector
s3.region=eu-central-1
tasks.max=2
topics=kafka-demo
s3.credentials.provider.class=io.confluent.connect.s3.auth.AwsAssumeRoleCredentialsProvider
s3.credentials.provider.sts.role.arn=arn:aws:iam::xxxxxxxx:role/-demo-owner-readwrite-role
s3.credentials.provider.sts.role.session.name=session-name
s3.credentials.provider.sts.role.external.id=data-key

[Worker-0db15ad1ff259517f] Caused by: com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.

@timvryan
Copy link

@hafizmujadidKhalid Getting the same error, did you find a way to solve?

@hafizmujadidKhalid
Copy link

@hafizmujadidKhalid Getting the same error, did you find a way to solve?

No, it did not work for me.

@akhon
Copy link

akhon commented Oct 13, 2022

same problem

@mjallday
Copy link

public AWSCredentialsProvider getCredentialsProvider() {
try {
AWSCredentialsProvider provider = ((Class<? extends AWSCredentialsProvider>)
getClass(S3SinkConnectorConfig.CREDENTIALS_PROVIDER_CLASS_CONFIG)).newInstance();
if (provider instanceof Configurable) {
Map<String, Object> configs = originalsWithPrefix(CREDENTIALS_PROVIDER_CONFIG_PREFIX);
configs.remove(CREDENTIALS_PROVIDER_CLASS_CONFIG.substring(
CREDENTIALS_PROVIDER_CONFIG_PREFIX.length()
));
configs.put(AWS_ACCESS_KEY_ID_CONFIG, awsAccessKeyId());
configs.put(AWS_SECRET_ACCESS_KEY_CONFIG, awsSecretKeyId().value());
((Configurable) provider).configure(configs);
} else {
final String accessKeyId = awsAccessKeyId();
final String secretKey = awsSecretKeyId().value();
if (StringUtils.isNotBlank(accessKeyId) && StringUtils.isNotBlank(secretKey)) {
BasicAWSCredentials basicCredentials = new BasicAWSCredentials(accessKeyId, secretKey);
provider = new AWSStaticCredentialsProvider(basicCredentials);
}
}
return provider;
} catch (IllegalAccessException | InstantiationException e) {
throw new ConnectException(
"Invalid class for: " + S3SinkConnectorConfig.CREDENTIALS_PROVIDER_CLASS_CONFIG,
e
);
}
}

protected AWSCredentialsProvider newCredentialsProvider(S3SinkConnectorConfig config) {
log.info("Returning new credentials provider based on the configured "
+ "credentials provider class");
return config.getCredentialsProvider();
}

public AWSCredentials getCredentials() {
if (basicCredentials != null) {
return new STSAssumeRoleSessionCredentialsProvider.Builder(roleArn, roleSessionName)
.withStsClient(AWSSecurityTokenServiceClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(basicCredentials)).build())
.withExternalId(roleExternalId)
.build()
.getCredentials();
} else {
return new STSAssumeRoleSessionCredentialsProvider.Builder(roleArn, roleSessionName)
.withStsClient(AWSSecurityTokenServiceClientBuilder.defaultClient())
.withExternalId(roleExternalId)
.build()
.getCredentials();
}
}

first glance it looks like the credential provider is being invoked correctly. we don't see similar issues when using a regular role so it's something related to the assume role configuration.

@gotpist1
Copy link

Im also getting the same error using localstack and root role. Any updates on this? Also, specifying AWS_REGION ENV variable gives another error:

org.apache.kafka.connect.errors.ConnectException: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, com.amazonaws.auth.profile.ProfileCredentialsProvider@16971f4e: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@7cecef68: Failed to connect to service endpoint: ]
  at io.confluent.connect.s3.S3SinkTask.start(S3SinkTask.java:138)```

@karlamaria100
Copy link

Region issue still happening

munkyboy added a commit to munkyboy/kafka-connect-storage-cloud that referenced this issue Oct 28, 2023
This passes the `s3.region` configuration value into credential
providers. It also updates the AwsAssumeRoleCredentialsProvider to
specify a region when building a AWSSecurityTokenServiceClient.

Fixes confluentinc#366
munkyboy added a commit to munkyboy/kafka-connect-storage-cloud that referenced this issue Oct 29, 2023
This adds a `region` configuration item for the
AwsAssumeRoleCredentialsProvider. It is used when building a
AWSSecurityTokenServiceClient.

It not specified, uses the default region selector.

Fixes confluentinc#366
@chriline
Copy link

https://docs.confluent.io/kafka-connectors/s3-sink/current/overview.html#credentials-providers is still using s3.credentials.provider.sts.role.arn, s3.credentials.provider.sts.role.session.name and s3.credentials.provider.sts.role.external.id while https://docs.confluent.io/kafka-connectors/s3-sink/current/configuration_options.html is using sts.role.arn, sts.role.session.name, and sts.role.external.id.

I believe we're also running into @hafizmujadidKhalid's issue:
We've set up io.confluent.connect.s3.auth.AwsAssumeRoleCredentialsProvider like so

s3.credentials.provider.class=io.confluent.connect.s3.auth.AwsAssumeRoleCredentialsProvider
s3.credentials.provider.sts.role.arn=arn:aws:iam::xxxxxxxx:role/my-role
s3.credentials.provider.sts.role.session.name=session-name
s3.credentials.provider.sts.role.external.id=external-id

and get the following error

[Worker-0f3f4a9ad02a8f900] Caused by: com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.

@AyoubOukh
Copy link

getting the same issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
9 participants