Skip to content

Commit

Permalink
Add instructions for assume-role
Browse files Browse the repository at this point in the history
  • Loading branch information
Garza, Jose Angel Q committed Mar 1, 2019
1 parent 0d1e9f3 commit d190533
Showing 1 changed file with 60 additions and 2 deletions.
62 changes: 60 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
The SQS connector plugin provides the ability to use AWS SQS queues as both a source (from an SQS queue into a Kafka topic) or sink (out of a Kafka topic into an SQS queue).

## Supported Kafka and AWS versions
The `kafka-connect-sqs` connector has been tested with `connect-api:2.1.0` and `aws-java-sdk-sqs:1.11.452`
The `kafka-connect-sqs` connector has been tested with `connect-api:2.1.0` and `aws-java-sdk-sqs:1.11.501`

# Building
You can build the connector with Maven using the standard lifecycle goals:
Expand All @@ -19,7 +19,7 @@ A source connector configuration has two required fields:
* `sqs.queue.url`: The URL of the SQS queue to be read from.
* `topics`: The Kafka topic to be written to.

There are optional fields:
These are optional fields:
* `sqs.max.messages`: Maximum number of messages to read from SQS queue for each poll interval. Range is 0 - 10 with default of 1.
* `sqs.wait.time.seconds`: Duration (in seconds) to wait for a message to arrive in the queue. Default is 1.

Expand Down Expand Up @@ -49,6 +49,13 @@ A sink connector configuration has two required fields:
* `sqs.queue.url`: The URL of the SQS queue to be written to.
* `topics`: The Kafka topic to be read from.

### AWS Assume Role Support options
The connector can assume a cross-account role to enable such features as Server Side Encryption of a queue:
* `sqs.credentials.provider.class=com.nordstrom.kafka.connect.auth.AWSAssumeRoleCredentialsProvider`: REQUIRED Class providing cross-account role assumption.
* `sqs.credentials.provider.role.arn`: REQUIRED AWS Role ARN providing the access.
* `sqs.credentials.provider.session.name`: REQUIRED Session name
* `sqs.credentials.provider.external.id`: OPTIONAL (but recommended) External identifier used by the `kafka-connect-sqs` when assuming the role.

### Sample Configuration
```json
{
Expand Down Expand Up @@ -86,6 +93,57 @@ For a `sink` connector, the minimum actions required are:
}
```

### AWS Assume Role Support
* Define the AWS IAM Role that `kafka-connect-sqs` will assume when writing to the queue (e.g., `kafka-connect-sqs-role`) with a Trust Relationship where `xxxxxxxxxxxx` is the AWS Account in which Kafka Connect executes:

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::xxxxxxxxxxxx:root"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "my-queue-external-id"
}
}
}
]
}```

* Define an SQS Queue Policy Document for the queue to allow `SendMessage`. An example policy is:

```json
{
"Version": "2012-10-17",
"Id": "arn:aws:sqs:us-west-2:nnnnnnnnnnnn:my-queue/SQSDefaultPolicy",
"Statement": [
{
"Sid": "kafka-connect-sqs-sendmessage",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::nnnnnnnnnnnn:role/kafka-connect-sqs-role"
},
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:us-west-2:nnnnnnnnnnnn:my-queue"
}
]
}
```

The sink connector configuration would then include the additional fields:

```json
sqs.credentials.provider.class=com.nordstrom.kafka.connect.auth.AWSAssumeRoleCredentialsProvider
sqs.credentials.provider.role.arn=arn:aws:iam::nnnnnnnnnnnn:role/kafka-connect-sqs-role
sqs.credentials.provider.session.name=my-queue-session
sqs.credentials.provider.external.id=my-queue-external-id
```

For a `source` connector, the minimum actions required are:

```json
Expand Down

0 comments on commit d190533

Please sign in to comment.