The sqs-sink drop events when sqs sending fails

[RickardCardell]

SqsSinkConnectorTask lack proper error handling when the sending to SQS fails. The task doesn't fail or retry upon an error. It just logs the error and keeps going.

https://github.com/Nordstrom/kafka-connect-sqs/blob/master/src/main/java/com/nordstrom/kafka/connect/sqs/SqsSinkConnectorTask.java#L91

[alessioerosferri]

Hi @RickardCardell, does this mean that the offset to Kafka is committed regardless of the successful acknowledgment of the message in SQS?

[RickardCardell]

Hi @RickardCardell, does this mean that the offset to Kafka is committed regardless of the successful acknowledgment of the message in SQS?

Yes. Proper error handling is just not implemented yet.

[yordis]

You can either send as a batch the messages to keep the throughput or commit per everything single message slowing the system down.

Inspired by this, we are doing something similar in Elixir lang, and we are facing some issues due to SQS message size and figuring out how to tune the batch size.