add sqs.send.mode config and allow async send #53
Conversation
|
@dylanmei It would be great if you could review this. We weren't able to get a good enough throughput with the sync send, and the async mode helped sustain our production load. |
|
What limits were you reaching and what throughput are you achieving now? |
|
Please include information about the sync mode on the README. |
|
|
||
| for (Future<SendMessageResult> futureResult: futureResults) { | ||
| try { | ||
| SendMessageResult result = futureResult.get(30, TimeUnit.SECONDS); |
There was a problem hiding this comment.
Tell us more about this the error conditions here and this 30 second timeout.
There was a problem hiding this comment.
The 30 second timeout is to make sure that we yield control to the caller with a precise exception (timeout exception), rather than using a get() call without timeout and getting blocked on it without logging anything where it could be hard to debug issues with a "hung" connector.
I am not very sure about the best way to know if a send() op succeeded. The result.getMessageId() being valid is a guess that it would denote a successful operation.
We noticed that with a single topic-partition and a single Sink task, I noticed about 2-3 messages per second throughput. One way to get around this would have been to increase the topic's partition count and include more tasks for the connector. However, using the async mode got us a throughput of about 80 messages per second. Note that the individual messages themselves were small (~250 bytes each) in our experiment. While network latency would be a factor affecting the overall latency, I think the use of async mode would alleviate the throughput in any scenario without needing to scale horizontally. |
Sync send calls are very limited in throughput. Utilizing async send calls boosts the throughput.
The config is optional with the default being "sync" mode.