Initial cut of AWS Kinesis streaming

[dpbevin]

Relates to #8966

An initial attempt at producing a persistent stream implementation for AWS Kinesis streams. This is 100% an early PR to solicit feedback (and gauge interest?).

Tested locally against: https://github.com/etspaceman/kinesis-mock

This approach uses the AmazonKinesisClient from the .NET SDK, not the "KCL Consumer" that requires a running JVM 🤮

Includes

• Kinesis stream provider
• A grain-based queue checkpointer (instead of needing to require DynamoDB, EF Core, or some other approach). This checkpointer could be used by EventHub too (instead of requiring Azure Table Storage).

Remaining work

• A few "TODO"s
• Tests

Useful references:

• https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/Kinesis/TKinesisClient.html
• https://kashifsoofi.github.io/aspnetcore,%20aws,%20localstack/kinesis-with-localstack-and-dotnetcore/

Microsoft Reviewers: Open in CodeFlow

[scalalang2]

this is what I've been waiting for. thanks :)

[scalalang2]

I'm here to drop a very trivial question : )
Is there any reason not to use the native checkpoint of Kinesis?

[dpbevin]

@scalalang2 to my knowledge, Kinesis doesn't have the ability to "keep track" of the current position for a consumer so that it can resume from a specified position if the consumer is stopped.
Kafka has the capability (I think it's called "Commit").

But if you can point me in the direction of the Kinesis API you have in mind, I'm happy to take a look.

[scalalang2]

@dpbevin
Thanks for reply : )

to my knowledge, Kinesis doesn't have the ability to "keep track" of the current position for a consumer

You're right, I misunderstood Kinesis natively has concept of "checkpoint".

It uses DynamoDB to save checkpoint sample code official docs,

[dpbevin]

Appreciate the info.

I could have added a DynamoDB checkpointer, which would have been similar to how the Azure EventHub streaming uses Azure Table storage for checkpoints but I felt that a grain-based solution would work in more places and... my intended use of Kinesis streaming won't be able to make use of DynamoDB.

[scalalang2]

I also agree this, thx : )

I felt that a grain-based solution would work in more places

[dpbevin]

CI build failing due to missing package even though the package can be selected from that feed locally.

Unable to find package AWSSDK.Kinesis. No packages exist with this id in source(s): dotnet-public

[scalalang2]

CI build failing due to missing package even though the package can be selected from that feed locally.

Unable to find package AWSSDK.Kinesis. No packages exist with this id in source(s): dotnet-public

@ReubenBond
Could you please check this out?

[idanbenyehoshua]

Just wanted to check in and see if there’s been any progress on this. It’s something I’ve been hoping to have for a while.
Currently, I'm running on AWS and using SQS. However, I have another project where Kinesis would be a better fit.