Example project for consuming AWS Kinesis streamming and save data on Amazon Redshift using Apache Spark
Code from: Processing IoT realtime data - Medium
You need to set Amazon Credentials on your enviroment.
export AWS_ACCESS_KEY_ID=""
export AWS_ACCESS_KEY=""
export AWS_SECRET_ACCESS_KEY=""
export AWS_SECRET_KEY=""Must be included on --packages flag.
org.apache.spark:spark-streaming-kinesis-asl_2.10:1.6.1
How run Kinesis locally?
A few months ago I created a Docker image with Kinesalite (amazin project to simulate Amazon Kinesis), you can use this image, or run Kinesalite directly.
docker run -d -p 4567:4567 vsouza/kinesis-local -p 4567 --createStreaMs 5
check the project
I should have DynamoDB too?
Yes, 😢 . The AWS SDK Kinesis module make checkpoints of your Kinesis tunnel, and store this on DynamoDB. You don't need to create tables or else, the SDK will create for you.
Remember to configure your throughput value in DynamoDB correctly
MIT License © Vinicius Souza