🏫 Awesome Stream Processing 🏫

The term "stream processing" might sound intimidating to many people. We often hear statements like:

• "Stream processing is too difficult to learn and use!" 😱
• "Stream processing is very expensive!" 😱
• "I don’t see any business use cases for stream processing!" 😱

However, we believe this isn't true. ❌

Streaming data is everywhere, generated from operational databases, messaging queues, IoT devices, and many other sources. People can leverage modern stream processing technology to easily address classic real-world problems, using SQL as the programming language.

In this repository, we provide a series of executable demos demonstrating how stream processing can be applied in practical scenarios:

0. Getting started ✅

   • Install Kafka, PostgreSQL, and RisingWave, and run minimal toy examples on your device.
   • Integrate RisingWave with other data platforms.

1. Basic stream processing workflow

   Learn the fundamentals of ingesting, processing, transforming, and offloading data from streaming systems.

   1. Querying and processing event streaming data ✅
   • Directly query data stored in event streaming systems (e.g., Kafka, Redpanda).
   • Continuously ingest and analyze data from event streaming systems.
   2. Bringing analytics closer to operational databases ✅
   • Offload event-driven queries (e.g., materialized views and triggers) from operational databases (e.g., MySQL, PostgreSQL).
   3. Real-time ETL (Extract, Transform, Load) ✅
   • Perform ETL continuously and incrementally.

2. Simple demonstrations ✅

   • A collection of simple, self-contained demos showcasing how stream processing can be applied in specific industry use cases.

3. Solution demonstrations ✅

   • A collection of comprehensive demos showcasing how to build a stream processing pipeline for real-world applications.

We use RisingWave as the default stream processing system to run these demos. We also assume that you have Kafka and/or PostgreSQL installed and possess basic knowledge of how to use these systems. These demos have been verified on Ubuntu and Mac.

All you need is a laptop 💻 - no cluster is required.

Any comments are welcome. Happy streaming!

Join our Slack community to engage in discussions with thousands of stream processing enthusiasts!