Blog : How to setup dbt for materialize db with streaming data from redpanda

[entechlog]

Hello,

Thank You for todays Hack day, I followed along todays demo and created this blog
https://www.entechlog.com/blog/data/how-to-setup-dbt-for-materialize-db-with-streaming-data-from-redpanda/

I also added this example to other dbt demo examples I maintain in my repo https://github.com/entechlog/dbt-examples

Thanks
Siva

[morsapaes]

Thanks for joining and for the write-up, @entechlog! I went through it and the step-by-step instructions are very neat. Learned a few more things about dbt (I'm pretty new to it)! 👍

Regarding:

Source represent a connection to the data you want Materialize to process, as well as details about the structure of that data. Materialize source is not same as dbt sources

We are going to refactor the dbt-materialize plugin soon: when it was created, Materialize was still in its early stages. In the future, sources will be modelled as external tables; from your experience with dbt, is this intuitive? If you have any further feedback about your experience with the adapter, and how hard (or not) it was to understand what Materialize does, it'd be really valuable!

[entechlog]

@morsapaes , Happy to hear that the write-up was able to share few more thing about dbt.

• Regarding the adapter, I have used dbt with Snowflake and Postgres in the path and So far I found materialize also easy to use (Of Couse with help from documentation).
• Regarding the planned change in the behavior of sources, Is the idea to define the external sources like an yml like the examples here https://github.com/dbt-labs/dbt-external-tables/tree/0.6.3/sample_sources ?

[morsapaes]

@entechlog, that would be the way to define a source with our planned work, yeah. What do you think?

In the end, a source in Materialize is akin to an external/foreign table (i.e. data that lives outside of the database, but that can be queried like any other table once you pin down its external location), and the adapter currently goes a bit against the grain of dbt by having users define these objects as models (which should be used exclusively for transformations, AFAIU).