Allow multiple materialized views to write to same target (#280)

[the4thamigo-uk]

Summary

Proposed mechanism to support having multiple materialized views pointing to the same target table.

In brief, we add tags to mark blocks of the sql which will form each of the materialized views to be created.

i.e. https://github.com/ClickHouse/dbt-clickhouse/pull/364/files#diff-b1cddf2e5c9b3d08ea2276e5cef4944b06b9744a4076c5970ff5a5edb0140f08R62.

The entire union of all such blocks is used to backfill the target table.

Checklist

Delete items not relevant to your PR:

• Unit and integration tests covering the common scenarios were added
• A human-readable description of the changes was provided to include in CHANGELOG
• For significant changes, documentation in https://github.com/ClickHouse/clickhouse-docs was updated with further explanations or tutorials

[the4thamigo-uk]

issue #280

[BentsiLeviav]

Hi @the4thamigo-uk

Thank you for your contribution!
Before reviewing your PR, it is required to add a short description with a PR link to the changelog (please keep the current format we have in the Changelog file).
With this kind of feature, I think it is also important to add a note in the documentation (readme file).

looking forward to reviewing this

[the4thamigo-uk]

@BentsiLeviav Ok, Ive done this.

[BentsiLeviav]

Could you also test for updates and full refresh?

[the4thamigo-uk]

Yeah I think if you change the model to add or rename MVs, then old ones will be left running, which is incorrect.

[BentsiLeviav]

Thanks @the4thamigo-uk
We appreciate this contribution!
The code LGTM. I left a small comment regarding the tests.

[BentsiLeviav]

I’ve added the tests through your branch.

Regarding the potential issue of unremoved materialized views (MVs): as you mentioned, this situation occurs when a user renames one (or more) of the MVs. What I’ve implemented for now is a query in ClickHouse to identify any MV that matches the pattern of the model name but is not listed in the model. If such an MV is found, a warning is raised to notify the user.

I’m hesitant about automatically dropping these MVs, as they could be valid and intentionally separate from the model.

Additionally, I’ve updated the README file to include this important information.

Let’s monitor this behavior for now. If anyone in the community has a better or safer solution, contributions are welcome!

[the4thamigo-uk]

Amazing, thanks for finishing this off. I was actually working on this yesterday and implemented a similar thing to drop the dangling views, but as you say this might be dangerous.

However, I did change the naming convention for the views so that the view name is always appended after a _mv_. I think in this way it is a bit safer to drop anything that is prefixed with hackers_mv_. It also ensures that all materialized view that are created by the model start with this prefix, including the current behaviour.

hackers_mv1 -> hackers_mv_mv1